Jul 17 01:14:02.484404 ip-10-0-128-44 systemd[1]: Starting Kubernetes Kubelet... Jul 17 01:14:02.919674 ip-10-0-128-44 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 01:14:02.919674 ip-10-0-128-44 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 01:14:02.919674 ip-10-0-128-44 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 01:14:02.919674 ip-10-0-128-44 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 01:14:02.919674 ip-10-0-128-44 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 01:14:02.920363 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.919734 2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 01:14:02.927206 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927181 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 17 01:14:02.927206 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927200 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 01:14:02.927206 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927204 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 01:14:02.927206 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927207 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 01:14:02.927206 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927212 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 01:14:02.927206 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927215 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 01:14:02.927206 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927218 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927221 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927224 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927226 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927229 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927232 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927234 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927237 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927239 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927242 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927245 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927250 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927254 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927258 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927261 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927264 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927267 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927270 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927272 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 01:14:02.927491 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927275 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927278 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927280 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927282 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927285 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927288 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927290 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927293 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927295 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927298 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927301 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927303 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927306 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927308 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927312 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927315 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927319 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927322 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927325 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 01:14:02.928015 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927329 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927331 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927334 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927336 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927339 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927341 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927343 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927346 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927349 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927351 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927354 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927357 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927360 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927362 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927365 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927367 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927370 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927372 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927375 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927377 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 01:14:02.928547 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927381 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927384 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927386 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927389 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927391 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927394 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927396 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927400 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927403 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927405 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927408 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927410 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927413 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927416 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927418 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927420 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927423 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927425 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927428 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927431 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 01:14:02.929020 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927433 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927436 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927438 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927442 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927446 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927448 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927451 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927453 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927456 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927458 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927461 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927463 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927466 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927469 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.927471 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928197 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928207 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928211 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928214 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 01:14:02.929526 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928218 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928221 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928223 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928227 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928229 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928232 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928235 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928238 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928240 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928243 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928245 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928248 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928250 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928253 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928256 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928259 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928261 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928264 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928266 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 01:14:02.929975 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928269 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928271 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928274 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928276 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928278 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928282 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928284 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928287 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928290 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928292 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928295 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928297 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928299 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928302 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928305 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928307 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928310 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928313 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928315 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928318 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 01:14:02.930460 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928320 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928322 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928325 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928328 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928330 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928333 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928335 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928338 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928341 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928343 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928346 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928348 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928350 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928352 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928355 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928357 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928359 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928363 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928365 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928367 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 01:14:02.930937 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928370 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928372 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928375 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928377 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928379 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928382 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928385 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928387 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928389 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928392 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928394 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928396 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928399 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928401 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928404 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928406 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928408 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928410 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928414 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928417 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 01:14:02.931462 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928420 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928423 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928425 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928428 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928432 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928435 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928438 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928440 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928444 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928449 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928452 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928455 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928458 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928460 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928463 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928466 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928855 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928862 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 01:14:02.931932 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928866 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928869 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928872 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928875 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928877 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928880 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928884 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928886 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928889 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928892 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928894 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928896 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928899 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928901 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928904 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928907 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928910 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928912 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928915 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 01:14:02.932380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928917 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928920 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928922 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928924 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928928 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928930 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928933 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928935 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928938 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928940 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928944 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928947 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928952 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928955 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928958 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928961 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928963 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928966 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928969 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928971 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 01:14:02.932835 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928974 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928976 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928978 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928981 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928984 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928986 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928989 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928991 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928994 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928997 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.928999 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929001 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929004 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929006 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929009 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929011 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929015 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929017 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929020 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 01:14:02.933330 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929022 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929026 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929028 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929031 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929034 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929037 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929040 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929043 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929045 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929048 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929050 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929052 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929055 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929057 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929060 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929062 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929064 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929067 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929069 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929071 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 01:14:02.933800 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929074 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929076 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929080 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929082 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929084 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929087 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929089 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929091 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929111 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929116 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929118 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929121 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929123 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929126 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929128 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929130 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929133 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929137 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929140 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929212 2583 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929220 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 01:14:02.934296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929225 2583 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929230 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929234 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929237 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929241 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929245 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929248 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929251 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929254 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929257 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929260 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929263 2583 flags.go:64] FLAG: --cgroup-root="" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929265 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929268 2583 flags.go:64] FLAG: --client-ca-file="" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929272 2583 flags.go:64] FLAG: --cloud-provider="external" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929275 2583 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929279 2583 flags.go:64] FLAG: --cluster-domain="" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929281 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929285 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929288 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929293 2583 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929297 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929299 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 01:14:02.934794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929302 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929306 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929309 2583 flags.go:64] FLAG: --contention-profiling="false" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929311 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929314 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929319 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929322 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929326 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929329 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929332 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929335 2583 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929337 2583 flags.go:64] FLAG: --enable-server="true" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929340 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929345 2583 flags.go:64] FLAG: --event-burst="100" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929348 2583 flags.go:64] FLAG: --event-qps="50" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929351 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929354 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929356 2583 flags.go:64] FLAG: --eviction-hard="" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929360 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929363 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929366 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929368 2583 flags.go:64] FLAG: --eviction-soft="" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929371 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929374 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929377 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 01:14:02.935345 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929380 2583 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929383 2583 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929385 2583 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929388 2583 flags.go:64] FLAG: --feature-gates="" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929393 2583 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929395 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929398 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929401 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929404 2583 flags.go:64] FLAG: --healthz-port="10248" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929408 2583 flags.go:64] FLAG: --help="false" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929410 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-128-44.ec2.internal" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929413 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929418 2583 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929421 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929424 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929427 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929430 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929433 2583 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929435 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929438 2583 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929441 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929444 2583 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929446 2583 flags.go:64] FLAG: --kube-reserved="" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929460 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 01:14:02.935987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929463 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929466 2583 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929469 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929472 2583 flags.go:64] FLAG: --lock-file="" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929474 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929477 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929480 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929485 2583 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929488 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929491 2583 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929494 2583 flags.go:64] FLAG: --logging-format="text" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929497 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929501 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929504 2583 flags.go:64] FLAG: --manifest-url="" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929507 2583 flags.go:64] FLAG: --manifest-url-header="" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929511 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929514 2583 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929517 2583 flags.go:64] FLAG: --max-pods="110" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929520 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929523 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929528 2583 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929531 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929534 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929537 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929540 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 01:14:02.936589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929547 2583 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929550 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929553 2583 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929556 2583 flags.go:64] FLAG: --pod-cidr="" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929559 2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929564 2583 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929566 2583 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929569 2583 flags.go:64] FLAG: --pods-per-core="0" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929572 2583 flags.go:64] FLAG: --port="10250" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929575 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929577 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08655b3040e10e5af" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929580 2583 flags.go:64] FLAG: --qos-reserved="" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929583 2583 flags.go:64] FLAG: --read-only-port="10255" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929586 2583 flags.go:64] FLAG: --register-node="true" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929589 2583 flags.go:64] FLAG: --register-with-taints="" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929592 2583 flags.go:64] FLAG: --registry-burst="10" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929595 2583 flags.go:64] FLAG: --registry-qps="5" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929598 2583 flags.go:64] FLAG: --reserved-cpus="" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929601 2583 flags.go:64] FLAG: --reserved-memory="" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929604 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929608 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929611 2583 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929613 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929616 2583 flags.go:64] FLAG: --runonce="false" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929618 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 01:14:02.937201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929621 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929624 2583 flags.go:64] FLAG: --seccomp-default="false" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929627 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929630 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929632 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929635 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929638 2583 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929640 2583 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929643 2583 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929646 2583 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929649 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929651 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929655 2583 flags.go:64] FLAG: --system-cgroups="" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929657 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929662 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929665 2583 flags.go:64] FLAG: --tls-cert-file="" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929667 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929672 2583 flags.go:64] FLAG: --tls-min-version="" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929675 2583 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929677 2583 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929680 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929682 2583 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929685 2583 flags.go:64] FLAG: --v="2" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929689 2583 flags.go:64] FLAG: --version="false" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929693 2583 flags.go:64] FLAG: --vmodule="" Jul 17 01:14:02.937815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929697 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.929700 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929813 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929816 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929819 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929822 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929825 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929827 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929830 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929833 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929835 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929838 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929840 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929843 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929845 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929847 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929850 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929852 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929855 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929857 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 01:14:02.938435 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929860 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929862 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929866 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929868 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929870 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929874 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929878 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929882 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929886 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929889 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929891 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929894 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929903 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929905 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929908 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929911 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929913 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929916 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929918 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 01:14:02.938917 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929921 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929923 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929926 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929928 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929931 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929933 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929935 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929938 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929940 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929943 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929945 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929947 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929950 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929952 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929955 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929958 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929961 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929964 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929967 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 01:14:02.939377 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929969 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929972 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929974 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929977 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929979 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929982 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929985 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929988 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929990 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929993 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929995 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.929998 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930000 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930002 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930005 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930007 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930009 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930012 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930014 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930016 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 01:14:02.939837 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930019 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930021 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930024 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930026 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930029 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930031 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930033 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930036 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930038 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930040 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930043 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930045 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930049 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930052 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930054 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930057 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930059 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930061 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930064 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930066 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 01:14:02.940368 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930069 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930072 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.930074 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.930079 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.937122 2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.937137 2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.937151 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.937157 2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937211 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937217 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937220 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937223 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937226 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937228 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937231 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 01:14:02.940844 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937234 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937237 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937240 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937243 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937245 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937248 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937250 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937253 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937255 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937258 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937261 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937263 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937266 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937268 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937271 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937273 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937276 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937278 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937280 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937284 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 01:14:02.941289 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937288 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937291 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937294 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937297 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937299 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937302 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937304 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937307 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937309 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937312 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937314 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937317 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937335 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937339 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937342 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937345 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937347 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937350 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937354 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937356 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 01:14:02.941780 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937359 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937362 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937364 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937366 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937369 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937371 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937374 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937376 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937379 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937381 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937383 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937386 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937388 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937391 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937395 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937398 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937400 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937403 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937406 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937408 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 01:14:02.942264 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937411 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937413 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937415 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937418 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937420 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937422 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937425 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937427 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937429 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937431 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937434 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937436 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937439 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937441 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937444 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937446 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937456 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937461 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937465 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 01:14:02.942729 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937467 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937470 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937472 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937475 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937478 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937480 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937483 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937486 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937490 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937494 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937496 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937499 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937501 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.937506 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937634 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 01:14:02.943202 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937639 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937642 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937644 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937647 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937650 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937652 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937655 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937657 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937660 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937662 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937664 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937667 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937670 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937674 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937677 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937679 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937681 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937684 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937687 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 01:14:02.943603 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937689 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937692 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937694 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937697 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937699 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937702 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937704 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937707 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937711 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937714 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937717 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937719 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937722 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937724 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937726 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937729 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937731 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937734 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937736 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 01:14:02.944051 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937738 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937742 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937745 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937748 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937750 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937753 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937756 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937758 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937761 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937763 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937765 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937768 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937770 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937773 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937775 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937778 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937780 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937782 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937785 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 01:14:02.944541 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937787 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937790 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937792 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937795 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937797 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937800 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937802 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937804 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937807 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937809 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937811 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937814 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937816 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937819 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937821 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937824 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937826 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937829 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937831 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937834 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 01:14:02.944983 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937836 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937839 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937841 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937843 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937846 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937848 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937850 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937853 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937855 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937857 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937860 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937863 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937865 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937867 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937870 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937872 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937875 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937877 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937880 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937882 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 01:14:02.945527 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:02.937884 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 01:14:02.946044 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.937888 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 01:14:02.946044 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.937987 2583 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 01:14:02.946044 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.940125 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 01:14:02.946044 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.941581 2583 server.go:1019] "Starting client certificate rotation" Jul 17 01:14:02.946044 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.941686 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 01:14:02.946044 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.942385 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 01:14:02.946044 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.944296 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 01:14:02.947446 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.947426 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 01:14:02.962109 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.962085 2583 log.go:25] "Validated CRI v1 runtime API" Jul 17 01:14:02.964276 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.964260 2583 log.go:25] "Validated CRI v1 image API" Jul 17 01:14:02.965056 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.965025 2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 01:14:02.967168 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.967150 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 01:14:02.967900 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.967875 2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ce886273-4163-43a4-93ad-6c4d5bcd033e:/dev/nvme0n1p4 dd1110b8-dc43-4da9-924c-23e486d6541e:/dev/nvme0n1p3] Jul 17 01:14:02.967973 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.967899 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 01:14:02.973401 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.973264 2583 manager.go:216] Machine: {Timestamp:2026-07-17 01:14:02.971867228 +0000 UTC m=+0.367482091 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098888 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec221367f2444f5152c3c21640951f5a SystemUUID:ec221367-f244-4f51-52c3-c21640951f5a BootID:00542f0f-916b-4028-95a6-191f902d261d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fc:f5:58:53:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fc:f5:58:53:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:c3:d5:06:d8:9b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 01:14:02.973401 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.973394 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 01:14:02.973578 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.973511 2583 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 01:14:02.976467 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.976443 2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 01:14:02.976651 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.976469 2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-44.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 01:14:02.977301 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.977288 2583 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 01:14:02.977363 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.977306 2583 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 01:14:02.977363 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.977323 2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 01:14:02.978086 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.978074 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 01:14:02.979416 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.979404 2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 01:14:02.982826 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.982812 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 17 01:14:02.982959 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.982948 2583 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 01:14:02.983026 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.982991 2583 kubelet.go:486] "Attempting to sync node with API server" Jul 17 01:14:02.983026 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.983006 2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 01:14:02.983026 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.983026 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 01:14:02.983181 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.983039 2583 kubelet.go:398] "Adding apiserver pod source" Jul 17 01:14:02.983181 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.983051 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 01:14:02.983670 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.983655 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 01:14:02.983733 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.983685 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 01:14:02.985613 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.985575 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c59rd" Jul 17 01:14:02.987561 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.987544 2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 01:14:02.989424 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.989409 2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 01:14:02.989498 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.989439 2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 01:14:02.990556 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.990543 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 01:14:02.990624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.990559 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 01:14:02.990624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.990565 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 01:14:02.990624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.990570 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 01:14:02.990624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.990578 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 01:14:02.990624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.990583 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 01:14:02.990624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.990591 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 01:14:02.990624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.990598 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 01:14:02.990624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.990605 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 01:14:02.990624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.990612 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 01:14:02.990624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.990620 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 01:14:02.990624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.990626 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 01:14:02.991427 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.991415 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 01:14:02.991427 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.991426 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 01:14:02.993353 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.993331 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c59rd" Jul 17 01:14:02.993516 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.993496 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-44.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 01:14:02.994164 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:02.994129 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-44.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 01:14:02.994292 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:02.994233 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 01:14:02.995811 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.995795 2583 server.go:1268] "Started kubelet" Jul 17 01:14:02.995896 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.995851 2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 01:14:02.995969 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.995940 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 01:14:02.996023 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.995989 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 01:14:02.996674 ip-10-0-128-44 systemd[1]: Started Kubernetes Kubelet. Jul 17 01:14:02.997224 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.997210 2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 01:14:02.997841 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:02.997827 2583 server.go:310] "Adding debug handlers to kubelet server" Jul 17 01:14:03.001287 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.001266 2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 01:14:03.004149 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.004126 2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 01:14:03.004229 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.004148 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 01:14:03.004314 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.004289 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-44.ec2.internal\" not found" Jul 17 01:14:03.004979 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.004741 2583 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 01:14:03.004979 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.004759 2583 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 01:14:03.004979 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.004835 2583 reconstruct.go:97] "Volume reconstruction finished" Jul 17 01:14:03.004979 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.004848 2583 reconciler.go:29] "Reconciler: start to sync state" Jul 17 01:14:03.004979 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.004871 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 01:14:03.005680 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.005666 2583 factory.go:55] Registering systemd factory Jul 17 01:14:03.005680 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.005682 2583 factory.go:223] Registration of the systemd container factory successfully Jul 17 01:14:03.005914 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.005899 2583 factory.go:153] Registering CRI-O factory Jul 17 01:14:03.005962 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.005919 2583 factory.go:223] Registration of the crio container factory successfully Jul 17 01:14:03.005993 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.005972 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 01:14:03.006027 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.005997 2583 factory.go:103] Registering Raw factory Jul 17 01:14:03.006027 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.006013 2583 manager.go:1196] Started watching for new ooms in manager Jul 17 01:14:03.006806 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.006786 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 01:14:03.007001 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.006983 2583 manager.go:349] Starting recovery of all containers Jul 17 01:14:03.009296 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.009268 2583 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-44.ec2.internal\" not found" node="ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.014723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.014694 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 01:14:03.016174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.016159 2583 manager.go:354] Recovery completed Jul 17 01:14:03.021624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.021604 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 01:14:03.023988 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.023971 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasSufficientMemory" Jul 17 01:14:03.024052 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.024008 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 01:14:03.024052 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.024023 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasSufficientPID" Jul 17 01:14:03.024601 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.024588 2583 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 01:14:03.024663 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.024600 2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 01:14:03.024663 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.024609 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 01:14:03.024663 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.024618 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 17 01:14:03.025367 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.025350 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasSufficientMemory" Jul 17 01:14:03.025474 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.025379 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 01:14:03.025474 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.025389 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasSufficientPID" Jul 17 01:14:03.026296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.026283 2583 policy_none.go:49] "None policy: Start" Jul 17 01:14:03.026352 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.026302 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 01:14:03.026352 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.026315 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 01:14:03.028744 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.028728 2583 policy_none.go:47] "Start" Jul 17 01:14:03.066664 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.066643 2583 manager.go:337] "Starting Device Plugin manager" Jul 17 01:14:03.066759 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.066677 2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 01:14:03.066759 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.066687 2583 server.go:87] "Starting device plugin registration server" Jul 17 01:14:03.066933 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.066922 2583 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 01:14:03.066970 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.066936 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 01:14:03.067069 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.067052 2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 01:14:03.067173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.067156 2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 01:14:03.067173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.067169 2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 01:14:03.067681 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.067663 2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 01:14:03.067751 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.067708 2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-44.ec2.internal\" not found" Jul 17 01:14:03.161646 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.161613 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 01:14:03.163396 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.161656 2583 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 01:14:03.163396 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.161678 2583 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 01:14:03.163396 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.161713 2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 01:14:03.163879 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.163859 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 01:14:03.167878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.167863 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 01:14:03.168743 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.168726 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasSufficientMemory" Jul 17 01:14:03.168834 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.168761 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 01:14:03.168834 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.168776 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasSufficientPID" Jul 17 01:14:03.168834 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.168805 2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.177958 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.177891 2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.177958 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.177915 2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-44.ec2.internal\": node \"ip-10-0-128-44.ec2.internal\" not found" Jul 17 01:14:03.191335 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.191313 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-44.ec2.internal\" not found" Jul 17 01:14:03.262085 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.262049 2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-44.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal"] Jul 17 01:14:03.262254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.262156 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 01:14:03.263978 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.263962 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasSufficientMemory" Jul 17 01:14:03.264071 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.263994 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 01:14:03.264071 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.264007 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasSufficientPID" Jul 17 01:14:03.265841 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.265830 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 01:14:03.265966 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.265951 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.266016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.265983 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 01:14:03.266545 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.266527 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasSufficientMemory" Jul 17 01:14:03.266611 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.266555 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 01:14:03.266611 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.266530 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasSufficientMemory" Jul 17 01:14:03.266611 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.266567 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasSufficientPID" Jul 17 01:14:03.266611 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.266579 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 01:14:03.266611 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.266588 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasSufficientPID" Jul 17 01:14:03.267869 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.267856 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.267927 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.267880 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 01:14:03.268539 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.268523 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasSufficientMemory" Jul 17 01:14:03.268603 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.268552 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 01:14:03.268603 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.268565 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeHasSufficientPID" Jul 17 01:14:03.285673 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.285651 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-44.ec2.internal\" not found" node="ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.289951 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.289934 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-44.ec2.internal\" not found" node="ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.291977 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.291963 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-44.ec2.internal\" not found" Jul 17 01:14:03.392751 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.392724 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-44.ec2.internal\" not found" Jul 17 01:14:03.406061 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.406033 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/031589d9c1527982687d362e9909b534-config\") pod \"kube-apiserver-proxy-ip-10-0-128-44.ec2.internal\" (UID: \"031589d9c1527982687d362e9909b534\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.406203 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.406062 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a6a1b43738adc5a0d7ebc1b414906ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal\" (UID: \"7a6a1b43738adc5a0d7ebc1b414906ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.406203 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.406113 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a6a1b43738adc5a0d7ebc1b414906ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal\" (UID: \"7a6a1b43738adc5a0d7ebc1b414906ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.493666 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.493446 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-44.ec2.internal\" not found" Jul 17 01:14:03.506785 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.506764 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/031589d9c1527982687d362e9909b534-config\") pod \"kube-apiserver-proxy-ip-10-0-128-44.ec2.internal\" (UID: \"031589d9c1527982687d362e9909b534\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.506861 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.506804 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a6a1b43738adc5a0d7ebc1b414906ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal\" (UID: \"7a6a1b43738adc5a0d7ebc1b414906ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.506861 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.506821 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a6a1b43738adc5a0d7ebc1b414906ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal\" (UID: \"7a6a1b43738adc5a0d7ebc1b414906ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.506962 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.506861 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a6a1b43738adc5a0d7ebc1b414906ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal\" (UID: \"7a6a1b43738adc5a0d7ebc1b414906ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.506962 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.506886 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/031589d9c1527982687d362e9909b534-config\") pod \"kube-apiserver-proxy-ip-10-0-128-44.ec2.internal\" (UID: \"031589d9c1527982687d362e9909b534\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.506962 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.506903 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a6a1b43738adc5a0d7ebc1b414906ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal\" (UID: \"7a6a1b43738adc5a0d7ebc1b414906ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.588963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.588921 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.594110 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.594071 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-44.ec2.internal\" not found" Jul 17 01:14:03.594110 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.594092 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal" Jul 17 01:14:03.698063 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.698026 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-44.ec2.internal\" not found" Jul 17 01:14:03.799658 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.799594 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-44.ec2.internal\" not found" Jul 17 01:14:03.900729 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:03.900683 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-44.ec2.internal\" not found" Jul 17 01:14:03.914474 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.914445 2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 01:14:03.942507 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.942472 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 01:14:03.942980 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.942609 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 01:14:03.942980 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.942664 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 01:14:03.942980 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.942680 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 01:14:03.985456 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.985415 2583 apiserver.go:52] "Watching apiserver" Jul 17 01:14:03.998344 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.998299 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 01:09:03 +0000 UTC" deadline="2028-04-09 09:29:22.284394472 +0000 UTC" Jul 17 01:14:03.998344 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:03.998343 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15176h15m18.286055276s" Jul 17 01:14:04.004716 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.004686 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 01:14:04.004716 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.004704 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-44.ec2.internal" Jul 17 01:14:04.004866 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.004794 2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 01:14:04.005711 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.005688 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-59v85","kube-system/konnectivity-agent-g4pc2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv","openshift-multus/multus-additional-cni-plugins-w6dvf","openshift-multus/multus-kmk5c","openshift-network-diagnostics/network-check-target-hjgm7","openshift-network-operator/iptables-alerter-qprdv","openshift-ovn-kubernetes/ovnkube-node-2w6g5","openshift-cluster-node-tuning-operator/tuned-rxqjf","openshift-dns/node-resolver-pzn6s","openshift-image-registry/node-ca-lnzqz","openshift-multus/network-metrics-daemon-9bx9f"] Jul 17 01:14:04.007525 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.007507 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:04.007602 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.007569 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:04.009295 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.009269 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g4pc2" Jul 17 01:14:04.010566 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.010546 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.012610 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.011959 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8g8dr\"" Jul 17 01:14:04.012610 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.012044 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 01:14:04.012779 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.012756 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gs4w4\"" Jul 17 01:14:04.012838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.012789 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 01:14:04.013549 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.013009 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 01:14:04.013549 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.013061 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 01:14:04.013549 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.013318 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.014817 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.014418 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.014817 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.014469 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 01:14:04.015459 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.015441 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 01:14:04.015538 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.015508 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal" Jul 17 01:14:04.015808 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.015791 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 01:14:04.016001 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.015980 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 01:14:04.016060 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.016007 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fng76\"" Jul 17 01:14:04.016117 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.016087 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 01:14:04.016408 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.016393 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 01:14:04.016483 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.016431 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 01:14:04.016624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.016602 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:04.016734 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.016656 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:04.018024 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.018002 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qprdv" Jul 17 01:14:04.019307 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.019286 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bkbm8\"" Jul 17 01:14:04.019424 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.019326 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 01:14:04.019580 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.019562 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 01:14:04.019651 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.019631 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.020264 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.020247 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 01:14:04.020360 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.020276 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5mc4j\"" Jul 17 01:14:04.020444 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.020420 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 01:14:04.020524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.020506 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 01:14:04.021242 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.021227 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.022934 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.022917 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pzn6s" Jul 17 01:14:04.023017 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.022964 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 01:14:04.023653 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.023636 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 01:14:04.023757 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.023740 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 01:14:04.023876 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.023858 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kkd9c\"" Jul 17 01:14:04.023955 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.023858 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 01:14:04.024825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.024805 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lnzqz" Jul 17 01:14:04.025114 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.024907 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 01:14:04.025114 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.024974 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 01:14:04.025260 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.025248 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 01:14:04.025324 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.025265 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qxp8x\"" Jul 17 01:14:04.025324 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.025280 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 01:14:04.025489 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.025465 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 01:14:04.025555 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.025510 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 01:14:04.025790 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.025776 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qwh4t\"" Jul 17 01:14:04.026685 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.026668 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-44.ec2.internal"] Jul 17 01:14:04.026753 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.026736 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:04.026800 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.026783 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:04.027764 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.027742 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 01:14:04.027764 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.027755 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ccd26\"" Jul 17 01:14:04.027924 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.027753 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 01:14:04.027924 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.027753 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 01:14:04.029216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.029197 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 01:14:04.029503 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.029485 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal"] Jul 17 01:14:04.038700 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.038683 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hqnxr" Jul 17 01:14:04.047739 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.047721 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hqnxr" Jul 17 01:14:04.108485 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.107839 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 01:14:04.109882 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.109852 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566vs\" (UniqueName: \"kubernetes.io/projected/2f6f658c-77f8-428a-8bb3-2eb42fa68315-kube-api-access-566vs\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.110010 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.109892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-dbus\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:04.110010 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.109921 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-slash\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.110010 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.109946 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f6f658c-77f8-428a-8bb3-2eb42fa68315-tmp\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.110010 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.109972 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa0e084d-6d85-492d-a50a-cf1e654d04a9-konnectivity-ca\") pod \"konnectivity-agent-g4pc2\" (UID: \"aa0e084d-6d85-492d-a50a-cf1e654d04a9\") " pod="kube-system/konnectivity-agent-g4pc2" Jul 17 01:14:04.110190 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110011 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f52d2f1-687d-4203-9c24-788062736e12-cni-binary-copy\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.110190 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110061 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a-host-slash\") pod \"iptables-alerter-qprdv\" (UID: \"12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a\") " pod="openshift-network-operator/iptables-alerter-qprdv" Jul 17 01:14:04.110190 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110097 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-run-systemd\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.110190 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110154 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-node-log\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.110190 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110179 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-log-socket\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.110398 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110203 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba8f230b-ad43-4811-9a71-a33c5fb4a918-env-overrides\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.110398 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110225 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwq9f\" (UniqueName: \"kubernetes.io/projected/ba8f230b-ad43-4811-9a71-a33c5fb4a918-kube-api-access-pwq9f\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.110398 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110243 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-run-netns\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.110398 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110270 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-multus-conf-dir\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.110398 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110286 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba8f230b-ad43-4811-9a71-a33c5fb4a918-ovnkube-script-lib\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.110398 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110318 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-run-multus-certs\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.110398 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110342 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-etc-kubernetes\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.110398 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110357 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvdnm\" (UniqueName: \"kubernetes.io/projected/5f52d2f1-687d-4203-9c24-788062736e12-kube-api-access-jvdnm\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.110398 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110374 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-sysconfig\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.110703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110411 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-kubernetes\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.110703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110432 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-host\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.110703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110449 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f571d61-7f50-41c1-aabc-19b67211094d-tmp-dir\") pod \"node-resolver-pzn6s\" (UID: \"2f571d61-7f50-41c1-aabc-19b67211094d\") " pod="openshift-dns/node-resolver-pzn6s" Jul 17 01:14:04.110703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110469 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba8f230b-ad43-4811-9a71-a33c5fb4a918-ovnkube-config\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.110703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110509 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-os-release\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.110703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110555 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-var-lib-cni-multus\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.110703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110582 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-run\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.110703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110607 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:04.110703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110631 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.110703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110666 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.110703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110695 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6fafa6fe-e9ba-4f2e-8043-8e3413afa6de-host\") pod \"node-ca-lnzqz\" (UID: \"6fafa6fe-e9ba-4f2e-8043-8e3413afa6de\") " pod="openshift-image-registry/node-ca-lnzqz" Jul 17 01:14:04.111064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110718 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6fafa6fe-e9ba-4f2e-8043-8e3413afa6de-serviceca\") pod \"node-ca-lnzqz\" (UID: \"6fafa6fe-e9ba-4f2e-8043-8e3413afa6de\") " pod="openshift-image-registry/node-ca-lnzqz" Jul 17 01:14:04.111064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110743 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-system-cni-dir\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.111064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110779 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.111064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110815 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84wqn\" (UniqueName: \"kubernetes.io/projected/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-kube-api-access-84wqn\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.111064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110841 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-sys\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.111064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110863 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-run-netns\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.111064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110886 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-run-openvswitch\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.111064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110912 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-system-cni-dir\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.111064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110959 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-multus-cni-dir\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.111064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.110979 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlwvk\" (UniqueName: \"kubernetes.io/projected/2f571d61-7f50-41c1-aabc-19b67211094d-kube-api-access-xlwvk\") pod \"node-resolver-pzn6s\" (UID: \"2f571d61-7f50-41c1-aabc-19b67211094d\") " pod="openshift-dns/node-resolver-pzn6s" Jul 17 01:14:04.111064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111006 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx49r\" (UniqueName: \"kubernetes.io/projected/8dbc5f6f-120c-42f2-b27f-5644ab623431-kube-api-access-fx49r\") pod \"network-metrics-daemon-9bx9f\" (UID: \"8dbc5f6f-120c-42f2-b27f-5644ab623431\") " pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:04.111064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111041 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-kubelet-config\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111073 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-tuned\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111122 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-registration-dir\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111147 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-sys-fs\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111168 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-882bp\" (UniqueName: \"kubernetes.io/projected/b1c00aee-257f-4211-bc8a-44d4dd710c77-kube-api-access-882bp\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111193 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111218 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-kubelet\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111240 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-systemd-units\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111265 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111282 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa0e084d-6d85-492d-a50a-cf1e654d04a9-agent-certs\") pod \"konnectivity-agent-g4pc2\" (UID: \"aa0e084d-6d85-492d-a50a-cf1e654d04a9\") " pod="kube-system/konnectivity-agent-g4pc2" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111311 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f52d2f1-687d-4203-9c24-788062736e12-multus-daemon-config\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111334 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-modprobe-d\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111383 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-lib-modules\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111424 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hzfm\" (UniqueName: \"kubernetes.io/projected/12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a-kube-api-access-7hzfm\") pod \"iptables-alerter-qprdv\" (UID: \"12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a\") " pod="openshift-network-operator/iptables-alerter-qprdv" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111470 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-cnibin\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111507 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p9dt\" (UniqueName: \"kubernetes.io/projected/6fafa6fe-e9ba-4f2e-8043-8e3413afa6de-kube-api-access-4p9dt\") pod \"node-ca-lnzqz\" (UID: \"6fafa6fe-e9ba-4f2e-8043-8e3413afa6de\") " pod="openshift-image-registry/node-ca-lnzqz" Jul 17 01:14:04.111608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111532 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs\") pod \"network-metrics-daemon-9bx9f\" (UID: \"8dbc5f6f-120c-42f2-b27f-5644ab623431\") " pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111572 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-socket-dir\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111592 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-run-ovn-kubernetes\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111617 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f571d61-7f50-41c1-aabc-19b67211094d-hosts-file\") pod \"node-resolver-pzn6s\" (UID: \"2f571d61-7f50-41c1-aabc-19b67211094d\") " pod="openshift-dns/node-resolver-pzn6s" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111644 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-device-dir\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111685 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-var-lib-kubelet\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111714 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-os-release\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111739 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-etc-openvswitch\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111764 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-cni-bin\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111795 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111835 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvtbh\" (UniqueName: \"kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh\") pod \"network-check-target-hjgm7\" (UID: \"5293dc76-7ef2-442f-9035-6403edb5a336\") " pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111881 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-run-k8s-cni-cncf-io\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111910 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-systemd\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111934 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-etc-selinux\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111959 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-cnibin\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.111988 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-var-lib-openvswitch\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.112275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.112010 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba8f230b-ad43-4811-9a71-a33c5fb4a918-ovn-node-metrics-cert\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.113026 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.112033 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-var-lib-cni-bin\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.113026 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.112082 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-var-lib-kubelet\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.113026 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.112099 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-hostroot\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.113026 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.112115 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-sysctl-d\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.113026 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.112130 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a-iptables-alerter-script\") pod \"iptables-alerter-qprdv\" (UID: \"12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a\") " pod="openshift-network-operator/iptables-alerter-qprdv" Jul 17 01:14:04.113026 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.112172 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-run-ovn\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.113026 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.112191 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-cni-netd\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.113026 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.112207 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-multus-socket-dir-parent\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.113026 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.112222 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-sysctl-conf\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.168428 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:04.168390 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6a1b43738adc5a0d7ebc1b414906ee.slice/crio-ed07d12e9a09321980d5ab4bad4664679c18139bc4d387e808ae8281152ed97b WatchSource:0}: Error finding container ed07d12e9a09321980d5ab4bad4664679c18139bc4d387e808ae8281152ed97b: Status 404 returned error can't find the container with id ed07d12e9a09321980d5ab4bad4664679c18139bc4d387e808ae8281152ed97b Jul 17 01:14:04.174372 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.174355 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 01:14:04.213356 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213329 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-var-lib-kubelet\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.213356 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213361 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-os-release\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.213581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213379 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-etc-openvswitch\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.213581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213394 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-cni-bin\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.213581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213416 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.213581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213444 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtbh\" (UniqueName: \"kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh\") pod \"network-check-target-hjgm7\" (UID: \"5293dc76-7ef2-442f-9035-6403edb5a336\") " pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:04.213581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213491 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-var-lib-kubelet\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.213581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213492 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-os-release\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.213581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213493 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-cni-bin\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.213581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213502 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-run-k8s-cni-cncf-io\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.213581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213496 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-etc-openvswitch\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.213581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213576 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-run-k8s-cni-cncf-io\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.213581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213575 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-systemd\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213608 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-etc-selinux\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213619 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-systemd\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213630 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-cnibin\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213647 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-var-lib-openvswitch\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213665 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba8f230b-ad43-4811-9a71-a33c5fb4a918-ovn-node-metrics-cert\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213689 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-var-lib-cni-bin\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213705 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-var-lib-kubelet\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213702 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-cnibin\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-hostroot\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213744 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-etc-selinux\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213770 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-var-lib-openvswitch\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213788 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-hostroot\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213742 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-sysctl-d\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213774 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-var-lib-cni-bin\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213803 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-var-lib-kubelet\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213832 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a-iptables-alerter-script\") pod \"iptables-alerter-qprdv\" (UID: \"12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a\") " pod="openshift-network-operator/iptables-alerter-qprdv" Jul 17 01:14:04.214104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213856 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-run-ovn\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213894 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-cni-netd\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213920 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-multus-socket-dir-parent\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213935 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-sysctl-d\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213946 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-sysctl-conf\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213981 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-cni-netd\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.213982 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-run-ovn\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214019 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566vs\" (UniqueName: \"kubernetes.io/projected/2f6f658c-77f8-428a-8bb3-2eb42fa68315-kube-api-access-566vs\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214041 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-dbus\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214066 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-slash\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214076 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-sysctl-conf\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214081 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-multus-socket-dir-parent\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214091 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f6f658c-77f8-428a-8bb3-2eb42fa68315-tmp\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214155 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa0e084d-6d85-492d-a50a-cf1e654d04a9-konnectivity-ca\") pod \"konnectivity-agent-g4pc2\" (UID: \"aa0e084d-6d85-492d-a50a-cf1e654d04a9\") " pod="kube-system/konnectivity-agent-g4pc2" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214154 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214177 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f52d2f1-687d-4203-9c24-788062736e12-cni-binary-copy\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214181 2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214161 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-slash\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.214905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214201 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a-host-slash\") pod \"iptables-alerter-qprdv\" (UID: \"12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a\") " pod="openshift-network-operator/iptables-alerter-qprdv" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214222 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-dbus\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214236 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-run-systemd\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214264 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-node-log\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214238 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a-host-slash\") pod \"iptables-alerter-qprdv\" (UID: \"12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a\") " pod="openshift-network-operator/iptables-alerter-qprdv" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214288 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-log-socket\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214322 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba8f230b-ad43-4811-9a71-a33c5fb4a918-env-overrides\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214326 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-log-socket\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214399 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-node-log\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214505 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a-iptables-alerter-script\") pod \"iptables-alerter-qprdv\" (UID: \"12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a\") " pod="openshift-network-operator/iptables-alerter-qprdv" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214354 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwq9f\" (UniqueName: \"kubernetes.io/projected/ba8f230b-ad43-4811-9a71-a33c5fb4a918-kube-api-access-pwq9f\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214573 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-run-netns\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214598 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-multus-conf-dir\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214623 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba8f230b-ad43-4811-9a71-a33c5fb4a918-ovnkube-script-lib\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214638 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-run-systemd\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214646 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-run-multus-certs\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214681 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-etc-kubernetes\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214683 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-run-multus-certs\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.216003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214722 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-run-netns\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvdnm\" (UniqueName: \"kubernetes.io/projected/5f52d2f1-687d-4203-9c24-788062736e12-kube-api-access-jvdnm\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214771 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-etc-kubernetes\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214774 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-sysconfig\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214804 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-kubernetes\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214808 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-sysconfig\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214848 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f52d2f1-687d-4203-9c24-788062736e12-cni-binary-copy\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214853 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba8f230b-ad43-4811-9a71-a33c5fb4a918-env-overrides\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214856 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa0e084d-6d85-492d-a50a-cf1e654d04a9-konnectivity-ca\") pod \"konnectivity-agent-g4pc2\" (UID: \"aa0e084d-6d85-492d-a50a-cf1e654d04a9\") " pod="kube-system/konnectivity-agent-g4pc2" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214858 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-host\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214902 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-multus-conf-dir\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214905 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f571d61-7f50-41c1-aabc-19b67211094d-tmp-dir\") pod \"node-resolver-pzn6s\" (UID: \"2f571d61-7f50-41c1-aabc-19b67211094d\") " pod="openshift-dns/node-resolver-pzn6s" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214944 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba8f230b-ad43-4811-9a71-a33c5fb4a918-ovnkube-config\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214971 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-os-release\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.214999 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-var-lib-cni-multus\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215027 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-run\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215052 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-os-release\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215056 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:04.216892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215084 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215147 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215195 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6fafa6fe-e9ba-4f2e-8043-8e3413afa6de-host\") pod \"node-ca-lnzqz\" (UID: \"6fafa6fe-e9ba-4f2e-8043-8e3413afa6de\") " pod="openshift-image-registry/node-ca-lnzqz" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215198 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f571d61-7f50-41c1-aabc-19b67211094d-tmp-dir\") pod \"node-resolver-pzn6s\" (UID: \"2f571d61-7f50-41c1-aabc-19b67211094d\") " pod="openshift-dns/node-resolver-pzn6s" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.215226 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215279 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-run\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.215333 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret podName:71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a nodeName:}" failed. No retries permitted until 2026-07-17 01:14:04.715268509 +0000 UTC m=+2.110883346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret") pod "global-pull-secret-syncer-59v85" (UID: "71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a") : object "kube-system"/"original-pull-secret" not registered Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215225 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6fafa6fe-e9ba-4f2e-8043-8e3413afa6de-serviceca\") pod \"node-ca-lnzqz\" (UID: \"6fafa6fe-e9ba-4f2e-8043-8e3413afa6de\") " pod="openshift-image-registry/node-ca-lnzqz" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215375 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215393 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-system-cni-dir\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215422 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215332 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-host\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215476 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84wqn\" (UniqueName: \"kubernetes.io/projected/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-kube-api-access-84wqn\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215504 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-sys\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215531 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-run-netns\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215579 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-kubernetes\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215533 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba8f230b-ad43-4811-9a71-a33c5fb4a918-ovnkube-script-lib\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.217838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215581 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-run-openvswitch\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215645 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-system-cni-dir\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215674 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-multus-cni-dir\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215682 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-run-openvswitch\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215702 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwvk\" (UniqueName: \"kubernetes.io/projected/2f571d61-7f50-41c1-aabc-19b67211094d-kube-api-access-xlwvk\") pod \"node-resolver-pzn6s\" (UID: \"2f571d61-7f50-41c1-aabc-19b67211094d\") " pod="openshift-dns/node-resolver-pzn6s" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215720 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6fafa6fe-e9ba-4f2e-8043-8e3413afa6de-serviceca\") pod \"node-ca-lnzqz\" (UID: \"6fafa6fe-e9ba-4f2e-8043-8e3413afa6de\") " pod="openshift-image-registry/node-ca-lnzqz" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215732 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx49r\" (UniqueName: \"kubernetes.io/projected/8dbc5f6f-120c-42f2-b27f-5644ab623431-kube-api-access-fx49r\") pod \"network-metrics-daemon-9bx9f\" (UID: \"8dbc5f6f-120c-42f2-b27f-5644ab623431\") " pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215743 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-kubelet-config\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215761 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba8f230b-ad43-4811-9a71-a33c5fb4a918-ovnkube-config\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215810 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-kubelet-config\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215811 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6fafa6fe-e9ba-4f2e-8043-8e3413afa6de-host\") pod \"node-ca-lnzqz\" (UID: \"6fafa6fe-e9ba-4f2e-8043-8e3413afa6de\") " pod="openshift-image-registry/node-ca-lnzqz" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215831 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-system-cni-dir\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215856 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-sys\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215882 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-host-var-lib-cni-multus\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215906 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-system-cni-dir\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215940 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-tuned\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.215999 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-registration-dir\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.218431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216035 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-multus-cni-dir\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216060 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-sys-fs\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216081 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-registration-dir\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216089 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-run-netns\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216127 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882bp\" (UniqueName: \"kubernetes.io/projected/b1c00aee-257f-4211-bc8a-44d4dd710c77-kube-api-access-882bp\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216140 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-sys-fs\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216198 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216234 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-cni-binary-copy\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216335 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-kubelet\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216437 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-systemd-units\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216502 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216528 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa0e084d-6d85-492d-a50a-cf1e654d04a9-agent-certs\") pod \"konnectivity-agent-g4pc2\" (UID: \"aa0e084d-6d85-492d-a50a-cf1e654d04a9\") " pod="kube-system/konnectivity-agent-g4pc2" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216547 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-systemd-units\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216558 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f52d2f1-687d-4203-9c24-788062736e12-multus-daemon-config\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-modprobe-d\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216627 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-lib-modules\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216676 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hzfm\" (UniqueName: \"kubernetes.io/projected/12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a-kube-api-access-7hzfm\") pod \"iptables-alerter-qprdv\" (UID: \"12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a\") " pod="openshift-network-operator/iptables-alerter-qprdv" Jul 17 01:14:04.218952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216452 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-kubelet\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216730 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-cnibin\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216823 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-modprobe-d\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216832 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f6f658c-77f8-428a-8bb3-2eb42fa68315-lib-modules\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216828 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216862 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p9dt\" (UniqueName: \"kubernetes.io/projected/6fafa6fe-e9ba-4f2e-8043-8e3413afa6de-kube-api-access-4p9dt\") pod \"node-ca-lnzqz\" (UID: \"6fafa6fe-e9ba-4f2e-8043-8e3413afa6de\") " pod="openshift-image-registry/node-ca-lnzqz" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216876 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f52d2f1-687d-4203-9c24-788062736e12-cnibin\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216896 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs\") pod \"network-metrics-daemon-9bx9f\" (UID: \"8dbc5f6f-120c-42f2-b27f-5644ab623431\") " pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216930 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-socket-dir\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216960 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-run-ovn-kubernetes\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216989 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f571d61-7f50-41c1-aabc-19b67211094d-hosts-file\") pod \"node-resolver-pzn6s\" (UID: \"2f571d61-7f50-41c1-aabc-19b67211094d\") " pod="openshift-dns/node-resolver-pzn6s" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216999 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f52d2f1-687d-4203-9c24-788062736e12-multus-daemon-config\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.216960 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.217061 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba8f230b-ad43-4811-9a71-a33c5fb4a918-host-run-ovn-kubernetes\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.217065 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f571d61-7f50-41c1-aabc-19b67211094d-hosts-file\") pod \"node-resolver-pzn6s\" (UID: \"2f571d61-7f50-41c1-aabc-19b67211094d\") " pod="openshift-dns/node-resolver-pzn6s" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.217098 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-device-dir\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.217169 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-socket-dir\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.217202 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1c00aee-257f-4211-bc8a-44d4dd710c77-device-dir\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.219464 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.217391 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 01:14:04.219969 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.217453 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs podName:8dbc5f6f-120c-42f2-b27f-5644ab623431 nodeName:}" failed. No retries permitted until 2026-07-17 01:14:04.717434766 +0000 UTC m=+2.113049600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs") pod "network-metrics-daemon-9bx9f" (UID: "8dbc5f6f-120c-42f2-b27f-5644ab623431") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 01:14:04.219969 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.218046 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f6f658c-77f8-428a-8bb3-2eb42fa68315-tmp\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.219969 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.218253 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba8f230b-ad43-4811-9a71-a33c5fb4a918-ovn-node-metrics-cert\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.219969 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.218304 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f6f658c-77f8-428a-8bb3-2eb42fa68315-etc-tuned\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.219969 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.219287 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa0e084d-6d85-492d-a50a-cf1e654d04a9-agent-certs\") pod \"konnectivity-agent-g4pc2\" (UID: \"aa0e084d-6d85-492d-a50a-cf1e654d04a9\") " pod="kube-system/konnectivity-agent-g4pc2" Jul 17 01:14:04.222855 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.222835 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 01:14:04.222907 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.222859 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 01:14:04.222907 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.222869 2583 projected.go:196] Error preparing data for projected volume kube-api-access-cvtbh for pod openshift-network-diagnostics/network-check-target-hjgm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 01:14:04.222969 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.222926 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh podName:5293dc76-7ef2-442f-9035-6403edb5a336 nodeName:}" failed. No retries permitted until 2026-07-17 01:14:04.72291031 +0000 UTC m=+2.118525155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cvtbh" (UniqueName: "kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh") pod "network-check-target-hjgm7" (UID: "5293dc76-7ef2-442f-9035-6403edb5a336") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 01:14:04.230509 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.230176 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlwvk\" (UniqueName: \"kubernetes.io/projected/2f571d61-7f50-41c1-aabc-19b67211094d-kube-api-access-xlwvk\") pod \"node-resolver-pzn6s\" (UID: \"2f571d61-7f50-41c1-aabc-19b67211094d\") " pod="openshift-dns/node-resolver-pzn6s" Jul 17 01:14:04.230509 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.230278 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-566vs\" (UniqueName: \"kubernetes.io/projected/2f6f658c-77f8-428a-8bb3-2eb42fa68315-kube-api-access-566vs\") pod \"tuned-rxqjf\" (UID: \"2f6f658c-77f8-428a-8bb3-2eb42fa68315\") " pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.230509 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.230410 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvdnm\" (UniqueName: \"kubernetes.io/projected/5f52d2f1-687d-4203-9c24-788062736e12-kube-api-access-jvdnm\") pod \"multus-kmk5c\" (UID: \"5f52d2f1-687d-4203-9c24-788062736e12\") " pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.230509 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.230453 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-882bp\" (UniqueName: \"kubernetes.io/projected/b1c00aee-257f-4211-bc8a-44d4dd710c77-kube-api-access-882bp\") pod \"aws-ebs-csi-driver-node-hm8lv\" (UID: \"b1c00aee-257f-4211-bc8a-44d4dd710c77\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.230908 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.230884 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwq9f\" (UniqueName: \"kubernetes.io/projected/ba8f230b-ad43-4811-9a71-a33c5fb4a918-kube-api-access-pwq9f\") pod \"ovnkube-node-2w6g5\" (UID: \"ba8f230b-ad43-4811-9a71-a33c5fb4a918\") " pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.231010 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.230950 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84wqn\" (UniqueName: \"kubernetes.io/projected/0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a-kube-api-access-84wqn\") pod \"multus-additional-cni-plugins-w6dvf\" (UID: \"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a\") " pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.231400 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.231383 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx49r\" (UniqueName: \"kubernetes.io/projected/8dbc5f6f-120c-42f2-b27f-5644ab623431-kube-api-access-fx49r\") pod \"network-metrics-daemon-9bx9f\" (UID: \"8dbc5f6f-120c-42f2-b27f-5644ab623431\") " pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:04.232870 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.232840 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hzfm\" (UniqueName: \"kubernetes.io/projected/12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a-kube-api-access-7hzfm\") pod \"iptables-alerter-qprdv\" (UID: \"12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a\") " pod="openshift-network-operator/iptables-alerter-qprdv" Jul 17 01:14:04.233427 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.233408 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p9dt\" (UniqueName: \"kubernetes.io/projected/6fafa6fe-e9ba-4f2e-8043-8e3413afa6de-kube-api-access-4p9dt\") pod \"node-ca-lnzqz\" (UID: \"6fafa6fe-e9ba-4f2e-8043-8e3413afa6de\") " pod="openshift-image-registry/node-ca-lnzqz" Jul 17 01:14:04.350670 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.350579 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g4pc2" Jul 17 01:14:04.356320 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.356301 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" Jul 17 01:14:04.360638 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:04.360600 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0e084d_6d85_492d_a50a_cf1e654d04a9.slice/crio-50a5b6d7ee35285da1cd520f137d6a95233dbd4d5cb776f98d2abf08ef1cd0da WatchSource:0}: Error finding container 50a5b6d7ee35285da1cd520f137d6a95233dbd4d5cb776f98d2abf08ef1cd0da: Status 404 returned error can't find the container with id 50a5b6d7ee35285da1cd520f137d6a95233dbd4d5cb776f98d2abf08ef1cd0da Jul 17 01:14:04.365031 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:04.365002 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c00aee_257f_4211_bc8a_44d4dd710c77.slice/crio-269647ab5580e0956ed5df16fc83d28e77e0e8e0d06762fc000a74bc43e41d1d WatchSource:0}: Error finding container 269647ab5580e0956ed5df16fc83d28e77e0e8e0d06762fc000a74bc43e41d1d: Status 404 returned error can't find the container with id 269647ab5580e0956ed5df16fc83d28e77e0e8e0d06762fc000a74bc43e41d1d Jul 17 01:14:04.369231 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.369193 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w6dvf" Jul 17 01:14:04.373580 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.373559 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kmk5c" Jul 17 01:14:04.376754 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:04.376730 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e763b7c_1b22_4c5e_8dd2_1f06ad72be6a.slice/crio-f3461c5d90e3f89b466c4a462e876a5a2f1b57ab66b873769b666d9fc0c7c52e WatchSource:0}: Error finding container f3461c5d90e3f89b466c4a462e876a5a2f1b57ab66b873769b666d9fc0c7c52e: Status 404 returned error can't find the container with id f3461c5d90e3f89b466c4a462e876a5a2f1b57ab66b873769b666d9fc0c7c52e Jul 17 01:14:04.381609 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:04.381586 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f52d2f1_687d_4203_9c24_788062736e12.slice/crio-04c4829872e09bf04255da91d3ed7716e51cf45cf96df1d7343ce5fec72c3cde WatchSource:0}: Error finding container 04c4829872e09bf04255da91d3ed7716e51cf45cf96df1d7343ce5fec72c3cde: Status 404 returned error can't find the container with id 04c4829872e09bf04255da91d3ed7716e51cf45cf96df1d7343ce5fec72c3cde Jul 17 01:14:04.402623 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.402598 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qprdv" Jul 17 01:14:04.410652 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:04.410615 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a4e0ba_2c69_4ee1_ab15_3dbaf09b5a6a.slice/crio-9c78758693e545355038a6c24d5675d2fde77e27404fa219368104938c14c3fb WatchSource:0}: Error finding container 9c78758693e545355038a6c24d5675d2fde77e27404fa219368104938c14c3fb: Status 404 returned error can't find the container with id 9c78758693e545355038a6c24d5675d2fde77e27404fa219368104938c14c3fb Jul 17 01:14:04.411859 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.411787 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:04.420241 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:04.420213 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8f230b_ad43_4811_9a71_a33c5fb4a918.slice/crio-dc67baa268f98d237628b6e1e9f1c1a3e1f70900ea5733126b7086d84b630c2a WatchSource:0}: Error finding container dc67baa268f98d237628b6e1e9f1c1a3e1f70900ea5733126b7086d84b630c2a: Status 404 returned error can't find the container with id dc67baa268f98d237628b6e1e9f1c1a3e1f70900ea5733126b7086d84b630c2a Jul 17 01:14:04.423408 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.423392 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" Jul 17 01:14:04.430871 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:04.430848 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6f658c_77f8_428a_8bb3_2eb42fa68315.slice/crio-b38a6540d50badb12f8105ff534950d9f0b9177b44ec3e4a30e0aafc5d96a4e0 WatchSource:0}: Error finding container b38a6540d50badb12f8105ff534950d9f0b9177b44ec3e4a30e0aafc5d96a4e0: Status 404 returned error can't find the container with id b38a6540d50badb12f8105ff534950d9f0b9177b44ec3e4a30e0aafc5d96a4e0 Jul 17 01:14:04.440554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.440535 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pzn6s" Jul 17 01:14:04.445962 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.445941 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lnzqz" Jul 17 01:14:04.448196 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:04.448159 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f571d61_7f50_41c1_aabc_19b67211094d.slice/crio-83abcff196c532539e06689ba7d680020d3e7edefd7f9739187a891a076d5c98 WatchSource:0}: Error finding container 83abcff196c532539e06689ba7d680020d3e7edefd7f9739187a891a076d5c98: Status 404 returned error can't find the container with id 83abcff196c532539e06689ba7d680020d3e7edefd7f9739187a891a076d5c98 Jul 17 01:14:04.454161 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:04.454140 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fafa6fe_e9ba_4f2e_8043_8e3413afa6de.slice/crio-51924d6b9ed3cbbfa85a05c2ff1165f72c530359791378ad0306e53a324eb558 WatchSource:0}: Error finding container 51924d6b9ed3cbbfa85a05c2ff1165f72c530359791378ad0306e53a324eb558: Status 404 returned error can't find the container with id 51924d6b9ed3cbbfa85a05c2ff1165f72c530359791378ad0306e53a324eb558 Jul 17 01:14:04.532469 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.532444 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 01:14:04.721230 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.721145 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:04.721230 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.721191 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs\") pod \"network-metrics-daemon-9bx9f\" (UID: \"8dbc5f6f-120c-42f2-b27f-5644ab623431\") " pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:04.721397 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.721292 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 01:14:04.721397 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.721353 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret podName:71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a nodeName:}" failed. No retries permitted until 2026-07-17 01:14:05.721336752 +0000 UTC m=+3.116951583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret") pod "global-pull-secret-syncer-59v85" (UID: "71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a") : object "kube-system"/"original-pull-secret" not registered Jul 17 01:14:04.721397 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.721297 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 01:14:04.721571 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.721428 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs podName:8dbc5f6f-120c-42f2-b27f-5644ab623431 nodeName:}" failed. No retries permitted until 2026-07-17 01:14:05.721415193 +0000 UTC m=+3.117030023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs") pod "network-metrics-daemon-9bx9f" (UID: "8dbc5f6f-120c-42f2-b27f-5644ab623431") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 01:14:04.744825 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:04.744792 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031589d9c1527982687d362e9909b534.slice/crio-f8b723182b044ccad330b6966e77552dd94923b3f1895387079bea7e74425455 WatchSource:0}: Error finding container f8b723182b044ccad330b6966e77552dd94923b3f1895387079bea7e74425455: Status 404 returned error can't find the container with id f8b723182b044ccad330b6966e77552dd94923b3f1895387079bea7e74425455 Jul 17 01:14:04.784229 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.784200 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 01:14:04.822290 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:04.822257 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtbh\" (UniqueName: \"kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh\") pod \"network-check-target-hjgm7\" (UID: \"5293dc76-7ef2-442f-9035-6403edb5a336\") " pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:04.822475 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.822461 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 01:14:04.822537 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.822484 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 01:14:04.822537 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.822497 2583 projected.go:196] Error preparing data for projected volume kube-api-access-cvtbh for pod openshift-network-diagnostics/network-check-target-hjgm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 01:14:04.822643 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:04.822556 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh podName:5293dc76-7ef2-442f-9035-6403edb5a336 nodeName:}" failed. No retries permitted until 2026-07-17 01:14:05.822536366 +0000 UTC m=+3.218151201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cvtbh" (UniqueName: "kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh") pod "network-check-target-hjgm7" (UID: "5293dc76-7ef2-442f-9035-6403edb5a336") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 01:14:05.056459 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.049530 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 01:09:04 +0000 UTC" deadline="2028-04-04 17:15:12.22126616 +0000 UTC" Jul 17 01:14:05.056459 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.049601 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15064h1m7.171669685s" Jul 17 01:14:05.056459 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.052357 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 01:14:05.162505 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.162477 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:05.162758 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:05.162593 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:05.170324 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.170203 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" event={"ID":"ba8f230b-ad43-4811-9a71-a33c5fb4a918","Type":"ContainerStarted","Data":"dc67baa268f98d237628b6e1e9f1c1a3e1f70900ea5733126b7086d84b630c2a"} Jul 17 01:14:05.172911 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.172880 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qprdv" event={"ID":"12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a","Type":"ContainerStarted","Data":"9c78758693e545355038a6c24d5675d2fde77e27404fa219368104938c14c3fb"} Jul 17 01:14:05.176506 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.176466 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6dvf" event={"ID":"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a","Type":"ContainerStarted","Data":"f3461c5d90e3f89b466c4a462e876a5a2f1b57ab66b873769b666d9fc0c7c52e"} Jul 17 01:14:05.183192 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.183079 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" event={"ID":"b1c00aee-257f-4211-bc8a-44d4dd710c77","Type":"ContainerStarted","Data":"269647ab5580e0956ed5df16fc83d28e77e0e8e0d06762fc000a74bc43e41d1d"} Jul 17 01:14:05.193811 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.193747 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g4pc2" event={"ID":"aa0e084d-6d85-492d-a50a-cf1e654d04a9","Type":"ContainerStarted","Data":"50a5b6d7ee35285da1cd520f137d6a95233dbd4d5cb776f98d2abf08ef1cd0da"} Jul 17 01:14:05.199623 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.199597 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-44.ec2.internal" event={"ID":"031589d9c1527982687d362e9909b534","Type":"ContainerStarted","Data":"f8b723182b044ccad330b6966e77552dd94923b3f1895387079bea7e74425455"} Jul 17 01:14:05.209633 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.209563 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lnzqz" event={"ID":"6fafa6fe-e9ba-4f2e-8043-8e3413afa6de","Type":"ContainerStarted","Data":"51924d6b9ed3cbbfa85a05c2ff1165f72c530359791378ad0306e53a324eb558"} Jul 17 01:14:05.215271 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.215237 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kmk5c" event={"ID":"5f52d2f1-687d-4203-9c24-788062736e12","Type":"ContainerStarted","Data":"04c4829872e09bf04255da91d3ed7716e51cf45cf96df1d7343ce5fec72c3cde"} Jul 17 01:14:05.223917 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.223844 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal" event={"ID":"7a6a1b43738adc5a0d7ebc1b414906ee","Type":"ContainerStarted","Data":"ed07d12e9a09321980d5ab4bad4664679c18139bc4d387e808ae8281152ed97b"} Jul 17 01:14:05.226794 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.226724 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pzn6s" event={"ID":"2f571d61-7f50-41c1-aabc-19b67211094d","Type":"ContainerStarted","Data":"83abcff196c532539e06689ba7d680020d3e7edefd7f9739187a891a076d5c98"} Jul 17 01:14:05.232884 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.232837 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" event={"ID":"2f6f658c-77f8-428a-8bb3-2eb42fa68315","Type":"ContainerStarted","Data":"b38a6540d50badb12f8105ff534950d9f0b9177b44ec3e4a30e0aafc5d96a4e0"} Jul 17 01:14:05.583691 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.583460 2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 01:14:05.731643 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.731539 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:05.731643 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.731621 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs\") pod \"network-metrics-daemon-9bx9f\" (UID: \"8dbc5f6f-120c-42f2-b27f-5644ab623431\") " pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:05.731878 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:05.731793 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 01:14:05.731878 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:05.731872 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs podName:8dbc5f6f-120c-42f2-b27f-5644ab623431 nodeName:}" failed. No retries permitted until 2026-07-17 01:14:07.731850169 +0000 UTC m=+5.127465021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs") pod "network-metrics-daemon-9bx9f" (UID: "8dbc5f6f-120c-42f2-b27f-5644ab623431") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 01:14:05.732296 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:05.732119 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 01:14:05.732296 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:05.732232 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret podName:71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a nodeName:}" failed. No retries permitted until 2026-07-17 01:14:07.732199957 +0000 UTC m=+5.127814813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret") pod "global-pull-secret-syncer-59v85" (UID: "71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a") : object "kube-system"/"original-pull-secret" not registered Jul 17 01:14:05.832780 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:05.832628 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtbh\" (UniqueName: \"kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh\") pod \"network-check-target-hjgm7\" (UID: \"5293dc76-7ef2-442f-9035-6403edb5a336\") " pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:05.832977 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:05.832830 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 01:14:05.832977 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:05.832849 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 01:14:05.832977 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:05.832859 2583 projected.go:196] Error preparing data for projected volume kube-api-access-cvtbh for pod openshift-network-diagnostics/network-check-target-hjgm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 01:14:05.832977 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:05.832911 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh podName:5293dc76-7ef2-442f-9035-6403edb5a336 nodeName:}" failed. No retries permitted until 2026-07-17 01:14:07.832897817 +0000 UTC m=+5.228512665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cvtbh" (UniqueName: "kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh") pod "network-check-target-hjgm7" (UID: "5293dc76-7ef2-442f-9035-6403edb5a336") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 01:14:06.050963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:06.050779 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 01:09:04 +0000 UTC" deadline="2028-01-23 00:25:18.488713752 +0000 UTC" Jul 17 01:14:06.050963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:06.050820 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13319h11m12.437898711s" Jul 17 01:14:06.162635 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:06.162594 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:06.163295 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:06.162758 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:06.163295 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:06.163244 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:06.163422 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:06.163357 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:07.164464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:07.164432 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:07.164929 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:07.164560 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:07.750145 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:07.749381 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs\") pod \"network-metrics-daemon-9bx9f\" (UID: \"8dbc5f6f-120c-42f2-b27f-5644ab623431\") " pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:07.750145 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:07.749466 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:07.750145 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:07.749590 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 01:14:07.750145 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:07.749651 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret podName:71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a nodeName:}" failed. No retries permitted until 2026-07-17 01:14:11.749633665 +0000 UTC m=+9.145248517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret") pod "global-pull-secret-syncer-59v85" (UID: "71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a") : object "kube-system"/"original-pull-secret" not registered Jul 17 01:14:07.750145 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:07.750041 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 01:14:07.750145 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:07.750090 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs podName:8dbc5f6f-120c-42f2-b27f-5644ab623431 nodeName:}" failed. No retries permitted until 2026-07-17 01:14:11.750074112 +0000 UTC m=+9.145688963 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs") pod "network-metrics-daemon-9bx9f" (UID: "8dbc5f6f-120c-42f2-b27f-5644ab623431") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 01:14:07.851476 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:07.850818 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtbh\" (UniqueName: \"kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh\") pod \"network-check-target-hjgm7\" (UID: \"5293dc76-7ef2-442f-9035-6403edb5a336\") " pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:07.851476 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:07.851002 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 01:14:07.851476 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:07.851028 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 01:14:07.851476 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:07.851040 2583 projected.go:196] Error preparing data for projected volume kube-api-access-cvtbh for pod openshift-network-diagnostics/network-check-target-hjgm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 01:14:07.851476 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:07.851116 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh podName:5293dc76-7ef2-442f-9035-6403edb5a336 nodeName:}" failed. No retries permitted until 2026-07-17 01:14:11.851081825 +0000 UTC m=+9.246696673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cvtbh" (UniqueName: "kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh") pod "network-check-target-hjgm7" (UID: "5293dc76-7ef2-442f-9035-6403edb5a336") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 01:14:08.162635 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:08.162556 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:08.162795 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:08.162556 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:08.162795 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:08.162715 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:08.162795 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:08.162766 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:09.164531 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:09.164495 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:09.164986 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:09.164625 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:10.162040 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:10.162009 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:10.162245 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:10.162166 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:10.162245 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:10.162205 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:10.162370 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:10.162275 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:11.162800 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:11.162703 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:11.163260 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:11.162835 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:11.783204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:11.783164 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:11.783401 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:11.783236 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs\") pod \"network-metrics-daemon-9bx9f\" (UID: \"8dbc5f6f-120c-42f2-b27f-5644ab623431\") " pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:11.783401 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:11.783347 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 01:14:11.783401 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:11.783378 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 01:14:11.783549 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:11.783422 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret podName:71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a nodeName:}" failed. No retries permitted until 2026-07-17 01:14:19.783400323 +0000 UTC m=+17.179015169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret") pod "global-pull-secret-syncer-59v85" (UID: "71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a") : object "kube-system"/"original-pull-secret" not registered Jul 17 01:14:11.783549 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:11.783441 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs podName:8dbc5f6f-120c-42f2-b27f-5644ab623431 nodeName:}" failed. No retries permitted until 2026-07-17 01:14:19.783430943 +0000 UTC m=+17.179045775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs") pod "network-metrics-daemon-9bx9f" (UID: "8dbc5f6f-120c-42f2-b27f-5644ab623431") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 01:14:11.884389 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:11.884348 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtbh\" (UniqueName: \"kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh\") pod \"network-check-target-hjgm7\" (UID: \"5293dc76-7ef2-442f-9035-6403edb5a336\") " pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:11.884587 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:11.884564 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 01:14:11.884587 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:11.884586 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 01:14:11.884672 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:11.884596 2583 projected.go:196] Error preparing data for projected volume kube-api-access-cvtbh for pod openshift-network-diagnostics/network-check-target-hjgm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 01:14:11.884672 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:11.884649 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh podName:5293dc76-7ef2-442f-9035-6403edb5a336 nodeName:}" failed. No retries permitted until 2026-07-17 01:14:19.884634405 +0000 UTC m=+17.280249234 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cvtbh" (UniqueName: "kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh") pod "network-check-target-hjgm7" (UID: "5293dc76-7ef2-442f-9035-6403edb5a336") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 01:14:12.162592 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:12.162518 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:12.162756 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:12.162688 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:12.163081 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:12.163061 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:12.163470 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:12.163200 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:13.163456 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:13.162990 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:13.163456 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:13.163133 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:14.162241 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:14.162207 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:14.162404 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:14.162209 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:14.162404 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:14.162344 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:14.162525 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:14.162457 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:15.162739 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:15.162689 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:15.163302 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:15.162828 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:16.162690 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:16.162653 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:16.162917 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:16.162792 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:16.162917 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:16.162869 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:16.163279 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:16.162957 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:17.162282 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:17.162245 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:17.162465 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:17.162374 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:18.162263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:18.162218 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:18.162263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:18.162265 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:18.162798 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:18.162367 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:18.162798 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:18.162673 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:19.162620 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:19.162580 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:19.163092 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:19.162713 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:19.846816 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:19.846776 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs\") pod \"network-metrics-daemon-9bx9f\" (UID: \"8dbc5f6f-120c-42f2-b27f-5644ab623431\") " pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:19.847035 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:19.846852 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:19.847035 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:19.846958 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 01:14:19.847035 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:19.846960 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 01:14:19.847035 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:19.847028 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret podName:71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a nodeName:}" failed. No retries permitted until 2026-07-17 01:14:35.847009291 +0000 UTC m=+33.242624139 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret") pod "global-pull-secret-syncer-59v85" (UID: "71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a") : object "kube-system"/"original-pull-secret" not registered Jul 17 01:14:19.847270 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:19.847047 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs podName:8dbc5f6f-120c-42f2-b27f-5644ab623431 nodeName:}" failed. No retries permitted until 2026-07-17 01:14:35.847037175 +0000 UTC m=+33.242652004 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs") pod "network-metrics-daemon-9bx9f" (UID: "8dbc5f6f-120c-42f2-b27f-5644ab623431") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 01:14:19.948114 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:19.948065 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtbh\" (UniqueName: \"kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh\") pod \"network-check-target-hjgm7\" (UID: \"5293dc76-7ef2-442f-9035-6403edb5a336\") " pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:19.948281 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:19.948236 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 01:14:19.948281 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:19.948263 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 01:14:19.948281 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:19.948274 2583 projected.go:196] Error preparing data for projected volume kube-api-access-cvtbh for pod openshift-network-diagnostics/network-check-target-hjgm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 01:14:19.948437 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:19.948335 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh podName:5293dc76-7ef2-442f-9035-6403edb5a336 nodeName:}" failed. No retries permitted until 2026-07-17 01:14:35.948321436 +0000 UTC m=+33.343936264 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cvtbh" (UniqueName: "kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh") pod "network-check-target-hjgm7" (UID: "5293dc76-7ef2-442f-9035-6403edb5a336") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 01:14:20.162214 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:20.162131 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:20.162214 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:20.162178 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:20.162409 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:20.162268 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:20.162463 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:20.162432 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:21.162555 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:21.162517 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:21.163072 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:21.162656 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:22.162503 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:22.162466 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:22.162503 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:22.162491 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:22.163056 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:22.162590 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:22.163056 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:22.162756 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:23.163454 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:23.163419 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:23.163863 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:23.163518 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:24.162150 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:24.162118 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:24.162150 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:24.162155 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:24.162391 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:24.162263 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:24.162391 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:24.162362 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:25.162185 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:25.162148 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:25.162733 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:25.162279 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:26.163000 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.162688 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:26.163641 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.162698 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:26.163641 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:26.163077 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:26.163641 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:26.163172 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:26.299121 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.299073 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lnzqz" event={"ID":"6fafa6fe-e9ba-4f2e-8043-8e3413afa6de","Type":"ContainerStarted","Data":"5ee8ea5554b1bcfba9ed599f1b388cde60c3bd0c90c9a58d20295d07640ec855"} Jul 17 01:14:26.300472 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.300449 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kmk5c" event={"ID":"5f52d2f1-687d-4203-9c24-788062736e12","Type":"ContainerStarted","Data":"51bc9f0e8f00a0053b6bd70e7a03a95877c14778b1532becdf42562c6aa47699"} Jul 17 01:14:26.301925 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.301901 2583 generic.go:356] "Generic (PLEG): container finished" podID="7a6a1b43738adc5a0d7ebc1b414906ee" containerID="bea855f6f67ee5d63a32b5e0ec4d0ea8acb648256702794122dacf0dfc24575e" exitCode=0 Jul 17 01:14:26.302012 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.301984 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal" event={"ID":"7a6a1b43738adc5a0d7ebc1b414906ee","Type":"ContainerDied","Data":"bea855f6f67ee5d63a32b5e0ec4d0ea8acb648256702794122dacf0dfc24575e"} Jul 17 01:14:26.306151 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.306092 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pzn6s" event={"ID":"2f571d61-7f50-41c1-aabc-19b67211094d","Type":"ContainerStarted","Data":"ba6f28adb1c0e593eceef95c255acef33af76c05c0b60e661bdbd6cb52cb8e8f"} Jul 17 01:14:26.307432 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.307408 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" event={"ID":"2f6f658c-77f8-428a-8bb3-2eb42fa68315","Type":"ContainerStarted","Data":"b2d4c9e580668eb68b545e73477fdfb888f99c00e94156985787846e478249e5"} Jul 17 01:14:26.309866 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.309851 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:14:26.310134 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.310116 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" event={"ID":"ba8f230b-ad43-4811-9a71-a33c5fb4a918","Type":"ContainerStarted","Data":"9f2658b8dc82d2fbec2770a9bb20e85dc6bb8e36fdaf8401fcf890562a31a4ef"} Jul 17 01:14:26.310200 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.310171 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" event={"ID":"ba8f230b-ad43-4811-9a71-a33c5fb4a918","Type":"ContainerStarted","Data":"95693a721cd80c9b0c1e3f7494093747288a60e1fd245de82eed39b1d42d7169"} Jul 17 01:14:26.310238 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.310114 2583 generic.go:356] "Generic (PLEG): container finished" podID="ba8f230b-ad43-4811-9a71-a33c5fb4a918" containerID="d2e40a40707b3077b5e943fecfe60d291990eb0ac9fad8961573a5b1dcadbb10" exitCode=1 Jul 17 01:14:26.310238 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.310209 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" event={"ID":"ba8f230b-ad43-4811-9a71-a33c5fb4a918","Type":"ContainerStarted","Data":"4ec58655320f2dcb70411e847565f570e92eb0f668bbccd13585ee8735440604"} Jul 17 01:14:26.310238 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.310236 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" event={"ID":"ba8f230b-ad43-4811-9a71-a33c5fb4a918","Type":"ContainerStarted","Data":"ccfe95b11354733a72b996efc1b0e1e72e80237bb228e5718d9c8775c338f652"} Jul 17 01:14:26.310357 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.310264 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" event={"ID":"ba8f230b-ad43-4811-9a71-a33c5fb4a918","Type":"ContainerDied","Data":"d2e40a40707b3077b5e943fecfe60d291990eb0ac9fad8961573a5b1dcadbb10"} Jul 17 01:14:26.310357 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.310303 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" event={"ID":"ba8f230b-ad43-4811-9a71-a33c5fb4a918","Type":"ContainerStarted","Data":"aeb0857d5e46c6f63495d2f0bace08a665af8c0fbaa8ec139f00281f4a9b03e8"} Jul 17 01:14:26.311529 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.311512 2583 generic.go:356] "Generic (PLEG): container finished" podID="0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a" containerID="41e7a7a180d81c051adf237e7f270218728519c64e4b98891fd3851273a07e72" exitCode=0 Jul 17 01:14:26.311601 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.311573 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6dvf" event={"ID":"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a","Type":"ContainerDied","Data":"41e7a7a180d81c051adf237e7f270218728519c64e4b98891fd3851273a07e72"} Jul 17 01:14:26.312977 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.312512 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lnzqz" podStartSLOduration=2.445057912 podStartE2EDuration="23.312496537s" totalImagesPullingTime="20.867438625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 01:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:26.311993197 +0000 UTC m=+23.707608048" watchObservedRunningTime="2026-07-17 01:14:26.312496537 +0000 UTC m=+23.708111389" Jul 17 01:14:26.313090 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.313064 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" event={"ID":"b1c00aee-257f-4211-bc8a-44d4dd710c77","Type":"ContainerStarted","Data":"f7dc9824c5b7c01d94471caf29012945b8dd1120f75f30ca26a9f9fe0134d001"} Jul 17 01:14:26.314480 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.314457 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g4pc2" event={"ID":"aa0e084d-6d85-492d-a50a-cf1e654d04a9","Type":"ContainerStarted","Data":"1fc218d449a6f6b059061495287126ba790b4d1819119d8f959e44c1cb7209c9"} Jul 17 01:14:26.315998 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.315979 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-44.ec2.internal" event={"ID":"031589d9c1527982687d362e9909b534","Type":"ContainerStarted","Data":"3339fe6a0b2bc9100ecab15d9b7847e00c1a94f68cca40962a33a199e56f32b6"} Jul 17 01:14:26.359741 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.359698 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kmk5c" podStartSLOduration=2.250434744 podStartE2EDuration="23.359684567s" totalImagesPullingTime="21.109249823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 01:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:26.359154676 +0000 UTC m=+23.754769524" watchObservedRunningTime="2026-07-17 01:14:26.359684567 +0000 UTC m=+23.755299417" Jul 17 01:14:26.374254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.374214 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rxqjf" podStartSLOduration=2.498645127 podStartE2EDuration="23.374200048s" totalImagesPullingTime="20.875554921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 01:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:26.373803743 +0000 UTC m=+23.769418596" watchObservedRunningTime="2026-07-17 01:14:26.374200048 +0000 UTC m=+23.769814899" Jul 17 01:14:26.386983 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.386823 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pzn6s" podStartSLOduration=2.529802813 podStartE2EDuration="23.386813316s" totalImagesPullingTime="20.857010503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 01:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:26.38635583 +0000 UTC m=+23.781970681" watchObservedRunningTime="2026-07-17 01:14:26.386813316 +0000 UTC m=+23.782428166" Jul 17 01:14:26.398838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.398798 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-44.ec2.internal" podStartSLOduration=22.398785041 podStartE2EDuration="22.398785041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 01:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:26.398328401 +0000 UTC m=+23.793943252" watchObservedRunningTime="2026-07-17 01:14:26.398785041 +0000 UTC m=+23.794399935" Jul 17 01:14:26.410581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.410546 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g4pc2" podStartSLOduration=2.428629006 podStartE2EDuration="23.410534059s" totalImagesPullingTime="20.981905053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:26.4105108 +0000 UTC m=+23.806125654" watchObservedRunningTime="2026-07-17 01:14:26.410534059 +0000 UTC m=+23.806148901" Jul 17 01:14:26.507260 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.507138 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:26.577797 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.577768 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:26.680019 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.679992 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:26.782491 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:26.782458 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:27.068847 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:27.068774 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:27.162731 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:27.162701 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:27.162834 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:27.162816 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:27.294674 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:27.294650 2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 01:14:27.320063 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:27.320000 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal" event={"ID":"7a6a1b43738adc5a0d7ebc1b414906ee","Type":"ContainerStarted","Data":"d7784dd8e749decfd61efc49de5fc01459515cac1a4c063b82474b6ea9cc03b8"} Jul 17 01:14:27.324078 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:27.324048 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qprdv" event={"ID":"12a4e0ba-2c69-4ee1-ab15-3dbaf09b5a6a","Type":"ContainerStarted","Data":"73fb2cf60141fd3c42f828212a26b3e8c5d2932948dafd34a334a9c2ce7c1447"} Jul 17 01:14:27.326636 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:27.326608 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" event={"ID":"b1c00aee-257f-4211-bc8a-44d4dd710c77","Type":"ContainerStarted","Data":"d6142f81ae9ad398af00c900fa53853b71d0d7fdc03f1e78eb92f103acfa1566"} Jul 17 01:14:27.334342 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:27.334298 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-44.ec2.internal" podStartSLOduration=23.334282082 podStartE2EDuration="23.334282082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 01:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:27.333674672 +0000 UTC m=+24.729289524" watchObservedRunningTime="2026-07-17 01:14:27.334282082 +0000 UTC m=+24.729896934" Jul 17 01:14:27.346454 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:27.346407 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qprdv" podStartSLOduration=3.410698627 podStartE2EDuration="24.34639455s" totalImagesPullingTime="20.935695923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 01:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:27.346235201 +0000 UTC m=+24.741850051" watchObservedRunningTime="2026-07-17 01:14:27.34639455 +0000 UTC m=+24.742009402" Jul 17 01:14:27.439517 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:27.439486 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:27.771265 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:27.771190 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:27.872878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:27.872830 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:28.058303 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:28.058270 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:28.081708 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:28.081611 2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T01:14:27.294669702Z","UUID":"cd699869-a2f3-4ff7-b19e-b590d6ff92ec","Handler":null,"Name":"","Endpoint":""} Jul 17 01:14:28.083948 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:28.083923 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 01:14:28.083948 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:28.083970 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 01:14:28.159016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:28.158963 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:28.161882 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:28.161856 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:28.162013 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:28.161863 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:28.162013 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:28.161991 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:28.162150 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:28.162051 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:28.331575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:28.331555 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:14:28.461476 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:28.461442 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:28.551982 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:28.551940 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:28.667059 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:28.667033 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:29.071593 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:29.071562 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:29.162161 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:29.162133 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:29.162336 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:29.162262 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:29.231248 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:29.231221 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g4pc2" Jul 17 01:14:29.231896 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:29.231878 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g4pc2" Jul 17 01:14:29.338303 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:29.338235 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:14:29.338816 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:29.338630 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" event={"ID":"ba8f230b-ad43-4811-9a71-a33c5fb4a918","Type":"ContainerStarted","Data":"9eb8e1b7219b8a059a9a1ddd8084740712127210e497b75ef9ab4678bcc8828c"} Jul 17 01:14:29.340911 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:29.340882 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" event={"ID":"b1c00aee-257f-4211-bc8a-44d4dd710c77","Type":"ContainerStarted","Data":"051c11d2b4cbdcf6823dfcf50948dbb96625a7cf6d83db43bb3ccad5282601e0"} Jul 17 01:14:29.341160 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:29.341137 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g4pc2" Jul 17 01:14:29.341653 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:29.341621 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g4pc2" Jul 17 01:14:29.357113 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:29.357059 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hm8lv" podStartSLOduration=2.774118337 podStartE2EDuration="26.357045853s" totalImagesPullingTime="23.582927516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 01:14:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:29.35644766 +0000 UTC m=+26.752062506" watchObservedRunningTime="2026-07-17 01:14:29.357045853 +0000 UTC m=+26.752660703" Jul 17 01:14:29.740203 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:29.740126 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:30.161953 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:30.161922 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:30.161953 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:30.161952 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:30.162189 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:30.162050 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:30.162189 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:30.162173 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:30.236667 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:30.236623 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:30.337750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:30.337715 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:31.162718 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:31.162684 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:31.163196 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:31.162803 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:32.162553 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:32.162378 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:32.162727 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:32.162438 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:32.162727 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:32.162640 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:32.163281 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:32.162745 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:32.351413 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:32.351382 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:14:32.351760 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:32.351734 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" event={"ID":"ba8f230b-ad43-4811-9a71-a33c5fb4a918","Type":"ContainerStarted","Data":"f64d206411dd24fddc98de4d694a9ef367f0ab390f26a6ad311a731ad207f985"} Jul 17 01:14:32.352004 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:32.351978 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:32.352221 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:32.352206 2583 scope.go:117] "RemoveContainer" containerID="d2e40a40707b3077b5e943fecfe60d291990eb0ac9fad8961573a5b1dcadbb10" Jul 17 01:14:32.354116 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:32.354071 2583 generic.go:356] "Generic (PLEG): container finished" podID="0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a" containerID="478526495f4fe4ecc134409ae02125ed262449d0624c808d474c30e3d77806de" exitCode=0 Jul 17 01:14:32.354219 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:32.354131 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6dvf" event={"ID":"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a","Type":"ContainerDied","Data":"478526495f4fe4ecc134409ae02125ed262449d0624c808d474c30e3d77806de"} Jul 17 01:14:32.368890 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:32.368871 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:33.173287 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:33.173254 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:33.173975 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:33.173382 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:33.361493 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:33.361274 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:14:33.361889 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:33.361859 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" event={"ID":"ba8f230b-ad43-4811-9a71-a33c5fb4a918","Type":"ContainerStarted","Data":"d41f21d559bd933691420a6dc0554cce276c0f91e33714b0f77c9c37c4299dc4"} Jul 17 01:14:33.362205 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:33.362174 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:33.362300 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:33.362244 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:33.380550 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:33.380515 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:14:33.387792 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:33.387723 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" podStartSLOduration=9.375970656 podStartE2EDuration="30.387686823s" totalImagesPullingTime="21.011716167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 01:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:33.386196484 +0000 UTC m=+30.781811335" watchObservedRunningTime="2026-07-17 01:14:33.387686823 +0000 UTC m=+30.783301675" Jul 17 01:14:33.438084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:33.437980 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hjgm7"] Jul 17 01:14:33.438223 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:33.438149 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:33.438267 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:33.438238 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:33.440785 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:33.440758 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9bx9f"] Jul 17 01:14:33.440910 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:33.440898 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:33.441036 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:33.441012 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:33.441408 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:33.441383 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-59v85"] Jul 17 01:14:33.441493 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:33.441481 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:33.441586 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:33.441566 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:34.366751 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:34.366699 2583 generic.go:356] "Generic (PLEG): container finished" podID="0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a" containerID="214be1567f68b227893a3c62b77ccfc2aa7308088886e5454ca7f7d38787b907" exitCode=0 Jul 17 01:14:34.367205 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:34.366788 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6dvf" event={"ID":"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a","Type":"ContainerDied","Data":"214be1567f68b227893a3c62b77ccfc2aa7308088886e5454ca7f7d38787b907"} Jul 17 01:14:34.624752 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:34.624677 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:34.728957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:34.728923 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:34.778491 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:34.778463 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:34.891798 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:34.891717 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:35.071894 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:35.071866 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:35.130323 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:35.130285 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:35.165253 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:35.165175 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:35.165424 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:35.165286 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:35.165580 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:35.165546 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:35.165580 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:35.165582 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:35.165815 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:35.165656 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:35.165815 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:35.165759 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:35.256395 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:35.256363 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:35.356516 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:35.356483 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:35.488109 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:35.488022 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:35.548381 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:35.548323 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:35.709521 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:35.709493 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:35.761143 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:35.761118 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:35.871145 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:35.871114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:35.871287 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:35.871165 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs\") pod \"network-metrics-daemon-9bx9f\" (UID: \"8dbc5f6f-120c-42f2-b27f-5644ab623431\") " pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:35.871287 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:35.871210 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 01:14:35.871287 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:35.871267 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 01:14:35.871287 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:35.871279 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret podName:71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a nodeName:}" failed. No retries permitted until 2026-07-17 01:15:07.871261432 +0000 UTC m=+65.266876264 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret") pod "global-pull-secret-syncer-59v85" (UID: "71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a") : object "kube-system"/"original-pull-secret" not registered Jul 17 01:14:35.871447 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:35.871316 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs podName:8dbc5f6f-120c-42f2-b27f-5644ab623431 nodeName:}" failed. No retries permitted until 2026-07-17 01:15:07.871302516 +0000 UTC m=+65.266917345 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs") pod "network-metrics-daemon-9bx9f" (UID: "8dbc5f6f-120c-42f2-b27f-5644ab623431") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 01:14:35.971678 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:35.971636 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtbh\" (UniqueName: \"kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh\") pod \"network-check-target-hjgm7\" (UID: \"5293dc76-7ef2-442f-9035-6403edb5a336\") " pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:35.971840 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:35.971817 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 01:14:35.971895 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:35.971844 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 01:14:35.971895 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:35.971855 2583 projected.go:196] Error preparing data for projected volume kube-api-access-cvtbh for pod openshift-network-diagnostics/network-check-target-hjgm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 01:14:35.971963 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:35.971924 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh podName:5293dc76-7ef2-442f-9035-6403edb5a336 nodeName:}" failed. No retries permitted until 2026-07-17 01:15:07.971909325 +0000 UTC m=+65.367524153 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cvtbh" (UniqueName: "kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh") pod "network-check-target-hjgm7" (UID: "5293dc76-7ef2-442f-9035-6403edb5a336") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 01:14:36.086863 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:36.086832 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:36.280963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:36.280924 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:36.373870 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:36.373780 2583 generic.go:356] "Generic (PLEG): container finished" podID="0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a" containerID="d3bb29399617e1cd2a511f4202633293c8757aef41c155c909e2069b364127e0" exitCode=0 Jul 17 01:14:36.373870 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:36.373841 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6dvf" event={"ID":"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a","Type":"ContainerDied","Data":"d3bb29399617e1cd2a511f4202633293c8757aef41c155c909e2069b364127e0"} Jul 17 01:14:36.381388 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:36.381365 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:37.165978 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:37.165950 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:37.166404 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:37.165985 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:37.166404 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:37.166057 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:37.166404 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:37.166064 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hjgm7" podUID="5293dc76-7ef2-442f-9035-6403edb5a336" Jul 17 01:14:37.166404 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:37.166207 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9bx9f" podUID="8dbc5f6f-120c-42f2-b27f-5644ab623431" Jul 17 01:14:37.166404 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:37.166260 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-59v85" podUID="71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a" Jul 17 01:14:37.550630 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:37.550601 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:37.677221 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:37.677195 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:37.916467 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:37.916391 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:38.481803 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.481777 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-44.ec2.internal" event="NodeReady" Jul 17 01:14:38.482200 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.481895 2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 01:14:38.524861 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.524827 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xjl9w"] Jul 17 01:14:38.526791 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.526759 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qhmb7"] Jul 17 01:14:38.526905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.526888 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:38.528873 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.528856 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qhmb7" Jul 17 01:14:38.529223 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.529187 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 01:14:38.529317 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.529273 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 01:14:38.529387 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.529290 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lh9xl\"" Jul 17 01:14:38.531056 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.531042 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 01:14:38.531164 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.531152 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 01:14:38.531306 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.531285 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 01:14:38.531442 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.531422 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-php8q\"" Jul 17 01:14:38.538229 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.538210 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qhmb7"] Jul 17 01:14:38.538995 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.538974 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xjl9w"] Jul 17 01:14:38.546213 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.546190 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q9bjq"] Jul 17 01:14:38.548746 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.548729 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.551112 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.551074 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 01:14:38.551369 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.551348 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 01:14:38.551467 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.551354 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 01:14:38.551467 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.551399 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h47bb\"" Jul 17 01:14:38.552074 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.552057 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 01:14:38.559006 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.558988 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q9bjq"] Jul 17 01:14:38.592920 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.592887 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dfeaf340-95bf-443c-a180-f6b883794466-config-volume\") pod \"dns-default-xjl9w\" (UID: \"dfeaf340-95bf-443c-a180-f6b883794466\") " pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:38.593132 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.592924 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpbxs\" (UniqueName: \"kubernetes.io/projected/dfeaf340-95bf-443c-a180-f6b883794466-kube-api-access-vpbxs\") pod \"dns-default-xjl9w\" (UID: \"dfeaf340-95bf-443c-a180-f6b883794466\") " pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:38.593197 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.593119 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfeaf340-95bf-443c-a180-f6b883794466-metrics-tls\") pod \"dns-default-xjl9w\" (UID: \"dfeaf340-95bf-443c-a180-f6b883794466\") " pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:38.593197 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.593159 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf340-95bf-443c-a180-f6b883794466-tmp-dir\") pod \"dns-default-xjl9w\" (UID: \"dfeaf340-95bf-443c-a180-f6b883794466\") " pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:38.652483 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.652445 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:38.693834 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.693801 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9ee2220-5d23-4a71-b795-7ff9e1a57868-crio-socket\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.693834 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.693837 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dfeaf340-95bf-443c-a180-f6b883794466-config-volume\") pod \"dns-default-xjl9w\" (UID: \"dfeaf340-95bf-443c-a180-f6b883794466\") " pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:38.694069 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.693856 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgxl\" (UniqueName: \"kubernetes.io/projected/a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83-kube-api-access-gqgxl\") pod \"ingress-canary-qhmb7\" (UID: \"a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83\") " pod="openshift-ingress-canary/ingress-canary-qhmb7" Jul 17 01:14:38.694069 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.693871 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9ee2220-5d23-4a71-b795-7ff9e1a57868-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.694069 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.693933 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbxs\" (UniqueName: \"kubernetes.io/projected/dfeaf340-95bf-443c-a180-f6b883794466-kube-api-access-vpbxs\") pod \"dns-default-xjl9w\" (UID: \"dfeaf340-95bf-443c-a180-f6b883794466\") " pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:38.694069 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.693986 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83-cert\") pod \"ingress-canary-qhmb7\" (UID: \"a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83\") " pod="openshift-ingress-canary/ingress-canary-qhmb7" Jul 17 01:14:38.694069 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.694019 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfeaf340-95bf-443c-a180-f6b883794466-metrics-tls\") pod \"dns-default-xjl9w\" (UID: \"dfeaf340-95bf-443c-a180-f6b883794466\") " pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:38.694069 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.694036 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf340-95bf-443c-a180-f6b883794466-tmp-dir\") pod \"dns-default-xjl9w\" (UID: \"dfeaf340-95bf-443c-a180-f6b883794466\") " pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:38.694388 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.694140 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9ee2220-5d23-4a71-b795-7ff9e1a57868-data-volume\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.694388 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.694165 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9ee2220-5d23-4a71-b795-7ff9e1a57868-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.694388 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.694207 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wp5\" (UniqueName: \"kubernetes.io/projected/e9ee2220-5d23-4a71-b795-7ff9e1a57868-kube-api-access-t8wp5\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.694537 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.694403 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfeaf340-95bf-443c-a180-f6b883794466-tmp-dir\") pod \"dns-default-xjl9w\" (UID: \"dfeaf340-95bf-443c-a180-f6b883794466\") " pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:38.694584 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.694537 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dfeaf340-95bf-443c-a180-f6b883794466-config-volume\") pod \"dns-default-xjl9w\" (UID: \"dfeaf340-95bf-443c-a180-f6b883794466\") " pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:38.698471 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.698367 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfeaf340-95bf-443c-a180-f6b883794466-metrics-tls\") pod \"dns-default-xjl9w\" (UID: \"dfeaf340-95bf-443c-a180-f6b883794466\") " pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:38.701064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.701034 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpbxs\" (UniqueName: \"kubernetes.io/projected/dfeaf340-95bf-443c-a180-f6b883794466-kube-api-access-vpbxs\") pod \"dns-default-xjl9w\" (UID: \"dfeaf340-95bf-443c-a180-f6b883794466\") " pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:38.752965 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.752885 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:38.795419 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.795387 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9ee2220-5d23-4a71-b795-7ff9e1a57868-crio-socket\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.795572 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.795438 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgxl\" (UniqueName: \"kubernetes.io/projected/a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83-kube-api-access-gqgxl\") pod \"ingress-canary-qhmb7\" (UID: \"a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83\") " pod="openshift-ingress-canary/ingress-canary-qhmb7" Jul 17 01:14:38.795572 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.795465 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9ee2220-5d23-4a71-b795-7ff9e1a57868-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.795572 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.795517 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83-cert\") pod \"ingress-canary-qhmb7\" (UID: \"a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83\") " pod="openshift-ingress-canary/ingress-canary-qhmb7" Jul 17 01:14:38.795701 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.795571 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9ee2220-5d23-4a71-b795-7ff9e1a57868-data-volume\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.795701 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.795595 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9ee2220-5d23-4a71-b795-7ff9e1a57868-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.795701 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.795618 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wp5\" (UniqueName: \"kubernetes.io/projected/e9ee2220-5d23-4a71-b795-7ff9e1a57868-kube-api-access-t8wp5\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.796169 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.796148 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9ee2220-5d23-4a71-b795-7ff9e1a57868-crio-socket\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.796630 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.796608 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9ee2220-5d23-4a71-b795-7ff9e1a57868-data-volume\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.797347 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.797255 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9ee2220-5d23-4a71-b795-7ff9e1a57868-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.799392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.799372 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83-cert\") pod \"ingress-canary-qhmb7\" (UID: \"a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83\") " pod="openshift-ingress-canary/ingress-canary-qhmb7" Jul 17 01:14:38.799750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.799730 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9ee2220-5d23-4a71-b795-7ff9e1a57868-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.805314 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.805288 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wp5\" (UniqueName: \"kubernetes.io/projected/e9ee2220-5d23-4a71-b795-7ff9e1a57868-kube-api-access-t8wp5\") pod \"insights-runtime-extractor-q9bjq\" (UID: \"e9ee2220-5d23-4a71-b795-7ff9e1a57868\") " pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:38.805945 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.805909 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgxl\" (UniqueName: \"kubernetes.io/projected/a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83-kube-api-access-gqgxl\") pod \"ingress-canary-qhmb7\" (UID: \"a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83\") " pod="openshift-ingress-canary/ingress-canary-qhmb7" Jul 17 01:14:38.837693 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.837661 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:38.847486 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.847463 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qhmb7" Jul 17 01:14:38.857630 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:38.857603 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q9bjq" Jul 17 01:14:39.006058 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.005971 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xjl9w"] Jul 17 01:14:39.011951 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.011925 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qhmb7"] Jul 17 01:14:39.012755 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:39.012728 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfeaf340_95bf_443c_a180_f6b883794466.slice/crio-cbd4101300c3d1ed217de3f8e879c082286fe260d24d64d78698d91d74a77843 WatchSource:0}: Error finding container cbd4101300c3d1ed217de3f8e879c082286fe260d24d64d78698d91d74a77843: Status 404 returned error can't find the container with id cbd4101300c3d1ed217de3f8e879c082286fe260d24d64d78698d91d74a77843 Jul 17 01:14:39.015622 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:39.015601 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a8c41d_9ea9_4ec3_a61c_9911f9c0ff83.slice/crio-5fdfcfb3dd8ccc56e37ca63ce474c6f06215eb6e8b48627caa71b2300f60324e WatchSource:0}: Error finding container 5fdfcfb3dd8ccc56e37ca63ce474c6f06215eb6e8b48627caa71b2300f60324e: Status 404 returned error can't find the container with id 5fdfcfb3dd8ccc56e37ca63ce474c6f06215eb6e8b48627caa71b2300f60324e Jul 17 01:14:39.018636 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.018616 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q9bjq"] Jul 17 01:14:39.022511 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:39.022491 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ee2220_5d23_4a71_b795_7ff9e1a57868.slice/crio-0b523751a1e79e1e8f8375ae7ca5ef04ef84185c93d99b1d2544bd32f26166a6 WatchSource:0}: Error finding container 0b523751a1e79e1e8f8375ae7ca5ef04ef84185c93d99b1d2544bd32f26166a6: Status 404 returned error can't find the container with id 0b523751a1e79e1e8f8375ae7ca5ef04ef84185c93d99b1d2544bd32f26166a6 Jul 17 01:14:39.162151 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.162115 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:14:39.162332 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.162113 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:14:39.162429 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.162126 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:14:39.164740 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.164718 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 01:14:39.164866 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.164757 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 01:14:39.164866 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.164765 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hzllc\"" Jul 17 01:14:39.164983 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.164926 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 01:14:39.165068 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.165047 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dtqbv\"" Jul 17 01:14:39.165209 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.165184 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 01:14:39.383901 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.383861 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q9bjq" event={"ID":"e9ee2220-5d23-4a71-b795-7ff9e1a57868","Type":"ContainerStarted","Data":"0b523751a1e79e1e8f8375ae7ca5ef04ef84185c93d99b1d2544bd32f26166a6"} Jul 17 01:14:39.385135 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.385088 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qhmb7" event={"ID":"a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83","Type":"ContainerStarted","Data":"5fdfcfb3dd8ccc56e37ca63ce474c6f06215eb6e8b48627caa71b2300f60324e"} Jul 17 01:14:39.386279 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.386254 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xjl9w" event={"ID":"dfeaf340-95bf-443c-a180-f6b883794466","Type":"ContainerStarted","Data":"cbd4101300c3d1ed217de3f8e879c082286fe260d24d64d78698d91d74a77843"} Jul 17 01:14:39.723556 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:39.723472 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:43.814451 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:43.814188 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:43.915260 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:43.915222 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:44.391637 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.391609 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:44.487377 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.487339 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:44.603456 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.603422 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:44.650216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.650137 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk"] Jul 17 01:14:44.674971 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.674895 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk"] Jul 17 01:14:44.675146 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.675023 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" Jul 17 01:14:44.677638 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.677612 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 17 01:14:44.678528 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.678483 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 01:14:44.678528 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.678506 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-k7j44\"" Jul 17 01:14:44.678528 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.678512 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 17 01:14:44.678716 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.678507 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 01:14:44.678716 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.678506 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 01:14:44.685853 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.685829 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-rcr67"] Jul 17 01:14:44.702941 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.702916 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-rcr67"] Jul 17 01:14:44.703071 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.703055 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.704122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.703331 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kvmg6"] Jul 17 01:14:44.705784 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.705766 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7cwnl\"" Jul 17 01:14:44.706854 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.706838 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 17 01:14:44.707922 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.707895 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 17 01:14:44.708547 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.708288 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 17 01:14:44.711398 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.710699 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:44.723206 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.723175 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.725857 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.725830 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 01:14:44.726577 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.726246 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 01:14:44.726577 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.726383 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 01:14:44.726577 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.726453 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k8bd4\"" Jul 17 01:14:44.783473 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.783438 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:44.845798 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.845764 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cf44875-46d9-4e1a-b594-87fd5ebe44b0-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-nfzqk\" (UID: \"5cf44875-46d9-4e1a-b594-87fd5ebe44b0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" Jul 17 01:14:44.846263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.845805 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-accelerators-collector-config\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.846263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.845828 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.846263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.845879 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74496ef6-aa60-4933-ba21-7fc27f0f2549-metrics-client-ca\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.846263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.845918 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.846263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.845982 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.846263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.846036 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7rqc\" (UniqueName: \"kubernetes.io/projected/5cf44875-46d9-4e1a-b594-87fd5ebe44b0-kube-api-access-k7rqc\") pod \"openshift-state-metrics-79f6958cdf-nfzqk\" (UID: \"5cf44875-46d9-4e1a-b594-87fd5ebe44b0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" Jul 17 01:14:44.846263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.846059 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-wtmp\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.846263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.846082 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.846263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.846127 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cf44875-46d9-4e1a-b594-87fd5ebe44b0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-nfzqk\" (UID: \"5cf44875-46d9-4e1a-b594-87fd5ebe44b0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" Jul 17 01:14:44.846263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.846164 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cf44875-46d9-4e1a-b594-87fd5ebe44b0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-nfzqk\" (UID: \"5cf44875-46d9-4e1a-b594-87fd5ebe44b0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" Jul 17 01:14:44.846263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.846204 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p55vc\" (UniqueName: \"kubernetes.io/projected/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-api-access-p55vc\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.846263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.846254 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74496ef6-aa60-4933-ba21-7fc27f0f2549-sys\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.846751 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.846300 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-textfile\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.846751 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.846345 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.846751 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.846370 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsfjc\" (UniqueName: \"kubernetes.io/projected/74496ef6-aa60-4933-ba21-7fc27f0f2549-kube-api-access-zsfjc\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.846751 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.846398 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74496ef6-aa60-4933-ba21-7fc27f0f2549-root\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.846751 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.846455 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-tls\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.846751 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.846504 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.883307 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.883268 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:44.947922 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.947831 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cf44875-46d9-4e1a-b594-87fd5ebe44b0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-nfzqk\" (UID: \"5cf44875-46d9-4e1a-b594-87fd5ebe44b0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" Jul 17 01:14:44.947922 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.947875 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cf44875-46d9-4e1a-b594-87fd5ebe44b0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-nfzqk\" (UID: \"5cf44875-46d9-4e1a-b594-87fd5ebe44b0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" Jul 17 01:14:44.947922 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.947893 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p55vc\" (UniqueName: \"kubernetes.io/projected/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-api-access-p55vc\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.947922 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.947917 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74496ef6-aa60-4933-ba21-7fc27f0f2549-sys\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.948233 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.947949 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-textfile\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.948233 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.947987 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.948233 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948011 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsfjc\" (UniqueName: \"kubernetes.io/projected/74496ef6-aa60-4933-ba21-7fc27f0f2549-kube-api-access-zsfjc\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.948233 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948026 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74496ef6-aa60-4933-ba21-7fc27f0f2549-sys\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.948233 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948113 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74496ef6-aa60-4933-ba21-7fc27f0f2549-root\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.948233 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948035 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74496ef6-aa60-4933-ba21-7fc27f0f2549-root\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.948233 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948164 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-tls\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.948233 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948219 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.948457 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948266 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cf44875-46d9-4e1a-b594-87fd5ebe44b0-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-nfzqk\" (UID: \"5cf44875-46d9-4e1a-b594-87fd5ebe44b0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" Jul 17 01:14:44.948457 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948293 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-accelerators-collector-config\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.948457 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948322 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.948457 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948344 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74496ef6-aa60-4933-ba21-7fc27f0f2549-metrics-client-ca\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.948457 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948365 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-textfile\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.948457 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948372 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.948457 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948429 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.948457 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:44.948446 2583 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 17 01:14:44.948814 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948479 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7rqc\" (UniqueName: \"kubernetes.io/projected/5cf44875-46d9-4e1a-b594-87fd5ebe44b0-kube-api-access-k7rqc\") pod \"openshift-state-metrics-79f6958cdf-nfzqk\" (UID: \"5cf44875-46d9-4e1a-b594-87fd5ebe44b0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" Jul 17 01:14:44.948814 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:44.948503 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-tls podName:f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb nodeName:}" failed. No retries permitted until 2026-07-17 01:14:45.448485514 +0000 UTC m=+42.844100342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-rcr67" (UID: "f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb") : secret "kube-state-metrics-tls" not found Jul 17 01:14:44.948814 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948527 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-wtmp\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.948814 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.948552 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.949584 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.949267 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.949707 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.949633 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.950539 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.949857 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.950539 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.949942 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-wtmp\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.950539 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.950365 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74496ef6-aa60-4933-ba21-7fc27f0f2549-metrics-client-ca\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.950539 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.950409 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-accelerators-collector-config\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.950539 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.950500 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cf44875-46d9-4e1a-b594-87fd5ebe44b0-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-nfzqk\" (UID: \"5cf44875-46d9-4e1a-b594-87fd5ebe44b0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" Jul 17 01:14:44.952055 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.952029 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.952427 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.952367 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.952427 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.952416 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cf44875-46d9-4e1a-b594-87fd5ebe44b0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-nfzqk\" (UID: \"5cf44875-46d9-4e1a-b594-87fd5ebe44b0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" Jul 17 01:14:44.952546 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.952459 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/74496ef6-aa60-4933-ba21-7fc27f0f2549-node-exporter-tls\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.952600 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.952568 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cf44875-46d9-4e1a-b594-87fd5ebe44b0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-nfzqk\" (UID: \"5cf44875-46d9-4e1a-b594-87fd5ebe44b0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" Jul 17 01:14:44.956736 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.956710 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p55vc\" (UniqueName: \"kubernetes.io/projected/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-api-access-p55vc\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:44.957782 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.957759 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7rqc\" (UniqueName: \"kubernetes.io/projected/5cf44875-46d9-4e1a-b594-87fd5ebe44b0-kube-api-access-k7rqc\") pod \"openshift-state-metrics-79f6958cdf-nfzqk\" (UID: \"5cf44875-46d9-4e1a-b594-87fd5ebe44b0\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" Jul 17 01:14:44.963936 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.963910 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsfjc\" (UniqueName: \"kubernetes.io/projected/74496ef6-aa60-4933-ba21-7fc27f0f2549-kube-api-access-zsfjc\") pod \"node-exporter-kvmg6\" (UID: \"74496ef6-aa60-4933-ba21-7fc27f0f2549\") " pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:44.967914 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.967871 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:44.987249 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:44.987221 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" Jul 17 01:14:45.034117 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.034066 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kvmg6" Jul 17 01:14:45.100062 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.100028 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:45.149615 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.149575 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:45.282130 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.282088 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:45.452701 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.452670 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:45.452889 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:45.452793 2583 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 17 01:14:45.452889 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:45.452868 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-tls podName:f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb nodeName:}" failed. No retries permitted until 2026-07-17 01:14:46.452846012 +0000 UTC m=+43.848460863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-rcr67" (UID: "f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb") : secret "kube-state-metrics-tls" not found Jul 17 01:14:45.563469 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.563386 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:45.661473 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.661435 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:45.694236 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.694207 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 01:14:45.710632 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.710600 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.711398 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.711345 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 01:14:45.713758 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.713268 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 01:14:45.713758 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.713411 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 01:14:45.713758 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.713644 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 01:14:45.713758 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.713650 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 01:14:45.714029 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.713905 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zdbz6\"" Jul 17 01:14:45.714029 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.713941 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 01:14:45.714221 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.714203 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 01:14:45.714514 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.714496 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 01:14:45.714620 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.714556 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 01:14:45.715412 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.715393 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 01:14:45.740362 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.740330 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:45.856215 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.856127 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e149373-0eec-4dc0-bec2-0987683e6f2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.856215 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.856195 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e149373-0eec-4dc0-bec2-0987683e6f2b-config-out\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.856734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.856229 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e149373-0eec-4dc0-bec2-0987683e6f2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.856734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.856263 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzn96\" (UniqueName: \"kubernetes.io/projected/6e149373-0eec-4dc0-bec2-0987683e6f2b-kube-api-access-mzn96\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.856734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.856289 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.856734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.856330 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.856734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.856361 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.856734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.856398 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e149373-0eec-4dc0-bec2-0987683e6f2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.856734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.856439 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.856734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.856463 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.856734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.856486 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.856734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.856540 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e149373-0eec-4dc0-bec2-0987683e6f2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.856734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.856565 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-web-config\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.957639 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.957605 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.957823 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.957647 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.957823 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.957665 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.957823 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.957693 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e149373-0eec-4dc0-bec2-0987683e6f2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.957823 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:45.957781 2583 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 17 01:14:45.957823 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.957791 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.957823 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.957820 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.958149 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:14:45.957848 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-main-tls podName:6e149373-0eec-4dc0-bec2-0987683e6f2b nodeName:}" failed. No retries permitted until 2026-07-17 01:14:46.457828824 +0000 UTC m=+43.853443671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6e149373-0eec-4dc0-bec2-0987683e6f2b") : secret "alertmanager-main-tls" not found Jul 17 01:14:45.958149 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.957867 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.958149 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.957926 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e149373-0eec-4dc0-bec2-0987683e6f2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.958149 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.957959 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-web-config\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.958149 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.958002 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e149373-0eec-4dc0-bec2-0987683e6f2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.961574 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.959701 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e149373-0eec-4dc0-bec2-0987683e6f2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.961574 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.959785 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e149373-0eec-4dc0-bec2-0987683e6f2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.961574 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.959813 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e149373-0eec-4dc0-bec2-0987683e6f2b-config-out\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.961574 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.959870 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e149373-0eec-4dc0-bec2-0987683e6f2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.961574 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.959933 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzn96\" (UniqueName: \"kubernetes.io/projected/6e149373-0eec-4dc0-bec2-0987683e6f2b-kube-api-access-mzn96\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.961574 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.960488 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e149373-0eec-4dc0-bec2-0987683e6f2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.961574 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.960862 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.961574 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.961043 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e149373-0eec-4dc0-bec2-0987683e6f2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.962054 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.962028 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.962782 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.962759 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.966452 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.966431 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e149373-0eec-4dc0-bec2-0987683e6f2b-config-out\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.966890 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.966871 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.966967 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.966906 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-web-config\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.967195 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.967173 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:45.969212 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:45.969192 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzn96\" (UniqueName: \"kubernetes.io/projected/6e149373-0eec-4dc0-bec2-0987683e6f2b-kube-api-access-mzn96\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:46.005358 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.005327 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:46.464250 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.464163 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:46.464250 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.464218 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:46.466785 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.466752 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-rcr67\" (UID: \"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:46.466899 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.466753 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:46.514876 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.514841 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" Jul 17 01:14:46.574375 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.574346 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:46.621781 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.621746 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:14:46.623868 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.623841 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6449bf7ffd-plq9l"] Jul 17 01:14:46.643147 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.643119 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6449bf7ffd-plq9l"] Jul 17 01:14:46.643291 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.643275 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.645785 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.645759 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zxksx\"" Jul 17 01:14:46.645922 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.645905 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 17 01:14:46.646052 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.646031 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 17 01:14:46.646174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.646116 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 17 01:14:46.646174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.646148 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4gre799vl5t3d\"" Jul 17 01:14:46.646174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.646164 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 17 01:14:46.646327 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.646210 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 17 01:14:46.766884 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.766787 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.766884 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.766836 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.767151 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.766907 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.767151 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.766979 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.767151 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.767002 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-metrics-client-ca\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.767151 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.767038 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-tls\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.767362 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.767165 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-grpc-tls\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.767362 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.767189 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4bhp\" (UniqueName: \"kubernetes.io/projected/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-kube-api-access-k4bhp\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.868402 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.868367 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-metrics-client-ca\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.868859 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.868425 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-tls\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.868859 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.868532 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-grpc-tls\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.868859 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.868564 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4bhp\" (UniqueName: \"kubernetes.io/projected/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-kube-api-access-k4bhp\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.868859 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.868607 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.868859 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.868636 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.868859 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.868666 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.868859 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.868710 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.869280 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.869228 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-metrics-client-ca\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.871716 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.871692 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-tls\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.872163 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.872092 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.872262 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.872208 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.872262 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.872237 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.872457 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.872434 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.872511 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.872455 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-secret-grpc-tls\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.875797 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.875776 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4bhp\" (UniqueName: \"kubernetes.io/projected/6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7-kube-api-access-k4bhp\") pod \"thanos-querier-6449bf7ffd-plq9l\" (UID: \"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7\") " pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:46.953178 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:46.953142 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:14:47.078855 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:47.078827 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74496ef6_aa60_4933_ba21_7fc27f0f2549.slice/crio-33b0939c4b4a3a44c736365943ea5f3d8326f4954a622a724c25e2266df8d9e8 WatchSource:0}: Error finding container 33b0939c4b4a3a44c736365943ea5f3d8326f4954a622a724c25e2266df8d9e8: Status 404 returned error can't find the container with id 33b0939c4b4a3a44c736365943ea5f3d8326f4954a622a724c25e2266df8d9e8 Jul 17 01:14:47.327700 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:47.327654 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6449bf7ffd-plq9l"] Jul 17 01:14:47.409087 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:47.409048 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kvmg6" event={"ID":"74496ef6-aa60-4933-ba21-7fc27f0f2549","Type":"ContainerStarted","Data":"33b0939c4b4a3a44c736365943ea5f3d8326f4954a622a724c25e2266df8d9e8"} Jul 17 01:14:47.410619 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:47.410590 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xjl9w" event={"ID":"dfeaf340-95bf-443c-a180-f6b883794466","Type":"ContainerStarted","Data":"214a812d0e38d57b6bb05f7ffe80be12dbb36ea7940a967c458a8b53ff2516a4"} Jul 17 01:14:47.412126 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:47.412085 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qhmb7" event={"ID":"a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83","Type":"ContainerStarted","Data":"90e3350f5278495c5a4144a59fbc3583b63d87026fc64661a59d5b08fad45b6f"} Jul 17 01:14:47.415242 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:47.415215 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6dvf" event={"ID":"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a","Type":"ContainerStarted","Data":"c989dda02e678ffd6e16f7fd749839def1e85a1377d667dbf751b48e4336e0e0"} Jul 17 01:14:47.429028 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:47.428973 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qhmb7" podStartSLOduration=1.375848297 podStartE2EDuration="9.428954081s" totalImagesPullingTime="8.053105784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:47.428184957 +0000 UTC m=+44.823799810" watchObservedRunningTime="2026-07-17 01:14:47.428954081 +0000 UTC m=+44.824568933" Jul 17 01:14:47.496740 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:47.496690 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk"] Jul 17 01:14:47.496894 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:47.496831 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-rcr67"] Jul 17 01:14:47.499774 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:47.499743 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 01:14:47.511348 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:47.511322 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e149373_0eec_4dc0_bec2_0987683e6f2b.slice/crio-73c9762f4558982382aed8fffaae637545bf5c8fc50e1ba1e24602b14c6cda8b WatchSource:0}: Error finding container 73c9762f4558982382aed8fffaae637545bf5c8fc50e1ba1e24602b14c6cda8b: Status 404 returned error can't find the container with id 73c9762f4558982382aed8fffaae637545bf5c8fc50e1ba1e24602b14c6cda8b Jul 17 01:14:47.586203 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:47.586173 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:47.689270 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:47.689231 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:47.976281 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:47.976149 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:48.325283 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:48.325247 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:48.420986 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:48.420952 2583 generic.go:356] "Generic (PLEG): container finished" podID="0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a" containerID="c989dda02e678ffd6e16f7fd749839def1e85a1377d667dbf751b48e4336e0e0" exitCode=0 Jul 17 01:14:48.421175 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:48.421032 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6dvf" event={"ID":"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a","Type":"ContainerDied","Data":"c989dda02e678ffd6e16f7fd749839def1e85a1377d667dbf751b48e4336e0e0"} Jul 17 01:14:48.422543 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:48.422522 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q9bjq" event={"ID":"e9ee2220-5d23-4a71-b795-7ff9e1a57868","Type":"ContainerStarted","Data":"919853010af5adfd6d521b98afb4c3ac2c389946559b2c58a714e5b923c408ff"} Jul 17 01:14:48.423605 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:48.423582 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" event={"ID":"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb","Type":"ContainerStarted","Data":"333118db3f186a76abdd5651d63fab277ed2379277019b447d72863d69ff686f"} Jul 17 01:14:48.425364 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:48.425343 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xjl9w" event={"ID":"dfeaf340-95bf-443c-a180-f6b883794466","Type":"ContainerStarted","Data":"0f422c28357108d0b17195bdf0f77116456e49ddb1e0fecb3aa092ce84d84e67"} Jul 17 01:14:48.425490 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:48.425473 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:48.426552 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:48.426533 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" event={"ID":"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7","Type":"ContainerStarted","Data":"0708b065804e1c2d9d816da744ac5eb9795deebe040cf91f4c9da376732e9f83"} Jul 17 01:14:48.427657 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:48.427637 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerStarted","Data":"73c9762f4558982382aed8fffaae637545bf5c8fc50e1ba1e24602b14c6cda8b"} Jul 17 01:14:48.429222 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:48.429202 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" event={"ID":"5cf44875-46d9-4e1a-b594-87fd5ebe44b0","Type":"ContainerStarted","Data":"2f9e588511c0e5758f8c0886a362830f062988183ff835d59d34f037ec150589"} Jul 17 01:14:48.429319 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:48.429264 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" event={"ID":"5cf44875-46d9-4e1a-b594-87fd5ebe44b0","Type":"ContainerStarted","Data":"684ffd4ec5803d4fe2069b93c509b4948c02d83601d20ef11c8f11c1647113e9"} Jul 17 01:14:48.429319 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:48.429301 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" event={"ID":"5cf44875-46d9-4e1a-b594-87fd5ebe44b0","Type":"ContainerStarted","Data":"866fe5b7e57d0c846502e71e6d01bef10208b258a23ebfa79240a2f45e4077ac"} Jul 17 01:14:48.455717 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:48.455649 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xjl9w" podStartSLOduration=2.394345946 podStartE2EDuration="10.455629773s" totalImagesPullingTime="8.061283827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:48.454942499 +0000 UTC m=+45.850557350" watchObservedRunningTime="2026-07-17 01:14:48.455629773 +0000 UTC m=+45.851244652" Jul 17 01:14:49.169672 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.168671 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-66c9bc58bf-q57jn"] Jul 17 01:14:49.181759 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.181703 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66c9bc58bf-q57jn"] Jul 17 01:14:49.181897 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.181833 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.184807 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.184607 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 17 01:14:49.185718 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.185552 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 01:14:49.185718 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.185572 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 17 01:14:49.185718 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.185572 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gqtv6\"" Jul 17 01:14:49.185718 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.185655 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5pvmeo531h1d3\"" Jul 17 01:14:49.186058 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.185782 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 17 01:14:49.295011 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.294559 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a6349637-3750-4429-a596-65de3a5824b6-secret-metrics-server-client-certs\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.295011 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.294626 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a6349637-3750-4429-a596-65de3a5824b6-secret-metrics-server-tls\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.295011 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.294680 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a6349637-3750-4429-a596-65de3a5824b6-audit-log\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.295011 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.294736 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a6349637-3750-4429-a596-65de3a5824b6-metrics-server-audit-profiles\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.295011 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.294780 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6349637-3750-4429-a596-65de3a5824b6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.295011 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.294803 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a6349637-3750-4429-a596-65de3a5824b6-client-ca-bundle\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.295011 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.294858 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjnhj\" (UniqueName: \"kubernetes.io/projected/a6349637-3750-4429-a596-65de3a5824b6-kube-api-access-mjnhj\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.399129 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.396320 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a6349637-3750-4429-a596-65de3a5824b6-metrics-server-audit-profiles\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.399129 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.396386 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6349637-3750-4429-a596-65de3a5824b6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.399129 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.396416 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a6349637-3750-4429-a596-65de3a5824b6-client-ca-bundle\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.399129 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.396477 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjnhj\" (UniqueName: \"kubernetes.io/projected/a6349637-3750-4429-a596-65de3a5824b6-kube-api-access-mjnhj\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.399129 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.396533 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a6349637-3750-4429-a596-65de3a5824b6-secret-metrics-server-client-certs\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.399129 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.396600 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a6349637-3750-4429-a596-65de3a5824b6-secret-metrics-server-tls\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.399129 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.396633 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a6349637-3750-4429-a596-65de3a5824b6-audit-log\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.399129 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.397033 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a6349637-3750-4429-a596-65de3a5824b6-audit-log\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.399129 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.397935 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a6349637-3750-4429-a596-65de3a5824b6-metrics-server-audit-profiles\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.399129 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.398478 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6349637-3750-4429-a596-65de3a5824b6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.401924 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.401855 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a6349637-3750-4429-a596-65de3a5824b6-client-ca-bundle\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.402586 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.402565 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a6349637-3750-4429-a596-65de3a5824b6-secret-metrics-server-client-certs\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.403751 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.403729 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a6349637-3750-4429-a596-65de3a5824b6-secret-metrics-server-tls\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.409350 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.409286 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:49.411782 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.411733 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjnhj\" (UniqueName: \"kubernetes.io/projected/a6349637-3750-4429-a596-65de3a5824b6-kube-api-access-mjnhj\") pod \"metrics-server-66c9bc58bf-q57jn\" (UID: \"a6349637-3750-4429-a596-65de3a5824b6\") " pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.420738 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.420670 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr"] Jul 17 01:14:49.441323 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.440719 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr"] Jul 17 01:14:49.441323 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.440844 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr" Jul 17 01:14:49.445847 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.444317 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7lvdq\"" Jul 17 01:14:49.445847 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.444805 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 17 01:14:49.463901 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.463865 2583 generic.go:356] "Generic (PLEG): container finished" podID="0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a" containerID="9489cdc9ce2c9ac1cb0291ba880366213df4eb841e270dfdcba2a9c51762dbb8" exitCode=0 Jul 17 01:14:49.464058 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.463944 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6dvf" event={"ID":"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a","Type":"ContainerDied","Data":"9489cdc9ce2c9ac1cb0291ba880366213df4eb841e270dfdcba2a9c51762dbb8"} Jul 17 01:14:49.467258 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.467224 2583 generic.go:356] "Generic (PLEG): container finished" podID="74496ef6-aa60-4933-ba21-7fc27f0f2549" containerID="f59a3c7fa74b099ba43a6f0a800a87563ec98c3ee59bfdb37b2f061fe200ac09" exitCode=0 Jul 17 01:14:49.467537 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.467508 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kvmg6" event={"ID":"74496ef6-aa60-4933-ba21-7fc27f0f2549","Type":"ContainerDied","Data":"f59a3c7fa74b099ba43a6f0a800a87563ec98c3ee59bfdb37b2f061fe200ac09"} Jul 17 01:14:49.472187 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.472159 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:49.495375 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.495350 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:14:49.600844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.599996 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flt68\" (UniqueName: \"kubernetes.io/projected/c9c7d960-19bd-410d-aab3-f811fc713b28-kube-api-access-flt68\") pod \"monitoring-plugin-7b7b7bc8bf-cm6lr\" (UID: \"c9c7d960-19bd-410d-aab3-f811fc713b28\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr" Jul 17 01:14:49.600844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.600114 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9c7d960-19bd-410d-aab3-f811fc713b28-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-cm6lr\" (UID: \"c9c7d960-19bd-410d-aab3-f811fc713b28\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr" Jul 17 01:14:49.701508 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.701468 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flt68\" (UniqueName: \"kubernetes.io/projected/c9c7d960-19bd-410d-aab3-f811fc713b28-kube-api-access-flt68\") pod \"monitoring-plugin-7b7b7bc8bf-cm6lr\" (UID: \"c9c7d960-19bd-410d-aab3-f811fc713b28\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr" Jul 17 01:14:49.701674 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.701543 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9c7d960-19bd-410d-aab3-f811fc713b28-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-cm6lr\" (UID: \"c9c7d960-19bd-410d-aab3-f811fc713b28\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr" Jul 17 01:14:49.705116 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.705068 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9c7d960-19bd-410d-aab3-f811fc713b28-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-cm6lr\" (UID: \"c9c7d960-19bd-410d-aab3-f811fc713b28\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr" Jul 17 01:14:49.705821 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.705745 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:49.714831 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.714770 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flt68\" (UniqueName: \"kubernetes.io/projected/c9c7d960-19bd-410d-aab3-f811fc713b28-kube-api-access-flt68\") pod \"monitoring-plugin-7b7b7bc8bf-cm6lr\" (UID: \"c9c7d960-19bd-410d-aab3-f811fc713b28\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr" Jul 17 01:14:49.714976 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.714882 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66c9bc58bf-q57jn"] Jul 17 01:14:49.754066 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:49.754037 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr" Jul 17 01:14:49.896356 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:49.896310 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6349637_3750_4429_a596_65de3a5824b6.slice/crio-d7f9c3edee33afeaa921ff86df0f5bdce59ebdb4d697b94fef3251808af5e6b4 WatchSource:0}: Error finding container d7f9c3edee33afeaa921ff86df0f5bdce59ebdb4d697b94fef3251808af5e6b4: Status 404 returned error can't find the container with id d7f9c3edee33afeaa921ff86df0f5bdce59ebdb4d697b94fef3251808af5e6b4 Jul 17 01:14:50.129658 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.129625 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:50.185307 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.185228 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:50.344260 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.344224 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:50.445425 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.445346 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:50.473831 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.473791 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kvmg6" event={"ID":"74496ef6-aa60-4933-ba21-7fc27f0f2549","Type":"ContainerStarted","Data":"5c365abd3eb18aafb6cf03f3404a60338fd34dd7c28b37d5612f7a1f1fbddb70"} Jul 17 01:14:50.473985 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.473859 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kvmg6" event={"ID":"74496ef6-aa60-4933-ba21-7fc27f0f2549","Type":"ContainerStarted","Data":"f14a140fe7fd305c99f06e5cc24c2d9d4c4154c7b8bb8fd7f59a2bc0416dfed1"} Jul 17 01:14:50.475530 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.475503 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" event={"ID":"a6349637-3750-4429-a596-65de3a5824b6","Type":"ContainerStarted","Data":"d7f9c3edee33afeaa921ff86df0f5bdce59ebdb4d697b94fef3251808af5e6b4"} Jul 17 01:14:50.480053 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.480028 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6dvf" event={"ID":"0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a","Type":"ContainerStarted","Data":"c173296fed4f1485e63fccf538e5c9ad3338a6dc125efcd232b9cb85204a2f55"} Jul 17 01:14:50.493669 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.493620 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kvmg6" podStartSLOduration=4.96848113 podStartE2EDuration="6.49360393s" totalImagesPullingTime="1.4113564s" totalInitContainerRuntime="113.7664ms" isStatefulPod=true podCreationTimestamp="2026-07-17 01:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:50.492028311 +0000 UTC m=+47.887643162" watchObservedRunningTime="2026-07-17 01:14:50.49360393 +0000 UTC m=+47.889218839" Jul 17 01:14:50.646153 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.646125 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:50.720753 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.720648 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:50.879935 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.879877 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w6dvf" podStartSLOduration=8.054814083 podStartE2EDuration="47.879859381s" totalImagesPullingTime="39.825045298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 01:14:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:50.511513361 +0000 UTC m=+47.907128222" watchObservedRunningTime="2026-07-17 01:14:50.879859381 +0000 UTC m=+48.275474231" Jul 17 01:14:50.880675 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.880651 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 01:14:50.880795 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.880696 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:50.906205 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.906172 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 01:14:50.906366 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.906310 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:50.908791 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.908766 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 01:14:50.908927 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.908899 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 01:14:50.909206 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.909185 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 01:14:50.909206 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.909188 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 01:14:50.909372 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.909244 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7wdq9\"" Jul 17 01:14:50.910311 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.910289 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 01:14:50.911118 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.911076 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 01:14:50.911369 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.911352 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 01:14:50.911786 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.911766 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 01:14:50.912167 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.912148 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 01:14:50.912394 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.912374 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 01:14:50.913030 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.913009 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-82c5h8eb2lbh3\"" Jul 17 01:14:50.913218 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.913201 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 01:14:50.918144 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.917866 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 01:14:50.918283 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.918200 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 01:14:50.928484 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:50.928456 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 01:14:51.016930 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.016848 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.016930 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.016893 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.016930 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.016923 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.016963 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017001 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20a4378a-66b2-4eed-b212-96e0a9708d06-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017028 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017062 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017088 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-config\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017182 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017425 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017297 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017425 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017334 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017425 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017358 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017425 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017375 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017425 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017411 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017447 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-web-config\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017472 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20a4378a-66b2-4eed-b212-96e0a9708d06-config-out\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017525 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017549 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017591 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.017765 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.017620 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n675\" (UniqueName: \"kubernetes.io/projected/20a4378a-66b2-4eed-b212-96e0a9708d06-kube-api-access-7n675\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.118557 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118522 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-config\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.118557 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118562 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.118796 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118611 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.118796 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118637 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.118796 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118666 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.118796 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118692 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.118796 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118730 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.118796 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118770 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-web-config\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.118796 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118800 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20a4378a-66b2-4eed-b212-96e0a9708d06-config-out\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.119157 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118817 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.119157 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118834 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.119157 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118861 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.119157 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118880 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n675\" (UniqueName: \"kubernetes.io/projected/20a4378a-66b2-4eed-b212-96e0a9708d06-kube-api-access-7n675\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.119157 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118938 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.119157 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118962 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.119157 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.118987 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.119157 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.119014 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.119157 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.119033 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20a4378a-66b2-4eed-b212-96e0a9708d06-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.119157 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.119049 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.119157 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.119081 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.119734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.119627 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.120149 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.119922 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.120149 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.119999 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.120149 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.120034 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.121312 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.121284 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.121880 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.121851 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-config\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.121967 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.121892 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.123204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.123179 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.123601 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.123577 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-web-config\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.123759 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.123743 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20a4378a-66b2-4eed-b212-96e0a9708d06-config-out\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.123814 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.123754 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.123953 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.123932 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.124488 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.124468 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.125446 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.125417 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.125446 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.125430 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20a4378a-66b2-4eed-b212-96e0a9708d06-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.126533 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.126511 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.126965 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.126939 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.127076 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.126974 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.127203 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.127130 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.129847 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.129825 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n675\" (UniqueName: \"kubernetes.io/projected/20a4378a-66b2-4eed-b212-96e0a9708d06-kube-api-access-7n675\") pod \"prometheus-k8s-0\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.226632 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.226598 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:14:51.481362 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.481329 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:51.588193 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.588157 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:51.682363 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:51.682334 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:52.923516 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:52.923471 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:52.972237 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:52.972188 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:53.754994 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:53.754960 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr"] Jul 17 01:14:54.312039 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:54.311997 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:54.412326 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:54.412289 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:54.516212 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:54.516178 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c7d960_19bd_410d_aab3_f811fc713b28.slice/crio-e24e8bc6b7159e8c4df284d5996b153319bc0a03c470a3b09403b0d7da7548db WatchSource:0}: Error finding container e24e8bc6b7159e8c4df284d5996b153319bc0a03c470a3b09403b0d7da7548db: Status 404 returned error can't find the container with id e24e8bc6b7159e8c4df284d5996b153319bc0a03c470a3b09403b0d7da7548db Jul 17 01:14:54.652090 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:54.652060 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 01:14:54.685406 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:14:54.685339 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a4378a_66b2_4eed_b212_96e0a9708d06.slice/crio-2fd9d04a300e0974195f1419cef354b376d4548b2519afa4107367f2c683fb41 WatchSource:0}: Error finding container 2fd9d04a300e0974195f1419cef354b376d4548b2519afa4107367f2c683fb41: Status 404 returned error can't find the container with id 2fd9d04a300e0974195f1419cef354b376d4548b2519afa4107367f2c683fb41 Jul 17 01:14:55.133030 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.133000 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:55.233543 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.233472 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:55.500899 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.500254 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" event={"ID":"5cf44875-46d9-4e1a-b594-87fd5ebe44b0","Type":"ContainerStarted","Data":"f753c5c0af54ab5a3a524c8015afeb59279e273516633490a33b4137491495f4"} Jul 17 01:14:55.502236 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.502197 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" event={"ID":"a6349637-3750-4429-a596-65de3a5824b6","Type":"ContainerStarted","Data":"eca8923f58794fdb7adda1587f7ad7d64bf7cfe462f740dc9432deed39e847c7"} Jul 17 01:14:55.504758 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.504724 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q9bjq" event={"ID":"e9ee2220-5d23-4a71-b795-7ff9e1a57868","Type":"ContainerStarted","Data":"b8f700c9c95d3c46a697947944c0d2875f0b6081720d0f5ffb7fe8e706731396"} Jul 17 01:14:55.506447 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.506418 2583 generic.go:356] "Generic (PLEG): container finished" podID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerID="ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582" exitCode=0 Jul 17 01:14:55.506579 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.506498 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerDied","Data":"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582"} Jul 17 01:14:55.506579 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.506566 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerStarted","Data":"2fd9d04a300e0974195f1419cef354b376d4548b2519afa4107367f2c683fb41"} Jul 17 01:14:55.509978 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.509952 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" event={"ID":"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb","Type":"ContainerStarted","Data":"996c55f04bc34465236196054f8282ec42a027ee55be5070f52d96d09e468ca0"} Jul 17 01:14:55.510055 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.510001 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" event={"ID":"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb","Type":"ContainerStarted","Data":"51d5bb7f0b7aef23e63471120fc1d4504c847f68bb8f357ec90e95dd18034f3f"} Jul 17 01:14:55.510055 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.510028 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" event={"ID":"f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb","Type":"ContainerStarted","Data":"0f68182ba9adc72dc3a1c674d89bf6189f02e4a827700b9d7e7b8463f2eea3a7"} Jul 17 01:14:55.511394 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.511367 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr" event={"ID":"c9c7d960-19bd-410d-aab3-f811fc713b28","Type":"ContainerStarted","Data":"e24e8bc6b7159e8c4df284d5996b153319bc0a03c470a3b09403b0d7da7548db"} Jul 17 01:14:55.513685 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.513661 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" event={"ID":"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7","Type":"ContainerStarted","Data":"f84414a18ac77e4f896abe9be447a601b6253f6cab882b0fa37973632d450c1d"} Jul 17 01:14:55.513810 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.513723 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" event={"ID":"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7","Type":"ContainerStarted","Data":"c8de0afdb815c0ae1da181170066bc4a2671881cf4501a287306121a159e956f"} Jul 17 01:14:55.513810 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.513765 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" event={"ID":"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7","Type":"ContainerStarted","Data":"d377650946fb3c058b7b45741aeac70d0bae8f3abf1370d51ad347613a6ae8b7"} Jul 17 01:14:55.515691 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.515621 2583 generic.go:356] "Generic (PLEG): container finished" podID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerID="66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c" exitCode=0 Jul 17 01:14:55.515691 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.515665 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerDied","Data":"66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c"} Jul 17 01:14:55.520799 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.520735 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nfzqk" podStartSLOduration=4.948389454 podStartE2EDuration="11.520717352s" totalImagesPullingTime="6.572327898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:55.520371916 +0000 UTC m=+52.915986767" watchObservedRunningTime="2026-07-17 01:14:55.520717352 +0000 UTC m=+52.916332203" Jul 17 01:14:55.545193 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.545140 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" podStartSLOduration=1.7901613950000002 podStartE2EDuration="6.545123071s" totalImagesPullingTime="4.754961676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:55.544492027 +0000 UTC m=+52.940106881" watchObservedRunningTime="2026-07-17 01:14:55.545123071 +0000 UTC m=+52.940737921" Jul 17 01:14:55.612255 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:55.612189 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-rcr67" podStartSLOduration=6.093362422 podStartE2EDuration="11.612164918s" totalImagesPullingTime="5.518802496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:55.611124916 +0000 UTC m=+53.006739758" watchObservedRunningTime="2026-07-17 01:14:55.612164918 +0000 UTC m=+53.007779771" Jul 17 01:14:56.503877 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:56.503841 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:56.597352 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:56.597312 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:57.525954 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:57.525915 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr" event={"ID":"c9c7d960-19bd-410d-aab3-f811fc713b28","Type":"ContainerStarted","Data":"b5893cde38e54e0ba8cd5e33233f53fc1a26da74b16129f90c14827aa30f7249"} Jul 17 01:14:57.526483 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:57.526137 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr" Jul 17 01:14:57.528705 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:57.528675 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" event={"ID":"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7","Type":"ContainerStarted","Data":"3192dec08eb6eba73fd841115d3614860ed58721894a8fc36df15fedf4a621de"} Jul 17 01:14:57.531510 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:57.531485 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr" Jul 17 01:14:57.541217 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:57.541163 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-cm6lr" podStartSLOduration=6.043127282 podStartE2EDuration="8.541147925s" totalImagesPullingTime="2.498020643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:14:57.540305778 +0000 UTC m=+54.935920655" watchObservedRunningTime="2026-07-17 01:14:57.541147925 +0000 UTC m=+54.936762776" Jul 17 01:14:58.318535 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:58.318491 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:58.417945 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:58.417903 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:58.471000 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:58.470967 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xjl9w" Jul 17 01:14:58.552479 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:58.552376 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:58.652360 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:58.652276 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:59.196157 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:59.196046 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:14:59.253729 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:14:59.253691 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:01.548851 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.548815 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" event={"ID":"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7","Type":"ContainerStarted","Data":"a63f67019c544fcecba1a0aec336459225d0e7a84e90eedece8ebd83113418f3"} Jul 17 01:15:01.549338 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.548885 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" event={"ID":"6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7","Type":"ContainerStarted","Data":"342924147f76cb1ba73f2e865735eb66e1f21b081420e6a7118e0ccf1a51de63"} Jul 17 01:15:01.549338 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.549038 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:15:01.552498 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.552473 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerStarted","Data":"a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093"} Jul 17 01:15:01.552657 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.552535 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerStarted","Data":"cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72"} Jul 17 01:15:01.552657 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.552568 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerStarted","Data":"b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93"} Jul 17 01:15:01.552657 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.552613 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerStarted","Data":"688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5"} Jul 17 01:15:01.552816 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.552661 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerStarted","Data":"9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3"} Jul 17 01:15:01.552816 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.552688 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerStarted","Data":"5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311"} Jul 17 01:15:01.554869 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.554842 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q9bjq" event={"ID":"e9ee2220-5d23-4a71-b795-7ff9e1a57868","Type":"ContainerStarted","Data":"aa7570838dd65d1cac84875cc2c87d5e03a20c62f4b81c86dfc6e2108cb45cc3"} Jul 17 01:15:01.555054 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.555037 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" Jul 17 01:15:01.558043 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.558023 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerStarted","Data":"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599"} Jul 17 01:15:01.558142 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.558072 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerStarted","Data":"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9"} Jul 17 01:15:01.558142 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.558115 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerStarted","Data":"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51"} Jul 17 01:15:01.558142 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.558142 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerStarted","Data":"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52"} Jul 17 01:15:01.558239 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.558169 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerStarted","Data":"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c"} Jul 17 01:15:01.558239 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.558193 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerStarted","Data":"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81"} Jul 17 01:15:01.575250 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.575203 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6449bf7ffd-plq9l" podStartSLOduration=6.993745582 podStartE2EDuration="15.575187621s" totalImagesPullingTime="8.581442039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:14:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:15:01.573202073 +0000 UTC m=+58.968816926" watchObservedRunningTime="2026-07-17 01:15:01.575187621 +0000 UTC m=+58.970802471" Jul 17 01:15:01.622198 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.622137 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.4522878089999995 podStartE2EDuration="11.62207314s" totalImagesPullingTime="5.151221322s" totalInitContainerRuntime="18.564009ms" isStatefulPod=false podCreationTimestamp="2026-07-17 01:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:15:01.621388397 +0000 UTC m=+59.017003252" watchObservedRunningTime="2026-07-17 01:15:01.62207314 +0000 UTC m=+59.017687992" Jul 17 01:15:01.638684 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.638626 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q9bjq" podStartSLOduration=2.276718924 podStartE2EDuration="23.638609565s" totalImagesPullingTime="21.361890641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 01:14:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:15:01.637651003 +0000 UTC m=+59.033265855" watchObservedRunningTime="2026-07-17 01:15:01.638609565 +0000 UTC m=+59.034224394" Jul 17 01:15:01.662040 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:01.661993 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.026199995 podStartE2EDuration="16.661975972s" totalImagesPullingTime="10.618816396s" totalInitContainerRuntime="16.959581ms" isStatefulPod=false podCreationTimestamp="2026-07-17 01:14:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:15:01.660618551 +0000 UTC m=+59.056233423" watchObservedRunningTime="2026-07-17 01:15:01.661975972 +0000 UTC m=+59.057590822" Jul 17 01:15:03.759271 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:03.759225 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:03.814499 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:03.814473 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:04.717273 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:04.717245 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:04.796248 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:04.796217 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:04.914257 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:04.914225 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:05.015747 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:05.015668 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:05.381237 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:05.381212 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2w6g5" Jul 17 01:15:05.562949 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:05.562917 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:05.663415 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:05.663334 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:06.227560 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:06.227518 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:07.272575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.272531 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:07.806941 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.806905 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:07.858465 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.858436 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:07.882241 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.882205 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs\") pod \"network-metrics-daemon-9bx9f\" (UID: \"8dbc5f6f-120c-42f2-b27f-5644ab623431\") " pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:15:07.882364 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.882278 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:15:07.885063 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.885041 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 01:15:07.885421 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.885405 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 01:15:07.895656 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.895630 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a-original-pull-secret\") pod \"global-pull-secret-syncer-59v85\" (UID: \"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a\") " pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:15:07.895771 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.895640 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbc5f6f-120c-42f2-b27f-5644ab623431-metrics-certs\") pod \"network-metrics-daemon-9bx9f\" (UID: \"8dbc5f6f-120c-42f2-b27f-5644ab623431\") " pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:15:07.933253 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.933221 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:07.972550 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.972518 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-59v85" Jul 17 01:15:07.983188 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.983049 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtbh\" (UniqueName: \"kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh\") pod \"network-check-target-hjgm7\" (UID: \"5293dc76-7ef2-442f-9035-6403edb5a336\") " pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:15:07.985151 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.985125 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hzllc\"" Jul 17 01:15:07.985285 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.985271 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 01:15:07.985859 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.985825 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:07.993244 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.993222 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9bx9f" Jul 17 01:15:07.995810 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:07.995598 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 01:15:08.006636 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:08.006610 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtbh\" (UniqueName: \"kubernetes.io/projected/5293dc76-7ef2-442f-9035-6403edb5a336-kube-api-access-cvtbh\") pod \"network-check-target-hjgm7\" (UID: \"5293dc76-7ef2-442f-9035-6403edb5a336\") " pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:15:08.114328 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:08.114296 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-59v85"] Jul 17 01:15:08.118511 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:15:08.118479 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f7c662_3fa3_4691_ad0a_34f6bfdc4d9a.slice/crio-6dc6c25392fa44feb4bd3071685eaf742b5668e553604b75acbf22755d32b404 WatchSource:0}: Error finding container 6dc6c25392fa44feb4bd3071685eaf742b5668e553604b75acbf22755d32b404: Status 404 returned error can't find the container with id 6dc6c25392fa44feb4bd3071685eaf742b5668e553604b75acbf22755d32b404 Jul 17 01:15:08.133413 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:08.133384 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9bx9f"] Jul 17 01:15:08.136079 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:15:08.136047 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbc5f6f_120c_42f2_b27f_5644ab623431.slice/crio-8ac9c646e2ddce49217c55c64f031a48eca2f42a54d7a708be3372a2d03cf79b WatchSource:0}: Error finding container 8ac9c646e2ddce49217c55c64f031a48eca2f42a54d7a708be3372a2d03cf79b: Status 404 returned error can't find the container with id 8ac9c646e2ddce49217c55c64f031a48eca2f42a54d7a708be3372a2d03cf79b Jul 17 01:15:08.280267 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:08.280240 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dtqbv\"" Jul 17 01:15:08.288393 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:08.288366 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:15:08.420735 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:08.420702 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hjgm7"] Jul 17 01:15:08.423744 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:15:08.423717 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5293dc76_7ef2_442f_9035_6403edb5a336.slice/crio-b14c0a35562163be0be41a3f4e3ee35d9b4e0a6b815c647e3283167508665a2b WatchSource:0}: Error finding container b14c0a35562163be0be41a3f4e3ee35d9b4e0a6b815c647e3283167508665a2b: Status 404 returned error can't find the container with id b14c0a35562163be0be41a3f4e3ee35d9b4e0a6b815c647e3283167508665a2b Jul 17 01:15:08.588549 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:08.588456 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9bx9f" event={"ID":"8dbc5f6f-120c-42f2-b27f-5644ab623431","Type":"ContainerStarted","Data":"8ac9c646e2ddce49217c55c64f031a48eca2f42a54d7a708be3372a2d03cf79b"} Jul 17 01:15:08.589865 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:08.589830 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-59v85" event={"ID":"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a","Type":"ContainerStarted","Data":"6dc6c25392fa44feb4bd3071685eaf742b5668e553604b75acbf22755d32b404"} Jul 17 01:15:08.591598 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:08.591559 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hjgm7" event={"ID":"5293dc76-7ef2-442f-9035-6403edb5a336","Type":"ContainerStarted","Data":"b14c0a35562163be0be41a3f4e3ee35d9b4e0a6b815c647e3283167508665a2b"} Jul 17 01:15:09.077877 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:09.077842 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:09.495973 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:09.495935 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:15:09.496386 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:09.496051 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:15:09.599248 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:09.599162 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9bx9f" event={"ID":"8dbc5f6f-120c-42f2-b27f-5644ab623431","Type":"ContainerStarted","Data":"e216cdb4f005434b0f88a0f7bc6cda59d113d041d2f65fa3dcc8a9480b4cb11e"} Jul 17 01:15:09.599248 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:09.599228 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9bx9f" event={"ID":"8dbc5f6f-120c-42f2-b27f-5644ab623431","Type":"ContainerStarted","Data":"5e550b168f6461c7c3a49df681b36ac466d9e47e7bace07a6c23d8f72a4d2fe5"} Jul 17 01:15:09.614594 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:09.614535 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9bx9f" podStartSLOduration=65.513111 podStartE2EDuration="1m6.61451487s" totalImagesPullingTime="1.10140387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:15:09.613684547 +0000 UTC m=+67.009299397" watchObservedRunningTime="2026-07-17 01:15:09.61451487 +0000 UTC m=+67.010129720" Jul 17 01:15:09.704021 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:09.703985 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:09.760878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:09.760839 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:13.621159 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:13.621043 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hjgm7" event={"ID":"5293dc76-7ef2-442f-9035-6403edb5a336","Type":"ContainerStarted","Data":"201c81517e823e1c181b68e0a7f8c42f53f7ce2e16823d472e4cf51c4a03edae"} Jul 17 01:15:13.621159 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:13.621143 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:15:13.622509 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:13.622485 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-59v85" event={"ID":"71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a","Type":"ContainerStarted","Data":"bb146a3e686dac553e18f85e982d4d736220276d230f2c7fc34cce628a43f9be"} Jul 17 01:15:13.636372 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:13.636323 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hjgm7" podStartSLOduration=65.776655499 podStartE2EDuration="1m10.636309137s" totalImagesPullingTime="4.859653638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:15:13.635570398 +0000 UTC m=+71.031185250" watchObservedRunningTime="2026-07-17 01:15:13.636309137 +0000 UTC m=+71.031923988" Jul 17 01:15:13.649727 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:13.649684 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-59v85" podStartSLOduration=65.473975999 podStartE2EDuration="1m10.649670012s" totalImagesPullingTime="5.175694013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 01:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:15:13.648419371 +0000 UTC m=+71.044034227" watchObservedRunningTime="2026-07-17 01:15:13.649670012 +0000 UTC m=+71.045284863" Jul 17 01:15:15.207057 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:15.207023 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:15.307240 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:15.307202 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:15.509597 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:15.509514 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:17.757957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:17.757917 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:17.806824 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:17.806779 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:17.864467 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:17.864427 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:17.916484 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:17.916453 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:23.039801 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:23.039679 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:23.099802 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:23.099773 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:29.273189 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:29.273146 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:29.325405 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:29.325371 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:29.501844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:29.501818 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:15:29.505585 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:29.505563 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-66c9bc58bf-q57jn" Jul 17 01:15:29.703380 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:29.703355 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:30.730982 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:30.730951 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:30.786982 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:30.786946 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:34.884316 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:34.884243 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:34.939263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:34.939221 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:34.986508 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:34.986470 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 01:15:34.986812 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:34.986788 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="alertmanager" containerID="cri-o://5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311" gracePeriod=120 Jul 17 01:15:34.987072 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:34.986868 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="kube-rbac-proxy-metric" containerID="cri-o://cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72" gracePeriod=120 Jul 17 01:15:34.987072 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:34.986909 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="config-reloader" containerID="cri-o://9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3" gracePeriod=120 Jul 17 01:15:34.987072 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:34.986927 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="prom-label-proxy" containerID="cri-o://a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093" gracePeriod=120 Jul 17 01:15:34.987072 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:34.986885 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="kube-rbac-proxy" containerID="cri-o://b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93" gracePeriod=120 Jul 17 01:15:34.987072 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:34.986872 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="kube-rbac-proxy-web" containerID="cri-o://688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5" gracePeriod=120 Jul 17 01:15:35.025306 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:35.025256 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:35.079788 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:35.079751 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:35.617574 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:35.617488 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:35.668377 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:35.668341 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:35.717734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:35.717702 2583 generic.go:356] "Generic (PLEG): container finished" podID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerID="a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093" exitCode=0 Jul 17 01:15:35.717734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:35.717728 2583 generic.go:356] "Generic (PLEG): container finished" podID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerID="b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93" exitCode=0 Jul 17 01:15:35.717734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:35.717737 2583 generic.go:356] "Generic (PLEG): container finished" podID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerID="9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3" exitCode=0 Jul 17 01:15:35.717953 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:35.717746 2583 generic.go:356] "Generic (PLEG): container finished" podID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerID="5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311" exitCode=0 Jul 17 01:15:35.717953 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:35.717772 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerDied","Data":"a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093"} Jul 17 01:15:35.717953 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:35.717858 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerDied","Data":"b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93"} Jul 17 01:15:35.717953 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:35.717884 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerDied","Data":"9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3"} Jul 17 01:15:35.717953 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:35.717909 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerDied","Data":"5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311"} Jul 17 01:15:36.378473 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.378441 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:15:36.452768 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.452729 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6e149373-0eec-4dc0-bec2-0987683e6f2b\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " Jul 17 01:15:36.452952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.452781 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzn96\" (UniqueName: \"kubernetes.io/projected/6e149373-0eec-4dc0-bec2-0987683e6f2b-kube-api-access-mzn96\") pod \"6e149373-0eec-4dc0-bec2-0987683e6f2b\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " Jul 17 01:15:36.452952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.452821 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e149373-0eec-4dc0-bec2-0987683e6f2b-metrics-client-ca\") pod \"6e149373-0eec-4dc0-bec2-0987683e6f2b\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " Jul 17 01:15:36.452952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.452846 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-config-volume\") pod \"6e149373-0eec-4dc0-bec2-0987683e6f2b\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " Jul 17 01:15:36.452952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.452871 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-cluster-tls-config\") pod \"6e149373-0eec-4dc0-bec2-0987683e6f2b\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " Jul 17 01:15:36.452952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.452908 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"6e149373-0eec-4dc0-bec2-0987683e6f2b\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " Jul 17 01:15:36.453337 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.453267 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy\") pod \"6e149373-0eec-4dc0-bec2-0987683e6f2b\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " Jul 17 01:15:36.453337 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.453297 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e149373-0eec-4dc0-bec2-0987683e6f2b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6e149373-0eec-4dc0-bec2-0987683e6f2b" (UID: "6e149373-0eec-4dc0-bec2-0987683e6f2b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:15:36.453337 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.453309 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e149373-0eec-4dc0-bec2-0987683e6f2b-tls-assets\") pod \"6e149373-0eec-4dc0-bec2-0987683e6f2b\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " Jul 17 01:15:36.453488 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.453369 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-main-tls\") pod \"6e149373-0eec-4dc0-bec2-0987683e6f2b\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " Jul 17 01:15:36.453488 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.453409 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-web-config\") pod \"6e149373-0eec-4dc0-bec2-0987683e6f2b\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " Jul 17 01:15:36.453488 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.453462 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e149373-0eec-4dc0-bec2-0987683e6f2b-alertmanager-main-db\") pod \"6e149373-0eec-4dc0-bec2-0987683e6f2b\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " Jul 17 01:15:36.453803 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.453504 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e149373-0eec-4dc0-bec2-0987683e6f2b-config-out\") pod \"6e149373-0eec-4dc0-bec2-0987683e6f2b\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " Jul 17 01:15:36.453803 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.453532 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e149373-0eec-4dc0-bec2-0987683e6f2b-alertmanager-trusted-ca-bundle\") pod \"6e149373-0eec-4dc0-bec2-0987683e6f2b\" (UID: \"6e149373-0eec-4dc0-bec2-0987683e6f2b\") " Jul 17 01:15:36.454649 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.453848 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e149373-0eec-4dc0-bec2-0987683e6f2b-metrics-client-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:36.454649 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.454376 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e149373-0eec-4dc0-bec2-0987683e6f2b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6e149373-0eec-4dc0-bec2-0987683e6f2b" (UID: "6e149373-0eec-4dc0-bec2-0987683e6f2b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:15:36.454649 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.454381 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e149373-0eec-4dc0-bec2-0987683e6f2b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6e149373-0eec-4dc0-bec2-0987683e6f2b" (UID: "6e149373-0eec-4dc0-bec2-0987683e6f2b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:15:36.457146 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.457064 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6e149373-0eec-4dc0-bec2-0987683e6f2b" (UID: "6e149373-0eec-4dc0-bec2-0987683e6f2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:36.457366 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.457331 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6e149373-0eec-4dc0-bec2-0987683e6f2b" (UID: "6e149373-0eec-4dc0-bec2-0987683e6f2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:36.457366 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.457341 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e149373-0eec-4dc0-bec2-0987683e6f2b-kube-api-access-mzn96" (OuterVolumeSpecName: "kube-api-access-mzn96") pod "6e149373-0eec-4dc0-bec2-0987683e6f2b" (UID: "6e149373-0eec-4dc0-bec2-0987683e6f2b"). InnerVolumeSpecName "kube-api-access-mzn96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:15:36.457654 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.457629 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e149373-0eec-4dc0-bec2-0987683e6f2b-config-out" (OuterVolumeSpecName: "config-out") pod "6e149373-0eec-4dc0-bec2-0987683e6f2b" (UID: "6e149373-0eec-4dc0-bec2-0987683e6f2b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:15:36.458342 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.458305 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6e149373-0eec-4dc0-bec2-0987683e6f2b" (UID: "6e149373-0eec-4dc0-bec2-0987683e6f2b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:36.458593 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.458562 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-config-volume" (OuterVolumeSpecName: "config-volume") pod "6e149373-0eec-4dc0-bec2-0987683e6f2b" (UID: "6e149373-0eec-4dc0-bec2-0987683e6f2b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:36.458686 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.458628 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6e149373-0eec-4dc0-bec2-0987683e6f2b" (UID: "6e149373-0eec-4dc0-bec2-0987683e6f2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:36.458803 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.458787 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e149373-0eec-4dc0-bec2-0987683e6f2b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6e149373-0eec-4dc0-bec2-0987683e6f2b" (UID: "6e149373-0eec-4dc0-bec2-0987683e6f2b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:15:36.462225 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.462202 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6e149373-0eec-4dc0-bec2-0987683e6f2b" (UID: "6e149373-0eec-4dc0-bec2-0987683e6f2b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:36.469163 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.469126 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-web-config" (OuterVolumeSpecName: "web-config") pod "6e149373-0eec-4dc0-bec2-0987683e6f2b" (UID: "6e149373-0eec-4dc0-bec2-0987683e6f2b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:36.554442 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.554410 2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e149373-0eec-4dc0-bec2-0987683e6f2b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:36.554442 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.554437 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:36.554442 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.554447 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzn96\" (UniqueName: \"kubernetes.io/projected/6e149373-0eec-4dc0-bec2-0987683e6f2b-kube-api-access-mzn96\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:36.554667 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.554458 2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-config-volume\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:36.554667 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.554466 2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-cluster-tls-config\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:36.554667 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.554474 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:36.554667 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.554483 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:36.554667 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.554491 2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e149373-0eec-4dc0-bec2-0987683e6f2b-tls-assets\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:36.554667 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.554499 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-secret-alertmanager-main-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:36.554667 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.554508 2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e149373-0eec-4dc0-bec2-0987683e6f2b-web-config\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:36.554667 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.554515 2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6e149373-0eec-4dc0-bec2-0987683e6f2b-alertmanager-main-db\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:36.554667 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.554523 2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e149373-0eec-4dc0-bec2-0987683e6f2b-config-out\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:36.725598 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.725512 2583 generic.go:356] "Generic (PLEG): container finished" podID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerID="cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72" exitCode=0 Jul 17 01:15:36.725598 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.725539 2583 generic.go:356] "Generic (PLEG): container finished" podID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerID="688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5" exitCode=0 Jul 17 01:15:36.725772 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.725601 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 01:15:36.725772 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.725600 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerDied","Data":"cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72"} Jul 17 01:15:36.725772 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.725727 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerDied","Data":"688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5"} Jul 17 01:15:36.725772 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.725770 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6e149373-0eec-4dc0-bec2-0987683e6f2b","Type":"ContainerDied","Data":"73c9762f4558982382aed8fffaae637545bf5c8fc50e1ba1e24602b14c6cda8b"} Jul 17 01:15:36.725911 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.725805 2583 scope.go:117] "RemoveContainer" containerID="a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093" Jul 17 01:15:36.734661 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.734636 2583 scope.go:117] "RemoveContainer" containerID="cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72" Jul 17 01:15:36.742143 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.742123 2583 scope.go:117] "RemoveContainer" containerID="b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93" Jul 17 01:15:36.750113 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.750070 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 01:15:36.750624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.750609 2583 scope.go:117] "RemoveContainer" containerID="688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5" Jul 17 01:15:36.756463 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.756438 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 01:15:36.760963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.760936 2583 scope.go:117] "RemoveContainer" containerID="9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3" Jul 17 01:15:36.771747 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.771723 2583 scope.go:117] "RemoveContainer" containerID="5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311" Jul 17 01:15:36.780971 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.780938 2583 scope.go:117] "RemoveContainer" containerID="66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c" Jul 17 01:15:36.788871 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.788848 2583 scope.go:117] "RemoveContainer" containerID="a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093" Jul 17 01:15:36.789203 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:15:36.789177 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093\": container with ID starting with a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093 not found: ID does not exist" containerID="a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093" Jul 17 01:15:36.789301 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.789212 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093"} err="failed to get container status \"a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093\": rpc error: code = NotFound desc = could not find container \"a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093\": container with ID starting with a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093 not found: ID does not exist" Jul 17 01:15:36.789301 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.789247 2583 scope.go:117] "RemoveContainer" containerID="cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72" Jul 17 01:15:36.789506 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:15:36.789492 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72\": container with ID starting with cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72 not found: ID does not exist" containerID="cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72" Jul 17 01:15:36.789541 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.789511 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72"} err="failed to get container status \"cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72\": rpc error: code = NotFound desc = could not find container \"cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72\": container with ID starting with cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72 not found: ID does not exist" Jul 17 01:15:36.789541 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.789524 2583 scope.go:117] "RemoveContainer" containerID="b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93" Jul 17 01:15:36.789721 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:15:36.789707 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93\": container with ID starting with b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93 not found: ID does not exist" containerID="b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93" Jul 17 01:15:36.789756 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.789725 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93"} err="failed to get container status \"b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93\": rpc error: code = NotFound desc = could not find container \"b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93\": container with ID starting with b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93 not found: ID does not exist" Jul 17 01:15:36.789756 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.789738 2583 scope.go:117] "RemoveContainer" containerID="688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5" Jul 17 01:15:36.789909 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:15:36.789896 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5\": container with ID starting with 688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5 not found: ID does not exist" containerID="688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5" Jul 17 01:15:36.789940 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.789914 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5"} err="failed to get container status \"688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5\": rpc error: code = NotFound desc = could not find container \"688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5\": container with ID starting with 688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5 not found: ID does not exist" Jul 17 01:15:36.789940 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.789930 2583 scope.go:117] "RemoveContainer" containerID="9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3" Jul 17 01:15:36.790160 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:15:36.790137 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3\": container with ID starting with 9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3 not found: ID does not exist" containerID="9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3" Jul 17 01:15:36.790212 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.790168 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3"} err="failed to get container status \"9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3\": rpc error: code = NotFound desc = could not find container \"9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3\": container with ID starting with 9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3 not found: ID does not exist" Jul 17 01:15:36.790212 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.790190 2583 scope.go:117] "RemoveContainer" containerID="5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311" Jul 17 01:15:36.790414 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:15:36.790400 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311\": container with ID starting with 5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311 not found: ID does not exist" containerID="5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311" Jul 17 01:15:36.790469 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.790416 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311"} err="failed to get container status \"5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311\": rpc error: code = NotFound desc = could not find container \"5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311\": container with ID starting with 5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311 not found: ID does not exist" Jul 17 01:15:36.790469 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.790433 2583 scope.go:117] "RemoveContainer" containerID="66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c" Jul 17 01:15:36.790641 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:15:36.790621 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c\": container with ID starting with 66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c not found: ID does not exist" containerID="66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c" Jul 17 01:15:36.790678 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.790648 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c"} err="failed to get container status \"66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c\": rpc error: code = NotFound desc = could not find container \"66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c\": container with ID starting with 66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c not found: ID does not exist" Jul 17 01:15:36.790678 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.790666 2583 scope.go:117] "RemoveContainer" containerID="a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093" Jul 17 01:15:36.790921 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.790900 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093"} err="failed to get container status \"a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093\": rpc error: code = NotFound desc = could not find container \"a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093\": container with ID starting with a525cf697160c07b0fc5493111d74777e068c0f5d87385efae601e03adf63093 not found: ID does not exist" Jul 17 01:15:36.790921 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.790922 2583 scope.go:117] "RemoveContainer" containerID="cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72" Jul 17 01:15:36.791162 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.791144 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72"} err="failed to get container status \"cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72\": rpc error: code = NotFound desc = could not find container \"cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72\": container with ID starting with cc3ef2fb4510b048737915d85e92920e6fb958dbf742694e09e43ef74aa58a72 not found: ID does not exist" Jul 17 01:15:36.791207 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.791164 2583 scope.go:117] "RemoveContainer" containerID="b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93" Jul 17 01:15:36.791425 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.791393 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93"} err="failed to get container status \"b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93\": rpc error: code = NotFound desc = could not find container \"b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93\": container with ID starting with b5bb03aed05187d44ef3f200b9265689d66fed8901b484cbbb1efd2091330d93 not found: ID does not exist" Jul 17 01:15:36.791425 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.791416 2583 scope.go:117] "RemoveContainer" containerID="688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5" Jul 17 01:15:36.791653 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.791636 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5"} err="failed to get container status \"688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5\": rpc error: code = NotFound desc = could not find container \"688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5\": container with ID starting with 688005409d9208a8b8a932ae1f296572bf81640d8641f639e2637bc2e6d33ef5 not found: ID does not exist" Jul 17 01:15:36.791702 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.791655 2583 scope.go:117] "RemoveContainer" containerID="9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3" Jul 17 01:15:36.791857 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.791837 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3"} err="failed to get container status \"9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3\": rpc error: code = NotFound desc = could not find container \"9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3\": container with ID starting with 9cc8638450caccae877c05426c3b4ce51734bad4f8212cfa0dd2f7ab6e54aab3 not found: ID does not exist" Jul 17 01:15:36.791896 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.791858 2583 scope.go:117] "RemoveContainer" containerID="5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311" Jul 17 01:15:36.792038 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.792023 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311"} err="failed to get container status \"5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311\": rpc error: code = NotFound desc = could not find container \"5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311\": container with ID starting with 5d2dc63f2b64c44f3c5cb912967f96cdc276293b405e19bc73d43172d7253311 not found: ID does not exist" Jul 17 01:15:36.792076 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.792039 2583 scope.go:117] "RemoveContainer" containerID="66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c" Jul 17 01:15:36.792278 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:36.792262 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c"} err="failed to get container status \"66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c\": rpc error: code = NotFound desc = could not find container \"66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c\": container with ID starting with 66ecf6096570e5cea2b99008d32c2720f1f908ab1c7ca0e336e3cb82172d372c not found: ID does not exist" Jul 17 01:15:37.169460 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:37.169427 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" path="/var/lib/kubelet/pods/6e149373-0eec-4dc0-bec2-0987683e6f2b/volumes" Jul 17 01:15:39.293485 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.293455 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 01:15:39.293900 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.293748 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="prometheus" containerID="cri-o://8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81" gracePeriod=600 Jul 17 01:15:39.293900 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.293762 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="kube-rbac-proxy" containerID="cri-o://5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9" gracePeriod=600 Jul 17 01:15:39.293900 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.293800 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="kube-rbac-proxy-web" containerID="cri-o://ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51" gracePeriod=600 Jul 17 01:15:39.293900 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.293806 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="thanos-sidecar" containerID="cri-o://2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52" gracePeriod=600 Jul 17 01:15:39.293900 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.293808 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="config-reloader" containerID="cri-o://d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c" gracePeriod=600 Jul 17 01:15:39.294173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.293919 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="kube-rbac-proxy-thanos" containerID="cri-o://822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599" gracePeriod=600 Jul 17 01:15:39.563756 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.563720 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.579637 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.579603 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-1\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.579828 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.579659 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-thanos-prometheus-http-client-file\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.579828 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.579691 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20a4378a-66b2-4eed-b212-96e0a9708d06-config-out\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.579828 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.579719 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-2\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.579828 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.579751 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-kube-rbac-proxy\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.579828 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.579773 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n675\" (UniqueName: \"kubernetes.io/projected/20a4378a-66b2-4eed-b212-96e0a9708d06-kube-api-access-7n675\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.579828 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.579801 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-grpc-tls\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.580147 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.579864 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.580147 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.579894 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20a4378a-66b2-4eed-b212-96e0a9708d06-tls-assets\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.580147 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.579924 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-trusted-ca-bundle\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.580147 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.579965 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-web-config\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.580147 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.579998 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-serving-certs-ca-bundle\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.580147 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.580030 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-tls\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.580147 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.580056 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.580147 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.580091 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-metrics-client-ca\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.580523 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.580184 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-config\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.580523 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.580213 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-kubelet-serving-ca-bundle\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.580523 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.580238 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-db\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.580523 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.580261 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-metrics-client-certs\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.580523 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.580285 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-0\") pod \"20a4378a-66b2-4eed-b212-96e0a9708d06\" (UID: \"20a4378a-66b2-4eed-b212-96e0a9708d06\") " Jul 17 01:15:39.580523 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.580444 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:15:39.580798 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.580634 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:15:39.580798 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.580699 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.582083 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.582046 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:15:39.582655 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.582629 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:15:39.582993 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.582968 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:15:39.583552 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.583522 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:15:39.584117 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.583937 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:39.584327 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.584306 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:39.585087 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.584940 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:15:39.585337 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.585308 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:39.585414 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.585382 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:39.585803 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.585635 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a4378a-66b2-4eed-b212-96e0a9708d06-kube-api-access-7n675" (OuterVolumeSpecName: "kube-api-access-7n675") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "kube-api-access-7n675". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:15:39.585803 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.585677 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:15:39.586074 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.586046 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a4378a-66b2-4eed-b212-96e0a9708d06-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:15:39.586294 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.586256 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a4378a-66b2-4eed-b212-96e0a9708d06-config-out" (OuterVolumeSpecName: "config-out") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 01:15:39.587747 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.587405 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:39.591284 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.589374 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-config" (OuterVolumeSpecName: "config") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:39.602277 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.602198 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:39.613342 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.613293 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:39.619401 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.619357 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-web-config" (OuterVolumeSpecName: "web-config") pod "20a4378a-66b2-4eed-b212-96e0a9708d06" (UID: "20a4378a-66b2-4eed-b212-96e0a9708d06"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:15:39.681673 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681628 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.681673 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681663 2583 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.681673 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681675 2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20a4378a-66b2-4eed-b212-96e0a9708d06-config-out\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.681673 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681684 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681694 2583 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-kube-rbac-proxy\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681709 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n675\" (UniqueName: \"kubernetes.io/projected/20a4378a-66b2-4eed-b212-96e0a9708d06-kube-api-access-7n675\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681719 2583 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-grpc-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681728 2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681738 2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20a4378a-66b2-4eed-b212-96e0a9708d06-tls-assets\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681746 2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-web-config\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681754 2583 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681762 2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681770 2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681782 2583 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-metrics-client-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681794 2583 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-config\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681802 2583 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681811 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-db\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681819 2583 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/20a4378a-66b2-4eed-b212-96e0a9708d06-secret-metrics-client-certs\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.682016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.681827 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/20a4378a-66b2-4eed-b212-96e0a9708d06-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:15:39.749339 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749297 2583 generic.go:356] "Generic (PLEG): container finished" podID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerID="822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599" exitCode=0 Jul 17 01:15:39.749339 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749335 2583 generic.go:356] "Generic (PLEG): container finished" podID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerID="5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9" exitCode=0 Jul 17 01:15:39.749339 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749348 2583 generic.go:356] "Generic (PLEG): container finished" podID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerID="ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51" exitCode=0 Jul 17 01:15:39.749575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749361 2583 generic.go:356] "Generic (PLEG): container finished" podID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerID="2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52" exitCode=0 Jul 17 01:15:39.749575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749370 2583 generic.go:356] "Generic (PLEG): container finished" podID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerID="d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c" exitCode=0 Jul 17 01:15:39.749575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749379 2583 generic.go:356] "Generic (PLEG): container finished" podID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerID="8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81" exitCode=0 Jul 17 01:15:39.749575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749387 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.749575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749380 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerDied","Data":"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599"} Jul 17 01:15:39.749575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749554 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerDied","Data":"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9"} Jul 17 01:15:39.749762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749582 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerDied","Data":"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51"} Jul 17 01:15:39.749762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749610 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerDied","Data":"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52"} Jul 17 01:15:39.749762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749622 2583 scope.go:117] "RemoveContainer" containerID="822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599" Jul 17 01:15:39.749762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749633 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerDied","Data":"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c"} Jul 17 01:15:39.749762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749657 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerDied","Data":"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81"} Jul 17 01:15:39.749762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.749681 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"20a4378a-66b2-4eed-b212-96e0a9708d06","Type":"ContainerDied","Data":"2fd9d04a300e0974195f1419cef354b376d4548b2519afa4107367f2c683fb41"} Jul 17 01:15:39.765333 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.762007 2583 scope.go:117] "RemoveContainer" containerID="5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9" Jul 17 01:15:39.774321 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.774298 2583 scope.go:117] "RemoveContainer" containerID="ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51" Jul 17 01:15:39.783476 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.783442 2583 scope.go:117] "RemoveContainer" containerID="2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52" Jul 17 01:15:39.783812 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.783788 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 01:15:39.787826 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.787798 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 01:15:39.791812 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.791790 2583 scope.go:117] "RemoveContainer" containerID="d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c" Jul 17 01:15:39.800506 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.800481 2583 scope.go:117] "RemoveContainer" containerID="8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81" Jul 17 01:15:39.809410 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.809385 2583 scope.go:117] "RemoveContainer" containerID="ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821048 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821465 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="thanos-sidecar" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821484 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="thanos-sidecar" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821500 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="prom-label-proxy" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821509 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="prom-label-proxy" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821523 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="config-reloader" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821532 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="config-reloader" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821544 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="kube-rbac-proxy-web" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821553 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="kube-rbac-proxy-web" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821567 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="config-reloader" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821576 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="config-reloader" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821585 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="kube-rbac-proxy-web" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821594 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="kube-rbac-proxy-web" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821606 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="kube-rbac-proxy" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821616 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="kube-rbac-proxy" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821628 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="init-config-reloader" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821637 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="init-config-reloader" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821647 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="prometheus" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821656 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="prometheus" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821666 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="kube-rbac-proxy" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821674 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="kube-rbac-proxy" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821682 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="kube-rbac-proxy-metric" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821690 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="kube-rbac-proxy-metric" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821703 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="init-config-reloader" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821712 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="init-config-reloader" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821729 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="alertmanager" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821740 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="alertmanager" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821755 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="kube-rbac-proxy-thanos" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821764 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="kube-rbac-proxy-thanos" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821829 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="kube-rbac-proxy-web" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821844 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="thanos-sidecar" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821854 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="kube-rbac-proxy-metric" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821863 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="prom-label-proxy" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821873 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="kube-rbac-proxy-web" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821884 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="kube-rbac-proxy" Jul 17 01:15:39.830762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821893 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="kube-rbac-proxy-thanos" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821901 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="prometheus" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821908 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="kube-rbac-proxy" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821917 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" containerName="config-reloader" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821925 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="alertmanager" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.821933 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e149373-0eec-4dc0-bec2-0987683e6f2b" containerName="config-reloader" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.824689 2583 scope.go:117] "RemoveContainer" containerID="822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:15:39.825089 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599\": container with ID starting with 822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599 not found: ID does not exist" containerID="822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.825145 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599"} err="failed to get container status \"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599\": rpc error: code = NotFound desc = could not find container \"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599\": container with ID starting with 822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599 not found: ID does not exist" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.825172 2583 scope.go:117] "RemoveContainer" containerID="5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:15:39.825479 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9\": container with ID starting with 5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9 not found: ID does not exist" containerID="5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.825509 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9"} err="failed to get container status \"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9\": rpc error: code = NotFound desc = could not find container \"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9\": container with ID starting with 5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9 not found: ID does not exist" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.825536 2583 scope.go:117] "RemoveContainer" containerID="ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:15:39.825816 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51\": container with ID starting with ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51 not found: ID does not exist" containerID="ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.825846 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51"} err="failed to get container status \"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51\": rpc error: code = NotFound desc = could not find container \"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51\": container with ID starting with ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51 not found: ID does not exist" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.825869 2583 scope.go:117] "RemoveContainer" containerID="2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:15:39.826140 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52\": container with ID starting with 2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52 not found: ID does not exist" containerID="2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.826166 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52"} err="failed to get container status \"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52\": rpc error: code = NotFound desc = could not find container \"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52\": container with ID starting with 2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52 not found: ID does not exist" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.826185 2583 scope.go:117] "RemoveContainer" containerID="d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c" Jul 17 01:15:39.832564 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:15:39.826420 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c\": container with ID starting with d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c not found: ID does not exist" containerID="d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.826444 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c"} err="failed to get container status \"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c\": rpc error: code = NotFound desc = could not find container \"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c\": container with ID starting with d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c not found: ID does not exist" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.826463 2583 scope.go:117] "RemoveContainer" containerID="8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:15:39.826684 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81\": container with ID starting with 8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81 not found: ID does not exist" containerID="8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.826707 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81"} err="failed to get container status \"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81\": rpc error: code = NotFound desc = could not find container \"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81\": container with ID starting with 8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81 not found: ID does not exist" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.826726 2583 scope.go:117] "RemoveContainer" containerID="ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:15:39.826961 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582\": container with ID starting with ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582 not found: ID does not exist" containerID="ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.826987 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582"} err="failed to get container status \"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582\": rpc error: code = NotFound desc = could not find container \"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582\": container with ID starting with ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582 not found: ID does not exist" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.827009 2583 scope.go:117] "RemoveContainer" containerID="822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.827239 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599"} err="failed to get container status \"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599\": rpc error: code = NotFound desc = could not find container \"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599\": container with ID starting with 822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599 not found: ID does not exist" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.827260 2583 scope.go:117] "RemoveContainer" containerID="5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.827472 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9"} err="failed to get container status \"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9\": rpc error: code = NotFound desc = could not find container \"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9\": container with ID starting with 5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9 not found: ID does not exist" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.827492 2583 scope.go:117] "RemoveContainer" containerID="ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.827714 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51"} err="failed to get container status \"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51\": rpc error: code = NotFound desc = could not find container \"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51\": container with ID starting with ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51 not found: ID does not exist" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.827735 2583 scope.go:117] "RemoveContainer" containerID="2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.827946 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52"} err="failed to get container status \"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52\": rpc error: code = NotFound desc = could not find container \"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52\": container with ID starting with 2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52 not found: ID does not exist" Jul 17 01:15:39.833340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.827970 2583 scope.go:117] "RemoveContainer" containerID="d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.828210 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c"} err="failed to get container status \"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c\": rpc error: code = NotFound desc = could not find container \"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c\": container with ID starting with d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c not found: ID does not exist" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.828233 2583 scope.go:117] "RemoveContainer" containerID="8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.828452 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81"} err="failed to get container status \"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81\": rpc error: code = NotFound desc = could not find container \"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81\": container with ID starting with 8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81 not found: ID does not exist" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.828473 2583 scope.go:117] "RemoveContainer" containerID="ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.828688 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582"} err="failed to get container status \"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582\": rpc error: code = NotFound desc = could not find container \"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582\": container with ID starting with ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582 not found: ID does not exist" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.828709 2583 scope.go:117] "RemoveContainer" containerID="822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.828940 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599"} err="failed to get container status \"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599\": rpc error: code = NotFound desc = could not find container \"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599\": container with ID starting with 822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599 not found: ID does not exist" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.828961 2583 scope.go:117] "RemoveContainer" containerID="5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.829207 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9"} err="failed to get container status \"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9\": rpc error: code = NotFound desc = could not find container \"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9\": container with ID starting with 5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9 not found: ID does not exist" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.829231 2583 scope.go:117] "RemoveContainer" containerID="ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.829443 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51"} err="failed to get container status \"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51\": rpc error: code = NotFound desc = could not find container \"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51\": container with ID starting with ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51 not found: ID does not exist" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.829464 2583 scope.go:117] "RemoveContainer" containerID="2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.829678 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52"} err="failed to get container status \"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52\": rpc error: code = NotFound desc = could not find container \"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52\": container with ID starting with 2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52 not found: ID does not exist" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.829700 2583 scope.go:117] "RemoveContainer" containerID="d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.829915 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c"} err="failed to get container status \"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c\": rpc error: code = NotFound desc = could not find container \"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c\": container with ID starting with d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c not found: ID does not exist" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.829939 2583 scope.go:117] "RemoveContainer" containerID="8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81" Jul 17 01:15:39.833957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.830206 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81"} err="failed to get container status \"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81\": rpc error: code = NotFound desc = could not find container \"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81\": container with ID starting with 8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81 not found: ID does not exist" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.830229 2583 scope.go:117] "RemoveContainer" containerID="ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.830499 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582"} err="failed to get container status \"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582\": rpc error: code = NotFound desc = could not find container \"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582\": container with ID starting with ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582 not found: ID does not exist" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.830525 2583 scope.go:117] "RemoveContainer" containerID="822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.830746 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599"} err="failed to get container status \"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599\": rpc error: code = NotFound desc = could not find container \"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599\": container with ID starting with 822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599 not found: ID does not exist" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.830767 2583 scope.go:117] "RemoveContainer" containerID="5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.831022 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9"} err="failed to get container status \"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9\": rpc error: code = NotFound desc = could not find container \"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9\": container with ID starting with 5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9 not found: ID does not exist" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.831056 2583 scope.go:117] "RemoveContainer" containerID="ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.831435 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51"} err="failed to get container status \"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51\": rpc error: code = NotFound desc = could not find container \"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51\": container with ID starting with ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51 not found: ID does not exist" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.831451 2583 scope.go:117] "RemoveContainer" containerID="2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.831629 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.831690 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52"} err="failed to get container status \"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52\": rpc error: code = NotFound desc = could not find container \"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52\": container with ID starting with 2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52 not found: ID does not exist" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.831719 2583 scope.go:117] "RemoveContainer" containerID="d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.831998 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c"} err="failed to get container status \"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c\": rpc error: code = NotFound desc = could not find container \"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c\": container with ID starting with d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c not found: ID does not exist" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.832020 2583 scope.go:117] "RemoveContainer" containerID="8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.832418 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81"} err="failed to get container status \"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81\": rpc error: code = NotFound desc = could not find container \"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81\": container with ID starting with 8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81 not found: ID does not exist" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.832439 2583 scope.go:117] "RemoveContainer" containerID="ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.832687 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582"} err="failed to get container status \"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582\": rpc error: code = NotFound desc = could not find container \"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582\": container with ID starting with ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582 not found: ID does not exist" Jul 17 01:15:39.834554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.832716 2583 scope.go:117] "RemoveContainer" containerID="822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.832977 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599"} err="failed to get container status \"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599\": rpc error: code = NotFound desc = could not find container \"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599\": container with ID starting with 822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599 not found: ID does not exist" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.832993 2583 scope.go:117] "RemoveContainer" containerID="5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.833287 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9"} err="failed to get container status \"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9\": rpc error: code = NotFound desc = could not find container \"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9\": container with ID starting with 5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9 not found: ID does not exist" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.833304 2583 scope.go:117] "RemoveContainer" containerID="ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.833567 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51"} err="failed to get container status \"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51\": rpc error: code = NotFound desc = could not find container \"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51\": container with ID starting with ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51 not found: ID does not exist" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.833591 2583 scope.go:117] "RemoveContainer" containerID="2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.833826 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52"} err="failed to get container status \"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52\": rpc error: code = NotFound desc = could not find container \"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52\": container with ID starting with 2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52 not found: ID does not exist" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.833886 2583 scope.go:117] "RemoveContainer" containerID="d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.834083 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c"} err="failed to get container status \"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c\": rpc error: code = NotFound desc = could not find container \"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c\": container with ID starting with d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c not found: ID does not exist" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.834116 2583 scope.go:117] "RemoveContainer" containerID="8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.834484 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81"} err="failed to get container status \"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81\": rpc error: code = NotFound desc = could not find container \"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81\": container with ID starting with 8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81 not found: ID does not exist" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.834499 2583 scope.go:117] "RemoveContainer" containerID="ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.834582 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.834898 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582"} err="failed to get container status \"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582\": rpc error: code = NotFound desc = could not find container \"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582\": container with ID starting with ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582 not found: ID does not exist" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.834926 2583 scope.go:117] "RemoveContainer" containerID="822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.835004 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.835014 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.835264 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599"} err="failed to get container status \"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599\": rpc error: code = NotFound desc = could not find container \"822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599\": container with ID starting with 822c794b398ba4090692a97c05a03220cebf9d5d92f08ede1b4a4e025ac7b599 not found: ID does not exist" Jul 17 01:15:39.835825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.835282 2583 scope.go:117] "RemoveContainer" containerID="5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9" Jul 17 01:15:39.836761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.835365 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 01:15:39.836761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.835371 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 01:15:39.836761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.835570 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7wdq9\"" Jul 17 01:15:39.836761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.835836 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9"} err="failed to get container status \"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9\": rpc error: code = NotFound desc = could not find container \"5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9\": container with ID starting with 5a76ab4db0a400eb13afa5bc07cf90ae5b74144aab0424fa0bf4aa7e80cd2bb9 not found: ID does not exist" Jul 17 01:15:39.836761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.836019 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 01:15:39.836761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.836062 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 01:15:39.836761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.836006 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 01:15:39.836761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.836290 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 01:15:39.836761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.836453 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-82c5h8eb2lbh3\"" Jul 17 01:15:39.836761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.835860 2583 scope.go:117] "RemoveContainer" containerID="ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51" Jul 17 01:15:39.839850 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.837943 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 01:15:39.839850 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.838400 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 01:15:39.839850 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.838649 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 01:15:39.839850 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.838868 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 01:15:39.839850 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.839499 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51"} err="failed to get container status \"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51\": rpc error: code = NotFound desc = could not find container \"ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51\": container with ID starting with ca47ccc7c17612d82689d0dfdcb5bee6bb9964c9c860b5bf445cf3b4329aad51 not found: ID does not exist" Jul 17 01:15:39.839850 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.839525 2583 scope.go:117] "RemoveContainer" containerID="2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52" Jul 17 01:15:39.839850 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.839816 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52"} err="failed to get container status \"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52\": rpc error: code = NotFound desc = could not find container \"2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52\": container with ID starting with 2c0f7b976b71cce649390fada86d93110756f26607453b973fc935a596d71a52 not found: ID does not exist" Jul 17 01:15:39.839850 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.839841 2583 scope.go:117] "RemoveContainer" containerID="d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c" Jul 17 01:15:39.840272 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.840024 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 01:15:39.840272 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.840188 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c"} err="failed to get container status \"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c\": rpc error: code = NotFound desc = could not find container \"d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c\": container with ID starting with d088ec7ae7a635b2bd47791ddbe057b5d89017fc41783dd209df4e8bac70a12c not found: ID does not exist" Jul 17 01:15:39.840272 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.840211 2583 scope.go:117] "RemoveContainer" containerID="8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81" Jul 17 01:15:39.840547 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.840519 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81"} err="failed to get container status \"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81\": rpc error: code = NotFound desc = could not find container \"8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81\": container with ID starting with 8efb592b2ac0ea4889efe27c76ea99b1b7ea85f957edacc3e592dc2bba0eac81 not found: ID does not exist" Jul 17 01:15:39.841005 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.840550 2583 scope.go:117] "RemoveContainer" containerID="ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582" Jul 17 01:15:39.841005 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.840965 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582"} err="failed to get container status \"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582\": rpc error: code = NotFound desc = could not find container \"ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582\": container with ID starting with ab9152b9dc004b6226bfad9c24ccfcbaa5708f3b59cf232d7345ce62ac175582 not found: ID does not exist" Jul 17 01:15:39.842242 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.842222 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.883700 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.883781 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.883815 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v9c7\" (UniqueName: \"kubernetes.io/projected/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-kube-api-access-2v9c7\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.883858 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-web-config\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.883902 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.883946 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.883972 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.884026 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.884052 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.884134 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-config-out\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.884163 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.884211 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.884238 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.884280 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.884316 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.884359 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885847 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.884390 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885847 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.884435 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-config\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885847 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.884462 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.885847 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.884531 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.985159 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985054 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.985330 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985184 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.985330 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985220 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.985464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985434 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.985520 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985469 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.985520 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985503 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.985611 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985532 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-config\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.985665 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985630 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.985719 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985685 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.985799 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985756 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.985799 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985793 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.985900 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985832 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.985900 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985859 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v9c7\" (UniqueName: \"kubernetes.io/projected/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-kube-api-access-2v9c7\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.985900 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985884 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-web-config\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.986042 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985931 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.986042 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985958 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.986042 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.985984 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.986042 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.986017 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.986042 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.986040 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.986301 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.986077 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.986301 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.986089 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-config-out\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.986301 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.986153 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.986805 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.986775 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.989190 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.989147 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.989520 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.989493 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-config-out\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.990187 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.989863 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.990187 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.990015 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-config\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.990339 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.990204 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.991422 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.990408 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.991422 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.990735 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.991422 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.991363 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.992034 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.991680 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.992034 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.991789 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.992034 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.991807 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.992034 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.991994 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.992435 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.992078 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-web-config\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.992435 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.992378 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.992543 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.992519 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.993297 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.993276 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:39.999197 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:39.999172 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v9c7\" (UniqueName: \"kubernetes.io/projected/dc7e42c6-77b7-46f6-ba5c-ddb640c19830-kube-api-access-2v9c7\") pod \"prometheus-k8s-0\" (UID: \"dc7e42c6-77b7-46f6-ba5c-ddb640c19830\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:40.147478 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:40.147378 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:40.317553 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:40.317348 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 01:15:40.320268 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:15:40.320241 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7e42c6_77b7_46f6_ba5c_ddb640c19830.slice/crio-a679246e97fdf4253045f43a3a0692b706309cfa4d5baaa738d7a1defbea58ba WatchSource:0}: Error finding container a679246e97fdf4253045f43a3a0692b706309cfa4d5baaa738d7a1defbea58ba: Status 404 returned error can't find the container with id a679246e97fdf4253045f43a3a0692b706309cfa4d5baaa738d7a1defbea58ba Jul 17 01:15:40.756994 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:40.756886 2583 generic.go:356] "Generic (PLEG): container finished" podID="dc7e42c6-77b7-46f6-ba5c-ddb640c19830" containerID="11af51e09ab55f69f0f91b025716e3c697a6e76323f0f942d5af0b68aff05d50" exitCode=0 Jul 17 01:15:40.756994 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:40.756974 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dc7e42c6-77b7-46f6-ba5c-ddb640c19830","Type":"ContainerDied","Data":"11af51e09ab55f69f0f91b025716e3c697a6e76323f0f942d5af0b68aff05d50"} Jul 17 01:15:40.757196 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:40.757118 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dc7e42c6-77b7-46f6-ba5c-ddb640c19830","Type":"ContainerStarted","Data":"a679246e97fdf4253045f43a3a0692b706309cfa4d5baaa738d7a1defbea58ba"} Jul 17 01:15:41.168616 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:41.168584 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a4378a-66b2-4eed-b212-96e0a9708d06" path="/var/lib/kubelet/pods/20a4378a-66b2-4eed-b212-96e0a9708d06/volumes" Jul 17 01:15:41.765488 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:41.765445 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dc7e42c6-77b7-46f6-ba5c-ddb640c19830","Type":"ContainerStarted","Data":"d64e3c6098b05dd5d1f4e4d6e8faa96fa2900f1fbac302dbbc1a4d55cdbba1e5"} Jul 17 01:15:41.765870 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:41.765511 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dc7e42c6-77b7-46f6-ba5c-ddb640c19830","Type":"ContainerStarted","Data":"f417a1f0accb39f2f9359edd38c8f078b587ca4199580f6fc30547013401f9f3"} Jul 17 01:15:41.765870 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:41.765539 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dc7e42c6-77b7-46f6-ba5c-ddb640c19830","Type":"ContainerStarted","Data":"2fca1dc301d7653c1adbed8735ed60813198ff6efbd7a622d3ae1d657944947d"} Jul 17 01:15:41.765870 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:41.765563 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dc7e42c6-77b7-46f6-ba5c-ddb640c19830","Type":"ContainerStarted","Data":"8a5cabf10999b4798d74a6080b44d081b42e8499356a8a34f8cd86466d75f75d"} Jul 17 01:15:41.765870 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:41.765586 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dc7e42c6-77b7-46f6-ba5c-ddb640c19830","Type":"ContainerStarted","Data":"6776113e3a1cf98bf93e16a6dd0ecaa20e8c2e4453a1a3841114df17a300c345"} Jul 17 01:15:41.765870 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:41.765609 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dc7e42c6-77b7-46f6-ba5c-ddb640c19830","Type":"ContainerStarted","Data":"6db7136e7175d60e6b611f50c9cc79002445af789a1e7bc12d4ebb1a89518e4d"} Jul 17 01:15:41.796134 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:41.796042 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.77410383 podStartE2EDuration="2.796019868s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.916038ms" isStatefulPod=false podCreationTimestamp="2026-07-17 01:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:15:41.793500441 +0000 UTC m=+99.189115328" watchObservedRunningTime="2026-07-17 01:15:41.796019868 +0000 UTC m=+99.191634722" Jul 17 01:15:44.629748 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:44.629714 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hjgm7" Jul 17 01:15:45.147745 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:45.147704 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:15:45.889061 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:45.889022 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:45.939254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:45.939221 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:45.993375 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:45.993329 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:47.609443 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:47.609398 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:47.665276 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:47.665243 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:52.933369 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:52.933337 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:15:53.034897 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:15:53.034856 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:16:32.603597 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:16:32.603552 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:16:32.699534 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:16:32.699481 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:16:40.148506 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:16:40.148464 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:16:40.164510 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:16:40.164484 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:16:41.013598 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:16:41.013569 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 01:16:57.015968 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:16:57.015870 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:16:57.066024 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:16:57.065993 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:16:59.050223 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:16:59.050193 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:16:59.208748 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:16:59.208716 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:00.022223 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:00.022195 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:02.821670 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:02.821632 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:02.999302 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:02.999257 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:03.578538 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:03.578506 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:03.679432 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:03.679397 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:04.511074 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:04.511018 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:04.615814 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:04.615715 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:04.710299 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:04.710266 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:06.961158 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:06.961118 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:07.172488 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:07.172455 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:10.012761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:10.012729 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:10.061931 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:10.061906 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:12.463912 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.463882 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:12.542992 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.542945 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:12.546163 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.546143 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sfttw"] Jul 17 01:17:12.548474 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.548460 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sfttw" Jul 17 01:17:12.551406 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.551381 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 17 01:17:12.551508 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.551412 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-s7hrt\"" Jul 17 01:17:12.551836 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.551817 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 17 01:17:12.558174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.558152 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sfttw"] Jul 17 01:17:12.656584 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.656560 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bff61221-4739-4f23-abd4-0e4db64f2550-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sfttw\" (UID: \"bff61221-4739-4f23-abd4-0e4db64f2550\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sfttw" Jul 17 01:17:12.656691 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.656610 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks6mg\" (UniqueName: \"kubernetes.io/projected/bff61221-4739-4f23-abd4-0e4db64f2550-kube-api-access-ks6mg\") pod \"cert-manager-webhook-6486ffd997-sfttw\" (UID: \"bff61221-4739-4f23-abd4-0e4db64f2550\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sfttw" Jul 17 01:17:12.757428 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.757363 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bff61221-4739-4f23-abd4-0e4db64f2550-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sfttw\" (UID: \"bff61221-4739-4f23-abd4-0e4db64f2550\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sfttw" Jul 17 01:17:12.757428 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.757415 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks6mg\" (UniqueName: \"kubernetes.io/projected/bff61221-4739-4f23-abd4-0e4db64f2550-kube-api-access-ks6mg\") pod \"cert-manager-webhook-6486ffd997-sfttw\" (UID: \"bff61221-4739-4f23-abd4-0e4db64f2550\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sfttw" Jul 17 01:17:12.765041 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.765014 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bff61221-4739-4f23-abd4-0e4db64f2550-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sfttw\" (UID: \"bff61221-4739-4f23-abd4-0e4db64f2550\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sfttw" Jul 17 01:17:12.765312 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.765290 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks6mg\" (UniqueName: \"kubernetes.io/projected/bff61221-4739-4f23-abd4-0e4db64f2550-kube-api-access-ks6mg\") pod \"cert-manager-webhook-6486ffd997-sfttw\" (UID: \"bff61221-4739-4f23-abd4-0e4db64f2550\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sfttw" Jul 17 01:17:12.875309 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.875282 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sfttw" Jul 17 01:17:12.999641 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:12.999611 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sfttw"] Jul 17 01:17:13.003366 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:17:13.003337 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbff61221_4739_4f23_abd4_0e4db64f2550.slice/crio-c5ac8ac252764c99a22bd3a8d1b292c3599ec752279b9c440700993cd868ff89 WatchSource:0}: Error finding container c5ac8ac252764c99a22bd3a8d1b292c3599ec752279b9c440700993cd868ff89: Status 404 returned error can't find the container with id c5ac8ac252764c99a22bd3a8d1b292c3599ec752279b9c440700993cd868ff89 Jul 17 01:17:13.114416 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:13.114376 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sfttw" event={"ID":"bff61221-4739-4f23-abd4-0e4db64f2550","Type":"ContainerStarted","Data":"c5ac8ac252764c99a22bd3a8d1b292c3599ec752279b9c440700993cd868ff89"} Jul 17 01:17:13.822529 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:13.822452 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:13.919441 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:13.919400 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:14.796215 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:14.796177 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:15.841314 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:15.841280 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-trfjl"] Jul 17 01:17:15.844177 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:15.844152 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-trfjl" Jul 17 01:17:15.847204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:15.847180 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 01:17:15.847999 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:15.847959 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-x7z98\"" Jul 17 01:17:15.847999 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:15.847967 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 01:17:15.861061 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:15.861037 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-trfjl"] Jul 17 01:17:15.882016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:15.881902 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b71eb19d-c886-4c3e-a52b-da0f09282c3d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-trfjl\" (UID: \"b71eb19d-c886-4c3e-a52b-da0f09282c3d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-trfjl" Jul 17 01:17:15.882016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:15.881960 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4qt5\" (UniqueName: \"kubernetes.io/projected/b71eb19d-c886-4c3e-a52b-da0f09282c3d-kube-api-access-n4qt5\") pod \"openshift-lws-operator-fd8ccff4c-trfjl\" (UID: \"b71eb19d-c886-4c3e-a52b-da0f09282c3d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-trfjl" Jul 17 01:17:15.983429 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:15.983394 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b71eb19d-c886-4c3e-a52b-da0f09282c3d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-trfjl\" (UID: \"b71eb19d-c886-4c3e-a52b-da0f09282c3d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-trfjl" Jul 17 01:17:15.983621 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:15.983450 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4qt5\" (UniqueName: \"kubernetes.io/projected/b71eb19d-c886-4c3e-a52b-da0f09282c3d-kube-api-access-n4qt5\") pod \"openshift-lws-operator-fd8ccff4c-trfjl\" (UID: \"b71eb19d-c886-4c3e-a52b-da0f09282c3d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-trfjl" Jul 17 01:17:15.983866 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:15.983834 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b71eb19d-c886-4c3e-a52b-da0f09282c3d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-trfjl\" (UID: \"b71eb19d-c886-4c3e-a52b-da0f09282c3d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-trfjl" Jul 17 01:17:15.991651 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:15.991624 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4qt5\" (UniqueName: \"kubernetes.io/projected/b71eb19d-c886-4c3e-a52b-da0f09282c3d-kube-api-access-n4qt5\") pod \"openshift-lws-operator-fd8ccff4c-trfjl\" (UID: \"b71eb19d-c886-4c3e-a52b-da0f09282c3d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-trfjl" Jul 17 01:17:16.162648 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:16.162533 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-trfjl" Jul 17 01:17:16.346121 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:16.346070 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-trfjl"] Jul 17 01:17:16.349850 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:17:16.349630 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb71eb19d_c886_4c3e_a52b_da0f09282c3d.slice/crio-e95c5bd007858158abf845a8d0434c4c42ce90e9d84a72482a7f223997f86e0d WatchSource:0}: Error finding container e95c5bd007858158abf845a8d0434c4c42ce90e9d84a72482a7f223997f86e0d: Status 404 returned error can't find the container with id e95c5bd007858158abf845a8d0434c4c42ce90e9d84a72482a7f223997f86e0d Jul 17 01:17:17.134017 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:17.133974 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-trfjl" event={"ID":"b71eb19d-c886-4c3e-a52b-da0f09282c3d","Type":"ContainerStarted","Data":"e95c5bd007858158abf845a8d0434c4c42ce90e9d84a72482a7f223997f86e0d"} Jul 17 01:17:17.136005 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:17.135965 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sfttw" event={"ID":"bff61221-4739-4f23-abd4-0e4db64f2550","Type":"ContainerStarted","Data":"ebd2b795a6c9533acc96fef94aabd842ba581b75745af711b99ba039e650bfb9"} Jul 17 01:17:17.136283 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:17.136237 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sfttw" Jul 17 01:17:17.151474 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:17.151163 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-sfttw" podStartSLOduration=1.891107865 podStartE2EDuration="5.151145289s" totalImagesPullingTime="3.260037424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:17:17.150702897 +0000 UTC m=+194.546317748" watchObservedRunningTime="2026-07-17 01:17:17.151145289 +0000 UTC m=+194.546760137" Jul 17 01:17:19.147825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:19.147784 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-trfjl" event={"ID":"b71eb19d-c886-4c3e-a52b-da0f09282c3d","Type":"ContainerStarted","Data":"ffa0b8ce1d90c8fb48ec82674e18135d7e5e5af9fff4533e620117b82ccb65cb"} Jul 17 01:17:19.162724 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:19.162528 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-trfjl" podStartSLOduration=1.457261748 podStartE2EDuration="4.162505556s" totalImagesPullingTime="2.705243808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:17:19.161711647 +0000 UTC m=+196.557326498" watchObservedRunningTime="2026-07-17 01:17:19.162505556 +0000 UTC m=+196.558120405" Jul 17 01:17:19.910502 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:19.910464 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:20.016616 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:20.016589 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:21.943233 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:21.943202 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:23.143020 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:23.142992 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sfttw" Jul 17 01:17:35.026149 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:35.026108 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:35.127119 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:35.127063 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:37.634976 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:37.634944 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:37.728412 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:37.728378 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:38.253468 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.253436 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:38.458607 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.458569 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:38.561282 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.561252 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:38.605710 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.605677 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc"] Jul 17 01:17:38.614385 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.614361 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.618511 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.618487 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-276j7\"" Jul 17 01:17:38.619109 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.619062 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 01:17:38.619173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.619127 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 01:17:38.619230 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.619132 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 01:17:38.619230 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.619219 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 01:17:38.630776 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.630754 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc"] Jul 17 01:17:38.685528 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.685494 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e07729fb-8c24-4db7-9030-45f84fc5fbef-tmp\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.685528 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.685531 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e07729fb-8c24-4db7-9030-45f84fc5fbef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.685972 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.685553 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtf4n\" (UniqueName: \"kubernetes.io/projected/e07729fb-8c24-4db7-9030-45f84fc5fbef-kube-api-access-rtf4n\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.685972 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.685637 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e07729fb-8c24-4db7-9030-45f84fc5fbef-webhook-cert\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.685972 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.685668 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e07729fb-8c24-4db7-9030-45f84fc5fbef-manifests\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.768892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.768850 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:38.786508 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.786474 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e07729fb-8c24-4db7-9030-45f84fc5fbef-tmp\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.786676 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.786520 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e07729fb-8c24-4db7-9030-45f84fc5fbef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.786676 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.786643 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtf4n\" (UniqueName: \"kubernetes.io/projected/e07729fb-8c24-4db7-9030-45f84fc5fbef-kube-api-access-rtf4n\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.786781 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.786683 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e07729fb-8c24-4db7-9030-45f84fc5fbef-webhook-cert\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.786781 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.786701 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e07729fb-8c24-4db7-9030-45f84fc5fbef-manifests\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.787079 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.787051 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e07729fb-8c24-4db7-9030-45f84fc5fbef-manifests\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.788893 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.788855 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e07729fb-8c24-4db7-9030-45f84fc5fbef-tmp\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.789349 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.789328 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e07729fb-8c24-4db7-9030-45f84fc5fbef-webhook-cert\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.789426 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.789328 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e07729fb-8c24-4db7-9030-45f84fc5fbef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.795007 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.794985 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtf4n\" (UniqueName: \"kubernetes.io/projected/e07729fb-8c24-4db7-9030-45f84fc5fbef-kube-api-access-rtf4n\") pod \"opendatahub-operator-controller-manager-56ddbf9bf4-smgsc\" (UID: \"e07729fb-8c24-4db7-9030-45f84fc5fbef\") " pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.864699 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.864584 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:38.924195 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.924163 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:38.965067 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:38.965046 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:39.046633 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:39.046600 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:39.095587 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:39.095563 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc"] Jul 17 01:17:39.098365 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:17:39.098339 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07729fb_8c24_4db7_9030_45f84fc5fbef.slice/crio-09ef316e98dae3b0a8f0dfcb5653ab7d83346ec4ee130036b216db2ab2a560f6 WatchSource:0}: Error finding container 09ef316e98dae3b0a8f0dfcb5653ab7d83346ec4ee130036b216db2ab2a560f6: Status 404 returned error can't find the container with id 09ef316e98dae3b0a8f0dfcb5653ab7d83346ec4ee130036b216db2ab2a560f6 Jul 17 01:17:39.105703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:39.105684 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:39.169458 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:39.169391 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:39.222956 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:39.222925 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" event={"ID":"e07729fb-8c24-4db7-9030-45f84fc5fbef","Type":"ContainerStarted","Data":"09ef316e98dae3b0a8f0dfcb5653ab7d83346ec4ee130036b216db2ab2a560f6"} Jul 17 01:17:42.276920 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:42.276889 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:42.378148 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:42.378119 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:43.243717 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:43.243678 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" event={"ID":"e07729fb-8c24-4db7-9030-45f84fc5fbef","Type":"ContainerStarted","Data":"8e3b2cb1cb062548b8847ec132bc8190e1f4e47cd28b06b32910f13bbbba96b5"} Jul 17 01:17:44.249053 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:44.249017 2583 generic.go:356] "Generic (PLEG): container finished" podID="e07729fb-8c24-4db7-9030-45f84fc5fbef" containerID="8e3b2cb1cb062548b8847ec132bc8190e1f4e47cd28b06b32910f13bbbba96b5" exitCode=0 Jul 17 01:17:44.249491 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:44.249129 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" event={"ID":"e07729fb-8c24-4db7-9030-45f84fc5fbef","Type":"ContainerDied","Data":"8e3b2cb1cb062548b8847ec132bc8190e1f4e47cd28b06b32910f13bbbba96b5"} Jul 17 01:17:45.254610 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:45.254562 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" event={"ID":"e07729fb-8c24-4db7-9030-45f84fc5fbef","Type":"ContainerStarted","Data":"9d73fd43763effaba7b2ce5b512f5fa37d503228e46a7cf091e7a8e3b277fd91"} Jul 17 01:17:45.254989 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:45.254666 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:45.275502 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:45.275448 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" podStartSLOduration=2.5851380170000002 podStartE2EDuration="7.275430369s" totalImagesPullingTime="3.658395201s" totalInitContainerRuntime="1.031897151s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:17:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:17:45.273603472 +0000 UTC m=+222.669218362" watchObservedRunningTime="2026-07-17 01:17:45.275430369 +0000 UTC m=+222.671045292" Jul 17 01:17:50.223226 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:50.223197 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:50.316975 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:50.316938 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:56.261755 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:56.261724 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56ddbf9bf4-smgsc" Jul 17 01:17:57.946267 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:57.946233 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:58.041912 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.041877 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:58.404160 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.404124 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:58.533210 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.533182 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw"] Jul 17 01:17:58.536359 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.536340 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw" Jul 17 01:17:58.539197 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.539171 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 17 01:17:58.540079 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.540059 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 01:17:58.540181 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.540088 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 01:17:58.540237 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.540196 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r6nnr\"" Jul 17 01:17:58.540237 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.540231 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 01:17:58.548197 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.548174 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw"] Jul 17 01:17:58.559946 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.559922 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2535c4a4-5a00-4efe-bed9-dd567cbe576f-tmp\") pod \"kube-auth-proxy-68f596cfdd-n7zfw\" (UID: \"2535c4a4-5a00-4efe-bed9-dd567cbe576f\") " pod="openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw" Jul 17 01:17:58.560056 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.559977 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdchz\" (UniqueName: \"kubernetes.io/projected/2535c4a4-5a00-4efe-bed9-dd567cbe576f-kube-api-access-sdchz\") pod \"kube-auth-proxy-68f596cfdd-n7zfw\" (UID: \"2535c4a4-5a00-4efe-bed9-dd567cbe576f\") " pod="openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw" Jul 17 01:17:58.560130 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.560091 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2535c4a4-5a00-4efe-bed9-dd567cbe576f-tls-certs\") pod \"kube-auth-proxy-68f596cfdd-n7zfw\" (UID: \"2535c4a4-5a00-4efe-bed9-dd567cbe576f\") " pod="openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw" Jul 17 01:17:58.629907 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.629877 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:58.660883 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.660808 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2535c4a4-5a00-4efe-bed9-dd567cbe576f-tls-certs\") pod \"kube-auth-proxy-68f596cfdd-n7zfw\" (UID: \"2535c4a4-5a00-4efe-bed9-dd567cbe576f\") " pod="openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw" Jul 17 01:17:58.660883 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.660850 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2535c4a4-5a00-4efe-bed9-dd567cbe576f-tmp\") pod \"kube-auth-proxy-68f596cfdd-n7zfw\" (UID: \"2535c4a4-5a00-4efe-bed9-dd567cbe576f\") " pod="openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw" Jul 17 01:17:58.661044 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.660901 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdchz\" (UniqueName: \"kubernetes.io/projected/2535c4a4-5a00-4efe-bed9-dd567cbe576f-kube-api-access-sdchz\") pod \"kube-auth-proxy-68f596cfdd-n7zfw\" (UID: \"2535c4a4-5a00-4efe-bed9-dd567cbe576f\") " pod="openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw" Jul 17 01:17:58.663464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.663442 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2535c4a4-5a00-4efe-bed9-dd567cbe576f-tmp\") pod \"kube-auth-proxy-68f596cfdd-n7zfw\" (UID: \"2535c4a4-5a00-4efe-bed9-dd567cbe576f\") " pod="openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw" Jul 17 01:17:58.663576 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.663514 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2535c4a4-5a00-4efe-bed9-dd567cbe576f-tls-certs\") pod \"kube-auth-proxy-68f596cfdd-n7zfw\" (UID: \"2535c4a4-5a00-4efe-bed9-dd567cbe576f\") " pod="openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw" Jul 17 01:17:58.670861 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.670835 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdchz\" (UniqueName: \"kubernetes.io/projected/2535c4a4-5a00-4efe-bed9-dd567cbe576f-kube-api-access-sdchz\") pod \"kube-auth-proxy-68f596cfdd-n7zfw\" (UID: \"2535c4a4-5a00-4efe-bed9-dd567cbe576f\") " pod="openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw" Jul 17 01:17:58.845899 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.845868 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw" Jul 17 01:17:58.877708 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.877669 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:58.978611 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.978572 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:58.987705 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:58.987679 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw"] Jul 17 01:17:58.990283 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:17:58.990254 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2535c4a4_5a00_4efe_bed9_dd567cbe576f.slice/crio-952138e03233a18d879e59dea71acb55b7409989961aaa4df6da88495c3396d7 WatchSource:0}: Error finding container 952138e03233a18d879e59dea71acb55b7409989961aaa4df6da88495c3396d7: Status 404 returned error can't find the container with id 952138e03233a18d879e59dea71acb55b7409989961aaa4df6da88495c3396d7 Jul 17 01:17:59.052282 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:59.052251 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:59.311588 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:59.311553 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw" event={"ID":"2535c4a4-5a00-4efe-bed9-dd567cbe576f","Type":"ContainerStarted","Data":"952138e03233a18d879e59dea71acb55b7409989961aaa4df6da88495c3396d7"} Jul 17 01:17:59.385597 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:59.385556 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:59.665895 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:59.665805 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:59.748146 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:59.748079 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:59.857609 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:59.857576 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:17:59.959830 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:17:59.959746 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:00.758469 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:00.758429 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:02.326878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:02.326841 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw" event={"ID":"2535c4a4-5a00-4efe-bed9-dd567cbe576f","Type":"ContainerStarted","Data":"5904b0b9037d1c9afe17575571a6637a795ad44bd1d7abd59625ec763baa103b"} Jul 17 01:18:02.343433 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:02.343382 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68f596cfdd-n7zfw" podStartSLOduration=1.4427234979999999 podStartE2EDuration="4.343367702s" totalImagesPullingTime="2.900644204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:18:02.342147174 +0000 UTC m=+239.737762029" watchObservedRunningTime="2026-07-17 01:18:02.343367702 +0000 UTC m=+239.738982553" Jul 17 01:18:05.517486 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.517446 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:05.616414 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.616381 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:05.868585 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.868552 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j"] Jul 17 01:18:05.873755 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.873736 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:05.877458 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.877423 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 17 01:18:05.877706 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.877685 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 17 01:18:05.877777 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.877713 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 17 01:18:05.877777 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.877699 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vgcn8\"" Jul 17 01:18:05.877777 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.877746 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 17 01:18:05.898236 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.898201 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j"] Jul 17 01:18:05.919273 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.919242 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b6e3bd69-af0f-4c90-a537-b2cc0244e085-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:05.919418 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.919278 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b6e3bd69-af0f-4c90-a537-b2cc0244e085-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:05.919418 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.919305 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b6e3bd69-af0f-4c90-a537-b2cc0244e085-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:05.919418 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.919395 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b6e3bd69-af0f-4c90-a537-b2cc0244e085-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:05.919520 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.919431 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gft78\" (UniqueName: \"kubernetes.io/projected/b6e3bd69-af0f-4c90-a537-b2cc0244e085-kube-api-access-gft78\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:05.919520 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.919483 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e3bd69-af0f-4c90-a537-b2cc0244e085-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:05.919581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:05.919528 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e3bd69-af0f-4c90-a537-b2cc0244e085-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.020162 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.020130 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e3bd69-af0f-4c90-a537-b2cc0244e085-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.020339 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.020195 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e3bd69-af0f-4c90-a537-b2cc0244e085-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.020339 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.020225 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b6e3bd69-af0f-4c90-a537-b2cc0244e085-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.020339 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.020256 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b6e3bd69-af0f-4c90-a537-b2cc0244e085-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.020339 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.020293 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b6e3bd69-af0f-4c90-a537-b2cc0244e085-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.020542 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.020344 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b6e3bd69-af0f-4c90-a537-b2cc0244e085-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.020542 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.020370 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gft78\" (UniqueName: \"kubernetes.io/projected/b6e3bd69-af0f-4c90-a537-b2cc0244e085-kube-api-access-gft78\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.020989 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.020934 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b6e3bd69-af0f-4c90-a537-b2cc0244e085-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.022603 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.022564 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6e3bd69-af0f-4c90-a537-b2cc0244e085-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.022720 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.022621 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b6e3bd69-af0f-4c90-a537-b2cc0244e085-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.022788 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.022722 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b6e3bd69-af0f-4c90-a537-b2cc0244e085-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.022921 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.022903 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b6e3bd69-af0f-4c90-a537-b2cc0244e085-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.030331 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.030307 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6e3bd69-af0f-4c90-a537-b2cc0244e085-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.030715 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.030694 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gft78\" (UniqueName: \"kubernetes.io/projected/b6e3bd69-af0f-4c90-a537-b2cc0244e085-kube-api-access-gft78\") pod \"istiod-openshift-gateway-94bb8fbfd-87p9j\" (UID: \"b6e3bd69-af0f-4c90-a537-b2cc0244e085\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.183145 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.183027 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:06.292739 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.292703 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:06.328686 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.328651 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j"] Jul 17 01:18:06.332405 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:18:06.332380 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e3bd69_af0f_4c90_a537_b2cc0244e085.slice/crio-07e002a5f03a8626b556cb617e627279e4f6a833746d9ed78cfbcde70858af3e WatchSource:0}: Error finding container 07e002a5f03a8626b556cb617e627279e4f6a833746d9ed78cfbcde70858af3e: Status 404 returned error can't find the container with id 07e002a5f03a8626b556cb617e627279e4f6a833746d9ed78cfbcde70858af3e Jul 17 01:18:06.344127 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.344077 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" event={"ID":"b6e3bd69-af0f-4c90-a537-b2cc0244e085","Type":"ContainerStarted","Data":"07e002a5f03a8626b556cb617e627279e4f6a833746d9ed78cfbcde70858af3e"} Jul 17 01:18:06.617779 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.617741 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:06.712476 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:06.712447 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:08.511444 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:08.511405 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:08.594474 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:08.594421 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:08.689630 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:08.689603 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:08.734604 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:08.734560 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 01:18:08.734721 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:08.734634 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 01:18:08.827653 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:08.827618 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:08.964052 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:08.964016 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:09.055118 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.055063 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:09.237905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.237873 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7"] Jul 17 01:18:09.241049 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.241030 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:09.244545 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.244524 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 01:18:09.244658 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.244525 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 01:18:09.244658 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.244525 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 01:18:09.244658 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.244526 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hz2wm\"" Jul 17 01:18:09.250217 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.250194 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7"] Jul 17 01:18:09.352482 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.352402 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/78792b02-f8d0-47f8-a104-644d97cf3d6a-manager-config\") pod \"lws-controller-manager-66ffdbf46d-zk9p7\" (UID: \"78792b02-f8d0-47f8-a104-644d97cf3d6a\") " pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:09.352482 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.352451 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/78792b02-f8d0-47f8-a104-644d97cf3d6a-metrics-cert\") pod \"lws-controller-manager-66ffdbf46d-zk9p7\" (UID: \"78792b02-f8d0-47f8-a104-644d97cf3d6a\") " pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:09.352666 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.352541 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s89w\" (UniqueName: \"kubernetes.io/projected/78792b02-f8d0-47f8-a104-644d97cf3d6a-kube-api-access-7s89w\") pod \"lws-controller-manager-66ffdbf46d-zk9p7\" (UID: \"78792b02-f8d0-47f8-a104-644d97cf3d6a\") " pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:09.352666 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.352577 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78792b02-f8d0-47f8-a104-644d97cf3d6a-cert\") pod \"lws-controller-manager-66ffdbf46d-zk9p7\" (UID: \"78792b02-f8d0-47f8-a104-644d97cf3d6a\") " pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:09.361449 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.361421 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" event={"ID":"b6e3bd69-af0f-4c90-a537-b2cc0244e085","Type":"ContainerStarted","Data":"fe7a9a6f099b7a2f299826669ed1a1783e2d36d5ba376ce5c00181bb81d0ead0"} Jul 17 01:18:09.361624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.361597 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:09.363239 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.363212 2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-87p9j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 01:18:09.363344 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.363271 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" podUID="b6e3bd69-af0f-4c90-a537-b2cc0244e085" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 01:18:09.411330 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.411248 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" podStartSLOduration=2.011556826 podStartE2EDuration="4.411228115s" totalImagesPullingTime="2.399671289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:18:09.410413966 +0000 UTC m=+246.806028818" watchObservedRunningTime="2026-07-17 01:18:09.411228115 +0000 UTC m=+246.806842967" Jul 17 01:18:09.453210 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.453173 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/78792b02-f8d0-47f8-a104-644d97cf3d6a-metrics-cert\") pod \"lws-controller-manager-66ffdbf46d-zk9p7\" (UID: \"78792b02-f8d0-47f8-a104-644d97cf3d6a\") " pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:09.453385 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.453364 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s89w\" (UniqueName: \"kubernetes.io/projected/78792b02-f8d0-47f8-a104-644d97cf3d6a-kube-api-access-7s89w\") pod \"lws-controller-manager-66ffdbf46d-zk9p7\" (UID: \"78792b02-f8d0-47f8-a104-644d97cf3d6a\") " pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:09.453438 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.453421 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78792b02-f8d0-47f8-a104-644d97cf3d6a-cert\") pod \"lws-controller-manager-66ffdbf46d-zk9p7\" (UID: \"78792b02-f8d0-47f8-a104-644d97cf3d6a\") " pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:09.453693 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.453671 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/78792b02-f8d0-47f8-a104-644d97cf3d6a-manager-config\") pod \"lws-controller-manager-66ffdbf46d-zk9p7\" (UID: \"78792b02-f8d0-47f8-a104-644d97cf3d6a\") " pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:09.454276 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.454255 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/78792b02-f8d0-47f8-a104-644d97cf3d6a-manager-config\") pod \"lws-controller-manager-66ffdbf46d-zk9p7\" (UID: \"78792b02-f8d0-47f8-a104-644d97cf3d6a\") " pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:09.456017 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.455995 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/78792b02-f8d0-47f8-a104-644d97cf3d6a-metrics-cert\") pod \"lws-controller-manager-66ffdbf46d-zk9p7\" (UID: \"78792b02-f8d0-47f8-a104-644d97cf3d6a\") " pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:09.456126 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.456017 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78792b02-f8d0-47f8-a104-644d97cf3d6a-cert\") pod \"lws-controller-manager-66ffdbf46d-zk9p7\" (UID: \"78792b02-f8d0-47f8-a104-644d97cf3d6a\") " pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:09.468410 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.468383 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s89w\" (UniqueName: \"kubernetes.io/projected/78792b02-f8d0-47f8-a104-644d97cf3d6a-kube-api-access-7s89w\") pod \"lws-controller-manager-66ffdbf46d-zk9p7\" (UID: \"78792b02-f8d0-47f8-a104-644d97cf3d6a\") " pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:09.551461 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.551420 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:09.698664 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:09.698637 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7"] Jul 17 01:18:09.701157 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:18:09.701129 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78792b02_f8d0_47f8_a104_644d97cf3d6a.slice/crio-b095acc0ac156480808d223e3bb024bcec7de23efd74cbafa4584bf50063ff31 WatchSource:0}: Error finding container b095acc0ac156480808d223e3bb024bcec7de23efd74cbafa4584bf50063ff31: Status 404 returned error can't find the container with id b095acc0ac156480808d223e3bb024bcec7de23efd74cbafa4584bf50063ff31 Jul 17 01:18:10.366678 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:10.366637 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" event={"ID":"78792b02-f8d0-47f8-a104-644d97cf3d6a","Type":"ContainerStarted","Data":"b095acc0ac156480808d223e3bb024bcec7de23efd74cbafa4584bf50063ff31"} Jul 17 01:18:10.367612 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:10.367580 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-87p9j" Jul 17 01:18:12.377216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:12.377185 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" event={"ID":"78792b02-f8d0-47f8-a104-644d97cf3d6a","Type":"ContainerStarted","Data":"e223c5daade6ac6614713363cf0d84a65739d332f0a0e55dff976ac854a7add0"} Jul 17 01:18:12.377664 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:12.377268 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:12.395852 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:12.395706 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" podStartSLOduration=1.697056652 podStartE2EDuration="3.395683981s" totalImagesPullingTime="1.698627329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:18:12.395313081 +0000 UTC m=+249.790927934" watchObservedRunningTime="2026-07-17 01:18:12.395683981 +0000 UTC m=+249.791298832" Jul 17 01:18:16.193433 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:16.193395 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:17.335631 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:17.335589 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:17.935185 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:17.935152 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:18.020427 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:18.020399 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:23.384465 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:23.384379 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-66ffdbf46d-zk9p7" Jul 17 01:18:59.421390 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:59.421344 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:59.517581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:59.517552 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:18:59.607750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:18:59.607715 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:00.355720 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:00.355693 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:00.550150 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:00.550117 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:03.024793 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:03.024757 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:19:03.028694 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:03.028670 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:19:03.035158 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:03.035136 2583 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 01:19:28.326193 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:28.326150 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:32.524527 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:32.524482 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:35.036052 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:35.036020 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:35.133767 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:35.133732 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:36.439672 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:36.439638 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:39.518061 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:39.518026 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:39.902960 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:39.902907 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:40.266167 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:40.266059 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:40.356889 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:40.356854 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:42.964012 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:42.963980 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:44.340030 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:44.339981 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:44.439256 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:44.439223 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:48.134304 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:48.134274 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:48.231955 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:48.231916 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:49.258743 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:49.258707 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:49.389015 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:49.388983 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:51.517003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:51.516968 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:51.617642 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:51.617602 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:19:59.960621 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:19:59.960523 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:00.058996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:00.058955 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:02.446736 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:02.446689 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:02.547218 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:02.547183 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:05.261020 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:05.260986 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:05.361732 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:05.361704 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:08.798713 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:08.798671 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:08.902176 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:08.902143 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:11.144925 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:11.144888 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:14.745924 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:14.745891 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:14.809253 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:14.809218 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:17.643021 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:17.642978 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:17.743078 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:17.743045 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:20.291983 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:20.291950 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:20.420199 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:20.420157 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:22.355984 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:22.355946 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:22.455567 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:22.455536 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:22.557707 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:22.557657 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:23.637994 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:23.637959 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:23.738574 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:23.738539 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:27.255739 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:27.255707 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:27.354774 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:27.354740 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:37.038818 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:37.038780 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:37.093134 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:37.093081 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:37.441990 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:37.441900 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:37.561078 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:37.561038 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:41.496214 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:41.496182 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:41.707037 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:41.706998 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:55.252448 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:55.252413 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:55.357220 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:55.357179 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:55.409423 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:55.409390 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:56.086608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:56.086563 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:20:56.147262 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:20:56.147233 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:21:11.600795 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:21:11.600761 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:21:11.696149 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:21:11.696093 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:21:12.082559 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:21:12.082524 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:21:21.126322 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:21:21.126286 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:21:21.226879 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:21:21.226845 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:21:24.872534 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:21:24.872437 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:21:24.972636 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:21:24.972568 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:21:31.231880 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:21:31.231847 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:21:31.332151 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:21:31.332121 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:21:31.391838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:21:31.391804 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:21:34.775392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:21:34.775356 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:21:34.874045 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:21:34.874007 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:21:40.628313 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:21:40.628279 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:21:40.679946 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:21:40.679907 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:22:04.798742 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:22:04.798695 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:22:04.850043 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:22:04.850001 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:22:06.620188 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:22:06.620154 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:22:06.709628 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:22:06.709591 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:22:07.287443 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:22:07.287406 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:22:07.406610 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:22:07.406563 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:22:07.755754 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:22:07.755673 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:22:07.843608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:22:07.843573 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:22:15.482328 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:22:15.482295 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:22:16.224719 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:22:16.224686 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:22:17.971334 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:22:17.971297 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:22:18.070402 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:22:18.070354 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:22:28.809139 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:22:28.809083 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:22:45.191117 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:22:45.191061 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:23:02.283770 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:23:02.283649 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:23:08.333686 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:23:08.333649 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:23:08.388714 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:23:08.388676 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:23:24.653977 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:23:24.653942 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:23:33.190717 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:23:33.190688 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:23:49.303997 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:23:49.303958 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:24:03.056592 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:24:03.056559 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:24:03.057889 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:24:03.057866 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:24:36.983717 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:24:36.983681 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:24:37.082703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:24:37.082669 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:24:38.095780 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:24:38.095752 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:24:38.195115 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:24:38.195064 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:26:42.952970 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:26:42.952935 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:26:44.161437 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:26:44.161408 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:27:31.868297 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:27:31.868193 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:27:31.990499 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:27:31.990466 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:27:32.128728 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:27:32.128643 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:27:37.801841 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:27:37.801805 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:27:37.903036 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:27:37.902982 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:27:38.001886 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:27:38.001846 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:27:38.236547 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:27:38.236462 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:27:44.007599 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:27:44.007561 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:27:44.065845 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:27:44.065804 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:28:03.573744 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:28:03.573709 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:28:28.430320 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:28:28.430283 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:29:03.083915 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:29:03.083844 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:29:03.083915 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:29:03.083911 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:30:00.142273 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:30:00.142235 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737530-dxdf8"] Jul 17 01:30:00.145504 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:30:00.145485 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737530-dxdf8" Jul 17 01:30:00.148087 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:30:00.148063 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:30:00.155257 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:30:00.155233 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737530-dxdf8" Jul 17 01:30:00.162604 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:30:00.162577 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737530-dxdf8"] Jul 17 01:30:00.293513 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:30:00.293487 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737530-dxdf8"] Jul 17 01:30:00.296216 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:30:00.296186 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2563445a_1d64_4165_a849_a50edd24de77.slice/crio-6b08ecbfc9a0ead36815ee46068387fefbc05386248d181a79be871e5695baca WatchSource:0}: Error finding container 6b08ecbfc9a0ead36815ee46068387fefbc05386248d181a79be871e5695baca: Status 404 returned error can't find the container with id 6b08ecbfc9a0ead36815ee46068387fefbc05386248d181a79be871e5695baca Jul 17 01:30:00.298156 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:30:00.298140 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 01:30:00.471707 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:30:00.471619 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737530-dxdf8" event={"ID":"2563445a-1d64-4165-a849-a50edd24de77","Type":"ContainerStarted","Data":"6b08ecbfc9a0ead36815ee46068387fefbc05386248d181a79be871e5695baca"} Jul 17 01:30:03.487227 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:30:03.487191 2583 generic.go:356] "Generic (PLEG): container finished" podID="2563445a-1d64-4165-a849-a50edd24de77" containerID="6437c285fd0d66a4535711fd505f346038e4f201fff60771c28474994fd30bd9" exitCode=0 Jul 17 01:30:03.487624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:30:03.487279 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737530-dxdf8" event={"ID":"2563445a-1d64-4165-a849-a50edd24de77","Type":"ContainerDied","Data":"6437c285fd0d66a4535711fd505f346038e4f201fff60771c28474994fd30bd9"} Jul 17 01:30:04.621260 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:30:04.621236 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737530-dxdf8" Jul 17 01:30:05.497759 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:30:05.497666 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737530-dxdf8" event={"ID":"2563445a-1d64-4165-a849-a50edd24de77","Type":"ContainerDied","Data":"6b08ecbfc9a0ead36815ee46068387fefbc05386248d181a79be871e5695baca"} Jul 17 01:30:05.497759 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:30:05.497699 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737530-dxdf8" Jul 17 01:30:05.497759 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:30:05.497730 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b08ecbfc9a0ead36815ee46068387fefbc05386248d181a79be871e5695baca" Jul 17 01:32:44.569964 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:32:44.569934 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:32:44.623903 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:32:44.623876 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:34:03.108614 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:34:03.108587 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:34:03.109802 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:34:03.109778 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:35:41.862067 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:35:41.862024 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:36:51.122389 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:36:51.122299 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:36:54.156655 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:36:54.156622 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:37:58.148657 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:37:58.148577 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:37:58.247889 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:37:58.247857 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:38:40.562593 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:38:40.562556 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:38:50.185701 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:38:50.185667 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:38:58.427409 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:38:58.427373 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:03.136818 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:03.136791 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:39:03.139461 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:03.139439 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:39:03.505903 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:03.505818 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:06.539772 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:06.539731 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:06.638676 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:06.638640 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:07.007923 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:07.007840 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:12.795722 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:12.795682 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:12.896576 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:12.896545 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:13.007191 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:13.007146 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:13.067220 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:13.067133 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:14.113620 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.113582 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:14.183093 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.183057 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:14.189863 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.189835 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw"] Jul 17 01:39:14.190266 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.190250 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2563445a-1d64-4165-a849-a50edd24de77" containerName="cleanup" Jul 17 01:39:14.190266 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.190268 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2563445a-1d64-4165-a849-a50edd24de77" containerName="cleanup" Jul 17 01:39:14.190415 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.190353 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2563445a-1d64-4165-a849-a50edd24de77" containerName="cleanup" Jul 17 01:39:14.193333 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.193313 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:14.195712 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.195687 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d7122b7f-serving-cert\"" Jul 17 01:39:14.195987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.195963 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:39:14.196786 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.196766 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 01:39:14.196889 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.196792 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5l8p8\"" Jul 17 01:39:14.203331 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.203307 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw"] Jul 17 01:39:14.255915 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.255877 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:14.312782 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.312746 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-openshift-service-ca\") pod \"maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw\" (UID: \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:14.312957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.312791 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2mvw\" (UniqueName: \"kubernetes.io/projected/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-kube-api-access-p2mvw\") pod \"maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw\" (UID: \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:14.312957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.312819 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-maas-api-tls\") pod \"maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw\" (UID: \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:14.413552 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.413454 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-openshift-service-ca\") pod \"maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw\" (UID: \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:14.413552 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.413501 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2mvw\" (UniqueName: \"kubernetes.io/projected/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-kube-api-access-p2mvw\") pod \"maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw\" (UID: \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:14.413552 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.413522 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-maas-api-tls\") pod \"maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw\" (UID: \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:14.414181 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.414159 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-openshift-service-ca\") pod \"maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw\" (UID: \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:14.416075 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.416055 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-maas-api-tls\") pod \"maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw\" (UID: \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:14.422997 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.422962 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2mvw\" (UniqueName: \"kubernetes.io/projected/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-kube-api-access-p2mvw\") pod \"maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw\" (UID: \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:14.461283 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.461249 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:14.506287 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.506247 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:14.660620 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.660525 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw"] Jul 17 01:39:14.663523 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:39:14.663494 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f45c4ee_7550_482b_9acc_bb68a8f95b3b.slice/crio-ea3280ac5b795ee00d6dbff0eb0cdf92b900628f8d1a0355ddca0681da6cf593 WatchSource:0}: Error finding container ea3280ac5b795ee00d6dbff0eb0cdf92b900628f8d1a0355ddca0681da6cf593: Status 404 returned error can't find the container with id ea3280ac5b795ee00d6dbff0eb0cdf92b900628f8d1a0355ddca0681da6cf593 Jul 17 01:39:14.665221 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.665206 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 01:39:14.760196 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.760054 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:14.863048 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.863016 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:14.895810 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:14.895775 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" event={"ID":"6f45c4ee-7550-482b-9acc-bb68a8f95b3b","Type":"ContainerStarted","Data":"ea3280ac5b795ee00d6dbff0eb0cdf92b900628f8d1a0355ddca0681da6cf593"} Jul 17 01:39:16.811121 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:16.811056 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:39:17.911617 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:17.911524 2583 generic.go:356] "Generic (PLEG): container finished" podID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerID="8d078e2801530e801d07c8b4baa20c85c18580884aa497e6ea4efd7245128f4d" exitCode=1 Jul 17 01:39:17.911975 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:17.911611 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" event={"ID":"6f45c4ee-7550-482b-9acc-bb68a8f95b3b","Type":"ContainerDied","Data":"8d078e2801530e801d07c8b4baa20c85c18580884aa497e6ea4efd7245128f4d"} Jul 17 01:39:17.911975 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:17.911820 2583 scope.go:117] "RemoveContainer" containerID="8d078e2801530e801d07c8b4baa20c85c18580884aa497e6ea4efd7245128f4d" Jul 17 01:39:18.917477 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:18.917390 2583 generic.go:356] "Generic (PLEG): container finished" podID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerID="7e1ad31dbb4b0090a58e0c0081b5480a51cf436cfe6e094564de83a7237ddc8c" exitCode=1 Jul 17 01:39:18.917477 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:18.917463 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" event={"ID":"6f45c4ee-7550-482b-9acc-bb68a8f95b3b","Type":"ContainerDied","Data":"7e1ad31dbb4b0090a58e0c0081b5480a51cf436cfe6e094564de83a7237ddc8c"} Jul 17 01:39:18.917923 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:18.917517 2583 scope.go:117] "RemoveContainer" containerID="8d078e2801530e801d07c8b4baa20c85c18580884aa497e6ea4efd7245128f4d" Jul 17 01:39:18.917923 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:18.917752 2583 scope.go:117] "RemoveContainer" containerID="7e1ad31dbb4b0090a58e0c0081b5480a51cf436cfe6e094564de83a7237ddc8c" Jul 17 01:39:18.918004 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:39:18.917965 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:39:19.506585 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:19.506556 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:19.923488 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:19.923460 2583 scope.go:117] "RemoveContainer" containerID="7e1ad31dbb4b0090a58e0c0081b5480a51cf436cfe6e094564de83a7237ddc8c" Jul 17 01:39:19.923855 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:39:19.923633 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:39:20.927620 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:20.927594 2583 scope.go:117] "RemoveContainer" containerID="7e1ad31dbb4b0090a58e0c0081b5480a51cf436cfe6e094564de83a7237ddc8c" Jul 17 01:39:20.928002 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:39:20.927772 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:39:24.507066 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:24.507025 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:24.507481 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:24.507444 2583 scope.go:117] "RemoveContainer" containerID="7e1ad31dbb4b0090a58e0c0081b5480a51cf436cfe6e094564de83a7237ddc8c" Jul 17 01:39:24.507638 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:39:24.507616 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:39:38.162542 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:38.162512 2583 scope.go:117] "RemoveContainer" containerID="7e1ad31dbb4b0090a58e0c0081b5480a51cf436cfe6e094564de83a7237ddc8c" Jul 17 01:39:39.010334 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:39.010301 2583 generic.go:356] "Generic (PLEG): container finished" podID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerID="d31a75509210c1afe656ff8025a65e20f427aaf11c312f39042f2ce3cc549b91" exitCode=1 Jul 17 01:39:39.010334 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:39.010342 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" event={"ID":"6f45c4ee-7550-482b-9acc-bb68a8f95b3b","Type":"ContainerDied","Data":"d31a75509210c1afe656ff8025a65e20f427aaf11c312f39042f2ce3cc549b91"} Jul 17 01:39:39.010584 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:39.010396 2583 scope.go:117] "RemoveContainer" containerID="7e1ad31dbb4b0090a58e0c0081b5480a51cf436cfe6e094564de83a7237ddc8c" Jul 17 01:39:39.010771 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:39.010752 2583 scope.go:117] "RemoveContainer" containerID="d31a75509210c1afe656ff8025a65e20f427aaf11c312f39042f2ce3cc549b91" Jul 17 01:39:39.011032 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:39:39.011004 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:39:39.506520 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:39.506477 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:40.017014 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:40.016984 2583 scope.go:117] "RemoveContainer" containerID="d31a75509210c1afe656ff8025a65e20f427aaf11c312f39042f2ce3cc549b91" Jul 17 01:39:40.017213 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:39:40.017184 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:39:44.506898 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:44.506854 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:39:44.507418 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:44.507299 2583 scope.go:117] "RemoveContainer" containerID="d31a75509210c1afe656ff8025a65e20f427aaf11c312f39042f2ce3cc549b91" Jul 17 01:39:44.507503 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:39:44.507480 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:39:59.163191 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:39:59.163161 2583 scope.go:117] "RemoveContainer" containerID="d31a75509210c1afe656ff8025a65e20f427aaf11c312f39042f2ce3cc549b91" Jul 17 01:40:00.108237 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:00.108205 2583 generic.go:356] "Generic (PLEG): container finished" podID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerID="5bc28e77a96aab114ab9183dde8388580805eb5bc158e8e5590c00f3a56baf17" exitCode=1 Jul 17 01:40:00.108415 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:00.108290 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" event={"ID":"6f45c4ee-7550-482b-9acc-bb68a8f95b3b","Type":"ContainerDied","Data":"5bc28e77a96aab114ab9183dde8388580805eb5bc158e8e5590c00f3a56baf17"} Jul 17 01:40:00.108415 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:00.108358 2583 scope.go:117] "RemoveContainer" containerID="d31a75509210c1afe656ff8025a65e20f427aaf11c312f39042f2ce3cc549b91" Jul 17 01:40:00.108724 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:00.108710 2583 scope.go:117] "RemoveContainer" containerID="5bc28e77a96aab114ab9183dde8388580805eb5bc158e8e5590c00f3a56baf17" Jul 17 01:40:00.108931 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:40:00.108906 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:40:04.506641 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:04.506607 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:40:04.507050 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:04.506666 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:40:04.507050 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:04.507026 2583 scope.go:117] "RemoveContainer" containerID="5bc28e77a96aab114ab9183dde8388580805eb5bc158e8e5590c00f3a56baf17" Jul 17 01:40:04.507262 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:40:04.507241 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:40:15.163251 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:15.163222 2583 scope.go:117] "RemoveContainer" containerID="5bc28e77a96aab114ab9183dde8388580805eb5bc158e8e5590c00f3a56baf17" Jul 17 01:40:15.163694 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:40:15.163421 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:40:26.162225 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:26.162192 2583 scope.go:117] "RemoveContainer" containerID="5bc28e77a96aab114ab9183dde8388580805eb5bc158e8e5590c00f3a56baf17" Jul 17 01:40:26.162651 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:40:26.162371 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:40:41.162722 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:41.162687 2583 scope.go:117] "RemoveContainer" containerID="5bc28e77a96aab114ab9183dde8388580805eb5bc158e8e5590c00f3a56baf17" Jul 17 01:40:42.303228 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:42.303190 2583 generic.go:356] "Generic (PLEG): container finished" podID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerID="a66ed0d45de13097a30df3f4ed95053c2a34b33e7df303a6123a3b585ac69039" exitCode=1 Jul 17 01:40:42.305503 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:42.303263 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" event={"ID":"6f45c4ee-7550-482b-9acc-bb68a8f95b3b","Type":"ContainerDied","Data":"a66ed0d45de13097a30df3f4ed95053c2a34b33e7df303a6123a3b585ac69039"} Jul 17 01:40:42.305503 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:42.303325 2583 scope.go:117] "RemoveContainer" containerID="5bc28e77a96aab114ab9183dde8388580805eb5bc158e8e5590c00f3a56baf17" Jul 17 01:40:42.305503 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:42.303725 2583 scope.go:117] "RemoveContainer" containerID="a66ed0d45de13097a30df3f4ed95053c2a34b33e7df303a6123a3b585ac69039" Jul 17 01:40:42.305503 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:40:42.303932 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:40:44.506796 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:44.506760 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:40:44.507221 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:44.506818 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:40:44.507221 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:44.507202 2583 scope.go:117] "RemoveContainer" containerID="a66ed0d45de13097a30df3f4ed95053c2a34b33e7df303a6123a3b585ac69039" Jul 17 01:40:44.507418 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:40:44.507396 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:40:59.162419 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:40:59.162341 2583 scope.go:117] "RemoveContainer" containerID="a66ed0d45de13097a30df3f4ed95053c2a34b33e7df303a6123a3b585ac69039" Jul 17 01:40:59.162887 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:40:59.162532 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:41:05.042058 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:41:05.042015 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:41:13.164893 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:41:13.164852 2583 scope.go:117] "RemoveContainer" containerID="a66ed0d45de13097a30df3f4ed95053c2a34b33e7df303a6123a3b585ac69039" Jul 17 01:41:13.165343 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:41:13.165136 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:41:28.162535 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:41:28.162501 2583 scope.go:117] "RemoveContainer" containerID="a66ed0d45de13097a30df3f4ed95053c2a34b33e7df303a6123a3b585ac69039" Jul 17 01:41:28.162940 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:41:28.162701 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:41:43.165174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:41:43.165144 2583 scope.go:117] "RemoveContainer" containerID="a66ed0d45de13097a30df3f4ed95053c2a34b33e7df303a6123a3b585ac69039" Jul 17 01:41:43.165646 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:41:43.165479 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:41:56.163067 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:41:56.163034 2583 scope.go:117] "RemoveContainer" containerID="a66ed0d45de13097a30df3f4ed95053c2a34b33e7df303a6123a3b585ac69039" Jul 17 01:41:56.163475 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:41:56.163266 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:42:08.088668 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:08.088639 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:08.162428 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:08.162402 2583 scope.go:117] "RemoveContainer" containerID="a66ed0d45de13097a30df3f4ed95053c2a34b33e7df303a6123a3b585ac69039" Jul 17 01:42:08.187257 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:08.187226 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:08.692635 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:08.692595 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" event={"ID":"6f45c4ee-7550-482b-9acc-bb68a8f95b3b","Type":"ContainerStarted","Data":"99fb0abfa0c0f3755fef0dd48761967109d5a680e8018eba86a8e843f6cfa6e8"} Jul 17 01:42:08.692831 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:08.692815 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:42:08.709282 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:08.709227 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podStartSLOduration=170.404324754 podStartE2EDuration="2m54.709210716s" totalImagesPullingTime="4.304885962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:39:14 +0000 UTC" imagePullSessionsCount=6 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:42:08.707626298 +0000 UTC m=+1686.103241174" watchObservedRunningTime="2026-07-17 01:42:08.709210716 +0000 UTC m=+1686.104825566" Jul 17 01:42:09.698610 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:09.698577 2583 generic.go:356] "Generic (PLEG): container finished" podID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerID="99fb0abfa0c0f3755fef0dd48761967109d5a680e8018eba86a8e843f6cfa6e8" exitCode=1 Jul 17 01:42:09.699064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:09.698656 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" event={"ID":"6f45c4ee-7550-482b-9acc-bb68a8f95b3b","Type":"ContainerDied","Data":"99fb0abfa0c0f3755fef0dd48761967109d5a680e8018eba86a8e843f6cfa6e8"} Jul 17 01:42:09.699064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:09.698718 2583 scope.go:117] "RemoveContainer" containerID="a66ed0d45de13097a30df3f4ed95053c2a34b33e7df303a6123a3b585ac69039" Jul 17 01:42:09.699064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:09.698885 2583 scope.go:117] "RemoveContainer" containerID="99fb0abfa0c0f3755fef0dd48761967109d5a680e8018eba86a8e843f6cfa6e8" Jul 17 01:42:09.699226 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:42:09.699115 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:42:10.704530 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:10.704498 2583 scope.go:117] "RemoveContainer" containerID="99fb0abfa0c0f3755fef0dd48761967109d5a680e8018eba86a8e843f6cfa6e8" Jul 17 01:42:10.704904 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:42:10.704697 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:42:14.507380 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:14.507342 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:42:14.507884 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:14.507710 2583 scope.go:117] "RemoveContainer" containerID="99fb0abfa0c0f3755fef0dd48761967109d5a680e8018eba86a8e843f6cfa6e8" Jul 17 01:42:14.507928 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:42:14.507892 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw_odh-ai-gateway-infra(6f45c4ee-7550-482b-9acc-bb68a8f95b3b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" Jul 17 01:42:18.691986 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:18.691953 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:18.792229 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:18.792197 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:19.389203 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:19.389171 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:19.491672 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:19.491635 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:25.181759 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.181724 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw"] Jul 17 01:42:25.202080 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.202051 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:25.315048 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.315025 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:42:25.452018 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.451932 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2mvw\" (UniqueName: \"kubernetes.io/projected/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-kube-api-access-p2mvw\") pod \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\" (UID: \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\") " Jul 17 01:42:25.452018 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.451972 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-openshift-service-ca\") pod \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\" (UID: \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\") " Jul 17 01:42:25.452018 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.452003 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-maas-api-tls\") pod \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\" (UID: \"6f45c4ee-7550-482b-9acc-bb68a8f95b3b\") " Jul 17 01:42:25.452435 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.452402 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f45c4ee-7550-482b-9acc-bb68a8f95b3b" (UID: "6f45c4ee-7550-482b-9acc-bb68a8f95b3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:42:25.454375 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.454348 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f45c4ee-7550-482b-9acc-bb68a8f95b3b" (UID: "6f45c4ee-7550-482b-9acc-bb68a8f95b3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:42:25.454455 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.454391 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-kube-api-access-p2mvw" (OuterVolumeSpecName: "kube-api-access-p2mvw") pod "6f45c4ee-7550-482b-9acc-bb68a8f95b3b" (UID: "6f45c4ee-7550-482b-9acc-bb68a8f95b3b"). InnerVolumeSpecName "kube-api-access-p2mvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:42:25.471868 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.471841 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:25.553392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.553363 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2mvw\" (UniqueName: \"kubernetes.io/projected/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-kube-api-access-p2mvw\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:42:25.553392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.553389 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-openshift-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:42:25.553392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.553399 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f45c4ee-7550-482b-9acc-bb68a8f95b3b-maas-api-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:42:25.773325 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.773255 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" Jul 17 01:42:25.773325 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.773283 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw" event={"ID":"6f45c4ee-7550-482b-9acc-bb68a8f95b3b","Type":"ContainerDied","Data":"ea3280ac5b795ee00d6dbff0eb0cdf92b900628f8d1a0355ddca0681da6cf593"} Jul 17 01:42:25.773521 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.773344 2583 scope.go:117] "RemoveContainer" containerID="99fb0abfa0c0f3755fef0dd48761967109d5a680e8018eba86a8e843f6cfa6e8" Jul 17 01:42:25.794215 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.794184 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw"] Jul 17 01:42:25.799963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:25.799938 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7122b7f-76fc678b7f-zqvrw"] Jul 17 01:42:27.167327 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:27.167294 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" path="/var/lib/kubelet/pods/6f45c4ee-7550-482b-9acc-bb68a8f95b3b/volumes" Jul 17 01:42:31.060484 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:31.060456 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:31.156990 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:31.156955 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:31.365760 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:31.365674 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:31.465918 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:31.465889 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:31.558370 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:31.558337 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:31.613074 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:31.613047 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:32.802451 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.802417 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:32.872923 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.872893 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7"] Jul 17 01:42:32.873233 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.873221 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:32.873296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.873236 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:32.873296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.873246 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:32.873296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.873252 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:32.873296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.873259 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:32.873296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.873264 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:32.873296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.873274 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:32.873296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.873279 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:32.873296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.873286 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:32.873296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.873291 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:32.873556 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.873348 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:32.873556 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.873357 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:32.873556 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.873365 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:32.873556 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.873370 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:32.876314 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.876292 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:42:32.878610 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.878590 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f0d41c05-serving-cert\"" Jul 17 01:42:32.878610 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.878597 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:42:32.878610 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.878609 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 01:42:32.879378 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.879361 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5l8p8\"" Jul 17 01:42:32.887041 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.887016 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7"] Jul 17 01:42:32.908722 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.908688 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:32.983374 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:32.983345 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:33.017622 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.017593 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1572ee5-f099-41e2-a964-8f52e7aeddd4-openshift-service-ca\") pod \"maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7\" (UID: \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:42:33.017761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.017634 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpqfl\" (UniqueName: \"kubernetes.io/projected/d1572ee5-f099-41e2-a964-8f52e7aeddd4-kube-api-access-qpqfl\") pod \"maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7\" (UID: \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:42:33.017761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.017696 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1572ee5-f099-41e2-a964-8f52e7aeddd4-maas-api-tls\") pod \"maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7\" (UID: \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:42:33.118619 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.118539 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1572ee5-f099-41e2-a964-8f52e7aeddd4-openshift-service-ca\") pod \"maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7\" (UID: \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:42:33.118619 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.118585 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpqfl\" (UniqueName: \"kubernetes.io/projected/d1572ee5-f099-41e2-a964-8f52e7aeddd4-kube-api-access-qpqfl\") pod \"maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7\" (UID: \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:42:33.118821 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.118644 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1572ee5-f099-41e2-a964-8f52e7aeddd4-maas-api-tls\") pod \"maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7\" (UID: \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:42:33.118821 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:42:33.118788 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-serving-cert: secret "maas-api-e2e-migrate-f0d41c05-serving-cert" not found Jul 17 01:42:33.118964 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:42:33.118864 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1572ee5-f099-41e2-a964-8f52e7aeddd4-maas-api-tls podName:d1572ee5-f099-41e2-a964-8f52e7aeddd4 nodeName:}" failed. No retries permitted until 2026-07-17 01:42:33.618842628 +0000 UTC m=+1711.014457458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1572ee5-f099-41e2-a964-8f52e7aeddd4-maas-api-tls") pod "maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" (UID: "d1572ee5-f099-41e2-a964-8f52e7aeddd4") : secret "maas-api-e2e-migrate-f0d41c05-serving-cert" not found Jul 17 01:42:33.119303 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.119273 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1572ee5-f099-41e2-a964-8f52e7aeddd4-openshift-service-ca\") pod \"maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7\" (UID: \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:42:33.132727 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.132703 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpqfl\" (UniqueName: \"kubernetes.io/projected/d1572ee5-f099-41e2-a964-8f52e7aeddd4-kube-api-access-qpqfl\") pod \"maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7\" (UID: \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:42:33.372506 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.372405 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:33.475273 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.475233 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:33.573481 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.573446 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:33.622771 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.622677 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1572ee5-f099-41e2-a964-8f52e7aeddd4-maas-api-tls\") pod \"maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7\" (UID: \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:42:33.625355 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.625330 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1572ee5-f099-41e2-a964-8f52e7aeddd4-maas-api-tls\") pod \"maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7\" (UID: \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:42:33.788489 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.788451 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:42:33.924953 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:33.924874 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7"] Jul 17 01:42:33.927268 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:42:33.927237 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1572ee5_f099_41e2_a964_8f52e7aeddd4.slice/crio-a6ba8de7b4c4b927e8592e68563d2a6618dd5697468b934ffb4a63d2fc89637a WatchSource:0}: Error finding container a6ba8de7b4c4b927e8592e68563d2a6618dd5697468b934ffb4a63d2fc89637a: Status 404 returned error can't find the container with id a6ba8de7b4c4b927e8592e68563d2a6618dd5697468b934ffb4a63d2fc89637a Jul 17 01:42:34.816677 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:34.816593 2583 generic.go:356] "Generic (PLEG): container finished" podID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerID="8c130c1e99e75d5e14c7ee8c18b9c44e6822df0fee8f239b362502b666524eaf" exitCode=1 Jul 17 01:42:34.816677 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:34.816644 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" event={"ID":"d1572ee5-f099-41e2-a964-8f52e7aeddd4","Type":"ContainerDied","Data":"8c130c1e99e75d5e14c7ee8c18b9c44e6822df0fee8f239b362502b666524eaf"} Jul 17 01:42:34.816854 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:34.816697 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" event={"ID":"d1572ee5-f099-41e2-a964-8f52e7aeddd4","Type":"ContainerStarted","Data":"a6ba8de7b4c4b927e8592e68563d2a6618dd5697468b934ffb4a63d2fc89637a"} Jul 17 01:42:34.816962 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:34.816948 2583 scope.go:117] "RemoveContainer" containerID="8c130c1e99e75d5e14c7ee8c18b9c44e6822df0fee8f239b362502b666524eaf" Jul 17 01:42:34.917412 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:34.917380 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:42:35.823200 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:35.823166 2583 generic.go:356] "Generic (PLEG): container finished" podID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerID="5a8cb433be8de3e53bde18d3590f4b7ff310612b950a6836d872740496919e09" exitCode=1 Jul 17 01:42:35.823619 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:35.823254 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" event={"ID":"d1572ee5-f099-41e2-a964-8f52e7aeddd4","Type":"ContainerDied","Data":"5a8cb433be8de3e53bde18d3590f4b7ff310612b950a6836d872740496919e09"} Jul 17 01:42:35.823619 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:35.823320 2583 scope.go:117] "RemoveContainer" containerID="8c130c1e99e75d5e14c7ee8c18b9c44e6822df0fee8f239b362502b666524eaf" Jul 17 01:42:35.823619 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:35.823528 2583 scope.go:117] "RemoveContainer" containerID="5a8cb433be8de3e53bde18d3590f4b7ff310612b950a6836d872740496919e09" Jul 17 01:42:35.823771 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:42:35.823715 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7_odh-ai-gateway-infra(d1572ee5-f099-41e2-a964-8f52e7aeddd4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" Jul 17 01:42:36.829567 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:36.829537 2583 scope.go:117] "RemoveContainer" containerID="5a8cb433be8de3e53bde18d3590f4b7ff310612b950a6836d872740496919e09" Jul 17 01:42:36.830044 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:42:36.829738 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7_odh-ai-gateway-infra(d1572ee5-f099-41e2-a964-8f52e7aeddd4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" Jul 17 01:42:37.383972 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:37.383928 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464"] Jul 17 01:42:37.384373 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:37.384355 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:37.384464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:37.384376 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:37.384464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:37.384460 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:42:37.388607 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:37.388584 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" Jul 17 01:42:37.390906 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:37.390883 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:42:37.394809 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:37.394786 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464"] Jul 17 01:42:37.459377 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:37.459338 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd2ace9-e626-4acd-8d88-786b7bc1890a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464\" (UID: \"3bd2ace9-e626-4acd-8d88-786b7bc1890a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" Jul 17 01:42:37.560419 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:37.560378 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd2ace9-e626-4acd-8d88-786b7bc1890a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464\" (UID: \"3bd2ace9-e626-4acd-8d88-786b7bc1890a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" Jul 17 01:42:37.561056 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:37.561027 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd2ace9-e626-4acd-8d88-786b7bc1890a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464\" (UID: \"3bd2ace9-e626-4acd-8d88-786b7bc1890a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" Jul 17 01:42:37.701893 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:37.701799 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" Jul 17 01:42:37.836567 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:37.836546 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464"] Jul 17 01:42:37.839041 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:42:37.839010 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd2ace9_e626_4acd_8d88_786b7bc1890a.slice/crio-57a3f4c41cbd12309baa44196532f76f935c4e90d51c8685bdb15e55cfe51a15 WatchSource:0}: Error finding container 57a3f4c41cbd12309baa44196532f76f935c4e90d51c8685bdb15e55cfe51a15: Status 404 returned error can't find the container with id 57a3f4c41cbd12309baa44196532f76f935c4e90d51c8685bdb15e55cfe51a15 Jul 17 01:42:38.788801 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:38.788770 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:42:38.789204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:38.789190 2583 scope.go:117] "RemoveContainer" containerID="5a8cb433be8de3e53bde18d3590f4b7ff310612b950a6836d872740496919e09" Jul 17 01:42:38.789373 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:42:38.789354 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7_odh-ai-gateway-infra(d1572ee5-f099-41e2-a964-8f52e7aeddd4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" Jul 17 01:42:38.840528 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:38.840491 2583 generic.go:356] "Generic (PLEG): container finished" podID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" containerID="8e0b4bb4c438ea64f2db8a10d866137048d3f788c617f6d00f7e31bbe4706644" exitCode=7 Jul 17 01:42:38.840528 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:38.840530 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" event={"ID":"3bd2ace9-e626-4acd-8d88-786b7bc1890a","Type":"ContainerDied","Data":"8e0b4bb4c438ea64f2db8a10d866137048d3f788c617f6d00f7e31bbe4706644"} Jul 17 01:42:38.840923 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:38.840565 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" event={"ID":"3bd2ace9-e626-4acd-8d88-786b7bc1890a","Type":"ContainerStarted","Data":"57a3f4c41cbd12309baa44196532f76f935c4e90d51c8685bdb15e55cfe51a15"} Jul 17 01:42:38.840923 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:38.840739 2583 scope.go:117] "RemoveContainer" containerID="8e0b4bb4c438ea64f2db8a10d866137048d3f788c617f6d00f7e31bbe4706644" Jul 17 01:42:39.847514 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:39.847481 2583 generic.go:356] "Generic (PLEG): container finished" podID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" containerID="265aa64bcd6095699d2dda960fedf4e38ac0f1257f0866884fbefadbc1c39ce6" exitCode=7 Jul 17 01:42:39.847963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:39.847573 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" event={"ID":"3bd2ace9-e626-4acd-8d88-786b7bc1890a","Type":"ContainerDied","Data":"265aa64bcd6095699d2dda960fedf4e38ac0f1257f0866884fbefadbc1c39ce6"} Jul 17 01:42:39.847963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:39.847644 2583 scope.go:117] "RemoveContainer" containerID="8e0b4bb4c438ea64f2db8a10d866137048d3f788c617f6d00f7e31bbe4706644" Jul 17 01:42:39.847963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:39.847725 2583 scope.go:117] "RemoveContainer" containerID="265aa64bcd6095699d2dda960fedf4e38ac0f1257f0866884fbefadbc1c39ce6" Jul 17 01:42:39.848118 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:42:39.847963 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464_odh-ai-gateway-infra(3bd2ace9-e626-4acd-8d88-786b7bc1890a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" podUID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" Jul 17 01:42:40.854138 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:40.854087 2583 scope.go:117] "RemoveContainer" containerID="265aa64bcd6095699d2dda960fedf4e38ac0f1257f0866884fbefadbc1c39ce6" Jul 17 01:42:40.854516 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:42:40.854291 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464_odh-ai-gateway-infra(3bd2ace9-e626-4acd-8d88-786b7bc1890a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" podUID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" Jul 17 01:42:43.789153 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:43.789090 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:42:43.789549 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:43.789518 2583 scope.go:117] "RemoveContainer" containerID="5a8cb433be8de3e53bde18d3590f4b7ff310612b950a6836d872740496919e09" Jul 17 01:42:43.789729 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:42:43.789708 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7_odh-ai-gateway-infra(d1572ee5-f099-41e2-a964-8f52e7aeddd4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" Jul 17 01:42:53.165155 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:53.165123 2583 scope.go:117] "RemoveContainer" containerID="265aa64bcd6095699d2dda960fedf4e38ac0f1257f0866884fbefadbc1c39ce6" Jul 17 01:42:53.914899 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:53.914866 2583 generic.go:356] "Generic (PLEG): container finished" podID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" containerID="b38791f58154bf50a1875b1abe1b77ee55cc93d21a4eab787fc710c0e0d61833" exitCode=7 Jul 17 01:42:53.915075 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:53.914951 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" event={"ID":"3bd2ace9-e626-4acd-8d88-786b7bc1890a","Type":"ContainerDied","Data":"b38791f58154bf50a1875b1abe1b77ee55cc93d21a4eab787fc710c0e0d61833"} Jul 17 01:42:53.915075 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:53.915015 2583 scope.go:117] "RemoveContainer" containerID="265aa64bcd6095699d2dda960fedf4e38ac0f1257f0866884fbefadbc1c39ce6" Jul 17 01:42:53.915410 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:53.915391 2583 scope.go:117] "RemoveContainer" containerID="b38791f58154bf50a1875b1abe1b77ee55cc93d21a4eab787fc710c0e0d61833" Jul 17 01:42:53.915666 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:42:53.915648 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464_odh-ai-gateway-infra(3bd2ace9-e626-4acd-8d88-786b7bc1890a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" podUID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" Jul 17 01:42:54.937530 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:54.937495 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464"] Jul 17 01:42:55.073915 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:55.073889 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" Jul 17 01:42:55.216937 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:55.216851 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd2ace9-e626-4acd-8d88-786b7bc1890a-maas-api-service-ca\") pod \"3bd2ace9-e626-4acd-8d88-786b7bc1890a\" (UID: \"3bd2ace9-e626-4acd-8d88-786b7bc1890a\") " Jul 17 01:42:55.217264 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:55.217242 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd2ace9-e626-4acd-8d88-786b7bc1890a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3bd2ace9-e626-4acd-8d88-786b7bc1890a" (UID: "3bd2ace9-e626-4acd-8d88-786b7bc1890a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:42:55.317779 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:55.317734 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd2ace9-e626-4acd-8d88-786b7bc1890a-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:42:55.926051 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:55.926023 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" Jul 17 01:42:55.926254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:55.926026 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464" event={"ID":"3bd2ace9-e626-4acd-8d88-786b7bc1890a","Type":"ContainerDied","Data":"57a3f4c41cbd12309baa44196532f76f935c4e90d51c8685bdb15e55cfe51a15"} Jul 17 01:42:55.926254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:55.926155 2583 scope.go:117] "RemoveContainer" containerID="b38791f58154bf50a1875b1abe1b77ee55cc93d21a4eab787fc710c0e0d61833" Jul 17 01:42:55.947688 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:55.947658 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464"] Jul 17 01:42:55.951767 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:55.951737 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-pz464"] Jul 17 01:42:57.168212 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:57.168183 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" path="/var/lib/kubelet/pods/3bd2ace9-e626-4acd-8d88-786b7bc1890a/volumes" Jul 17 01:42:58.162386 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:58.162357 2583 scope.go:117] "RemoveContainer" containerID="5a8cb433be8de3e53bde18d3590f4b7ff310612b950a6836d872740496919e09" Jul 17 01:42:58.942944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:58.942912 2583 generic.go:356] "Generic (PLEG): container finished" podID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerID="5d1abe2324a885c330878c42e7f799e81b8487484feaddab2964d3abd2dc531c" exitCode=1 Jul 17 01:42:58.943408 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:58.942996 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" event={"ID":"d1572ee5-f099-41e2-a964-8f52e7aeddd4","Type":"ContainerDied","Data":"5d1abe2324a885c330878c42e7f799e81b8487484feaddab2964d3abd2dc531c"} Jul 17 01:42:58.943408 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:58.943063 2583 scope.go:117] "RemoveContainer" containerID="5a8cb433be8de3e53bde18d3590f4b7ff310612b950a6836d872740496919e09" Jul 17 01:42:58.943408 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:42:58.943400 2583 scope.go:117] "RemoveContainer" containerID="5d1abe2324a885c330878c42e7f799e81b8487484feaddab2964d3abd2dc531c" Jul 17 01:42:58.943594 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:42:58.943575 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7_odh-ai-gateway-infra(d1572ee5-f099-41e2-a964-8f52e7aeddd4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" Jul 17 01:43:03.788570 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:03.788538 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:43:03.788986 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:03.788635 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:43:03.788986 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:03.788900 2583 scope.go:117] "RemoveContainer" containerID="5d1abe2324a885c330878c42e7f799e81b8487484feaddab2964d3abd2dc531c" Jul 17 01:43:03.789087 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:43:03.789068 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7_odh-ai-gateway-infra(d1572ee5-f099-41e2-a964-8f52e7aeddd4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" Jul 17 01:43:03.967952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:03.967927 2583 scope.go:117] "RemoveContainer" containerID="5d1abe2324a885c330878c42e7f799e81b8487484feaddab2964d3abd2dc531c" Jul 17 01:43:03.968142 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:43:03.968125 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7_odh-ai-gateway-infra(d1572ee5-f099-41e2-a964-8f52e7aeddd4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" Jul 17 01:43:17.162569 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:17.162530 2583 scope.go:117] "RemoveContainer" containerID="5d1abe2324a885c330878c42e7f799e81b8487484feaddab2964d3abd2dc531c" Jul 17 01:43:17.163029 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:43:17.162791 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7_odh-ai-gateway-infra(d1572ee5-f099-41e2-a964-8f52e7aeddd4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" Jul 17 01:43:27.422215 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.422178 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj"] Jul 17 01:43:27.422655 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.422508 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" containerName="revoke-keys" Jul 17 01:43:27.422655 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.422518 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" containerName="revoke-keys" Jul 17 01:43:27.422655 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.422532 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" containerName="revoke-keys" Jul 17 01:43:27.422655 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.422537 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" containerName="revoke-keys" Jul 17 01:43:27.422655 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.422544 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" containerName="revoke-keys" Jul 17 01:43:27.422655 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.422550 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" containerName="revoke-keys" Jul 17 01:43:27.422655 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.422592 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" containerName="revoke-keys" Jul 17 01:43:27.422655 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.422604 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f45c4ee-7550-482b-9acc-bb68a8f95b3b" containerName="maas-api" Jul 17 01:43:27.422655 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.422610 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" containerName="revoke-keys" Jul 17 01:43:27.426879 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.426861 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" Jul 17 01:43:27.429227 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.429204 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:43:27.434369 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.434348 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj"] Jul 17 01:43:27.502058 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.502014 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8aecfa-9826-4794-99d5-950bb9ab1b94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj\" (UID: \"7f8aecfa-9826-4794-99d5-950bb9ab1b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" Jul 17 01:43:27.603216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.603181 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8aecfa-9826-4794-99d5-950bb9ab1b94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj\" (UID: \"7f8aecfa-9826-4794-99d5-950bb9ab1b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" Jul 17 01:43:27.603840 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.603819 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8aecfa-9826-4794-99d5-950bb9ab1b94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj\" (UID: \"7f8aecfa-9826-4794-99d5-950bb9ab1b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" Jul 17 01:43:27.738657 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.738575 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" Jul 17 01:43:27.869357 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:27.869330 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj"] Jul 17 01:43:27.872191 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:43:27.872164 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8aecfa_9826_4794_99d5_950bb9ab1b94.slice/crio-60564f504bdc8017b0e87f395583140c494cf0e6a705c1de2f627aeab2218186 WatchSource:0}: Error finding container 60564f504bdc8017b0e87f395583140c494cf0e6a705c1de2f627aeab2218186: Status 404 returned error can't find the container with id 60564f504bdc8017b0e87f395583140c494cf0e6a705c1de2f627aeab2218186 Jul 17 01:43:28.078068 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:28.078030 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" event={"ID":"7f8aecfa-9826-4794-99d5-950bb9ab1b94","Type":"ContainerStarted","Data":"df3452694eb5cb02b801c0609cd954a358404a48b1c8c98d819d5b115f7de3f4"} Jul 17 01:43:28.078260 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:28.078090 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" event={"ID":"7f8aecfa-9826-4794-99d5-950bb9ab1b94","Type":"ContainerStarted","Data":"60564f504bdc8017b0e87f395583140c494cf0e6a705c1de2f627aeab2218186"} Jul 17 01:43:28.093683 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:28.093628 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" podStartSLOduration=1.093611233 podStartE2EDuration="1.093611233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:43:28.092675227 +0000 UTC m=+1765.488290077" watchObservedRunningTime="2026-07-17 01:43:28.093611233 +0000 UTC m=+1765.489226083" Jul 17 01:43:29.083892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:29.083858 2583 generic.go:356] "Generic (PLEG): container finished" podID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" containerID="df3452694eb5cb02b801c0609cd954a358404a48b1c8c98d819d5b115f7de3f4" exitCode=7 Jul 17 01:43:29.084346 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:29.083952 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" event={"ID":"7f8aecfa-9826-4794-99d5-950bb9ab1b94","Type":"ContainerDied","Data":"df3452694eb5cb02b801c0609cd954a358404a48b1c8c98d819d5b115f7de3f4"} Jul 17 01:43:29.084346 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:29.084173 2583 scope.go:117] "RemoveContainer" containerID="df3452694eb5cb02b801c0609cd954a358404a48b1c8c98d819d5b115f7de3f4" Jul 17 01:43:30.090289 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:30.090259 2583 generic.go:356] "Generic (PLEG): container finished" podID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" containerID="1405c71804149c71de65fb2bd46848dcab5d35c140a396ae0c462c5acca3fafb" exitCode=7 Jul 17 01:43:30.090682 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:30.090348 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" event={"ID":"7f8aecfa-9826-4794-99d5-950bb9ab1b94","Type":"ContainerDied","Data":"1405c71804149c71de65fb2bd46848dcab5d35c140a396ae0c462c5acca3fafb"} Jul 17 01:43:30.090682 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:30.090411 2583 scope.go:117] "RemoveContainer" containerID="df3452694eb5cb02b801c0609cd954a358404a48b1c8c98d819d5b115f7de3f4" Jul 17 01:43:30.090682 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:30.090612 2583 scope.go:117] "RemoveContainer" containerID="1405c71804149c71de65fb2bd46848dcab5d35c140a396ae0c462c5acca3fafb" Jul 17 01:43:30.090855 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:43:30.090811 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj_odh-ai-gateway-infra(7f8aecfa-9826-4794-99d5-950bb9ab1b94)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" podUID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" Jul 17 01:43:30.162915 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:30.162890 2583 scope.go:117] "RemoveContainer" containerID="5d1abe2324a885c330878c42e7f799e81b8487484feaddab2964d3abd2dc531c" Jul 17 01:43:31.095935 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:31.095903 2583 generic.go:356] "Generic (PLEG): container finished" podID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerID="b82f14435b418eaf6405dda2553b0a7ce4a19e488f378d34d90f5167f3178ae8" exitCode=1 Jul 17 01:43:31.096415 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:31.095973 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" event={"ID":"d1572ee5-f099-41e2-a964-8f52e7aeddd4","Type":"ContainerDied","Data":"b82f14435b418eaf6405dda2553b0a7ce4a19e488f378d34d90f5167f3178ae8"} Jul 17 01:43:31.096415 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:31.096033 2583 scope.go:117] "RemoveContainer" containerID="5d1abe2324a885c330878c42e7f799e81b8487484feaddab2964d3abd2dc531c" Jul 17 01:43:31.096550 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:31.096425 2583 scope.go:117] "RemoveContainer" containerID="b82f14435b418eaf6405dda2553b0a7ce4a19e488f378d34d90f5167f3178ae8" Jul 17 01:43:31.096696 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:43:31.096649 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7_odh-ai-gateway-infra(d1572ee5-f099-41e2-a964-8f52e7aeddd4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" Jul 17 01:43:31.097880 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:31.097838 2583 scope.go:117] "RemoveContainer" containerID="1405c71804149c71de65fb2bd46848dcab5d35c140a396ae0c462c5acca3fafb" Jul 17 01:43:31.098008 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:43:31.097990 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj_odh-ai-gateway-infra(7f8aecfa-9826-4794-99d5-950bb9ab1b94)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" podUID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" Jul 17 01:43:33.789291 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:33.789256 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:43:33.789684 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:33.789328 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:43:33.789684 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:33.789653 2583 scope.go:117] "RemoveContainer" containerID="b82f14435b418eaf6405dda2553b0a7ce4a19e488f378d34d90f5167f3178ae8" Jul 17 01:43:33.789831 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:43:33.789812 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7_odh-ai-gateway-infra(d1572ee5-f099-41e2-a964-8f52e7aeddd4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" Jul 17 01:43:37.547185 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:37.547142 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:37.652787 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:37.652748 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:43.224385 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:43.224302 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7"] Jul 17 01:43:43.250338 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:43.250306 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:43.355987 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:43.355956 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:43:43.449172 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:43.449116 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1572ee5-f099-41e2-a964-8f52e7aeddd4-openshift-service-ca\") pod \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\" (UID: \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\") " Jul 17 01:43:43.449386 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:43.449201 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpqfl\" (UniqueName: \"kubernetes.io/projected/d1572ee5-f099-41e2-a964-8f52e7aeddd4-kube-api-access-qpqfl\") pod \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\" (UID: \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\") " Jul 17 01:43:43.449386 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:43.449241 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1572ee5-f099-41e2-a964-8f52e7aeddd4-maas-api-tls\") pod \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\" (UID: \"d1572ee5-f099-41e2-a964-8f52e7aeddd4\") " Jul 17 01:43:43.449525 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:43.449430 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1572ee5-f099-41e2-a964-8f52e7aeddd4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d1572ee5-f099-41e2-a964-8f52e7aeddd4" (UID: "d1572ee5-f099-41e2-a964-8f52e7aeddd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:43:43.451545 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:43.451516 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1572ee5-f099-41e2-a964-8f52e7aeddd4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d1572ee5-f099-41e2-a964-8f52e7aeddd4" (UID: "d1572ee5-f099-41e2-a964-8f52e7aeddd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:43:43.451674 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:43.451597 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1572ee5-f099-41e2-a964-8f52e7aeddd4-kube-api-access-qpqfl" (OuterVolumeSpecName: "kube-api-access-qpqfl") pod "d1572ee5-f099-41e2-a964-8f52e7aeddd4" (UID: "d1572ee5-f099-41e2-a964-8f52e7aeddd4"). InnerVolumeSpecName "kube-api-access-qpqfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:43:43.531593 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:43.531565 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:43.550835 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:43.550793 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1572ee5-f099-41e2-a964-8f52e7aeddd4-openshift-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:43:43.550835 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:43.550829 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpqfl\" (UniqueName: \"kubernetes.io/projected/d1572ee5-f099-41e2-a964-8f52e7aeddd4-kube-api-access-qpqfl\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:43:43.550835 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:43.550843 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1572ee5-f099-41e2-a964-8f52e7aeddd4-maas-api-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:43:44.100905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:44.100872 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:44.162189 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:44.162159 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" event={"ID":"d1572ee5-f099-41e2-a964-8f52e7aeddd4","Type":"ContainerDied","Data":"a6ba8de7b4c4b927e8592e68563d2a6618dd5697468b934ffb4a63d2fc89637a"} Jul 17 01:43:44.162189 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:44.162176 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7" Jul 17 01:43:44.162452 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:44.162214 2583 scope.go:117] "RemoveContainer" containerID="b82f14435b418eaf6405dda2553b0a7ce4a19e488f378d34d90f5167f3178ae8" Jul 17 01:43:44.183776 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:44.183742 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7"] Jul 17 01:43:44.186389 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:44.186364 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f0d41c05-565c478854-4sgn7"] Jul 17 01:43:45.170203 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:45.170159 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" path="/var/lib/kubelet/pods/d1572ee5-f099-41e2-a964-8f52e7aeddd4/volumes" Jul 17 01:43:46.162824 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:46.162793 2583 scope.go:117] "RemoveContainer" containerID="1405c71804149c71de65fb2bd46848dcab5d35c140a396ae0c462c5acca3fafb" Jul 17 01:43:47.177977 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:47.177893 2583 generic.go:356] "Generic (PLEG): container finished" podID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" containerID="da9d7c4a33eda576087190282cd2010ec7d1b1d23b8351b98c3d89c082f881ae" exitCode=6 Jul 17 01:43:47.178384 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:47.177970 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" event={"ID":"7f8aecfa-9826-4794-99d5-950bb9ab1b94","Type":"ContainerDied","Data":"da9d7c4a33eda576087190282cd2010ec7d1b1d23b8351b98c3d89c082f881ae"} Jul 17 01:43:47.178384 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:47.178032 2583 scope.go:117] "RemoveContainer" containerID="1405c71804149c71de65fb2bd46848dcab5d35c140a396ae0c462c5acca3fafb" Jul 17 01:43:47.178472 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:47.178403 2583 scope.go:117] "RemoveContainer" containerID="da9d7c4a33eda576087190282cd2010ec7d1b1d23b8351b98c3d89c082f881ae" Jul 17 01:43:47.178650 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:43:47.178626 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj_odh-ai-gateway-infra(7f8aecfa-9826-4794-99d5-950bb9ab1b94)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" podUID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" Jul 17 01:43:48.203509 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:48.203476 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj"] Jul 17 01:43:48.333525 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:48.333501 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" Jul 17 01:43:48.392012 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:48.391975 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8aecfa-9826-4794-99d5-950bb9ab1b94-maas-api-service-ca\") pod \"7f8aecfa-9826-4794-99d5-950bb9ab1b94\" (UID: \"7f8aecfa-9826-4794-99d5-950bb9ab1b94\") " Jul 17 01:43:48.392375 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:48.392352 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8aecfa-9826-4794-99d5-950bb9ab1b94-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7f8aecfa-9826-4794-99d5-950bb9ab1b94" (UID: "7f8aecfa-9826-4794-99d5-950bb9ab1b94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:43:48.493524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:48.493444 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8aecfa-9826-4794-99d5-950bb9ab1b94-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:43:49.049161 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:49.049124 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:49.150704 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:49.150665 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:49.189479 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:49.189446 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" Jul 17 01:43:49.189479 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:49.189466 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj" event={"ID":"7f8aecfa-9826-4794-99d5-950bb9ab1b94","Type":"ContainerDied","Data":"60564f504bdc8017b0e87f395583140c494cf0e6a705c1de2f627aeab2218186"} Jul 17 01:43:49.189729 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:49.189533 2583 scope.go:117] "RemoveContainer" containerID="da9d7c4a33eda576087190282cd2010ec7d1b1d23b8351b98c3d89c082f881ae" Jul 17 01:43:49.205246 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:49.205212 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj"] Jul 17 01:43:49.213891 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:49.213863 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-bqppj"] Jul 17 01:43:49.280555 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:49.280528 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:49.343970 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:49.343883 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:50.383585 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.383549 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:50.485416 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.485385 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:50.578418 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578387 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh"] Jul 17 01:43:50.578721 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578706 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerName="maas-api" Jul 17 01:43:50.578721 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578721 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerName="maas-api" Jul 17 01:43:50.578827 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578732 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" containerName="revoke-keys" Jul 17 01:43:50.578827 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578738 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" containerName="revoke-keys" Jul 17 01:43:50.578827 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578747 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerName="maas-api" Jul 17 01:43:50.578827 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578752 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerName="maas-api" Jul 17 01:43:50.578827 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578758 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerName="maas-api" Jul 17 01:43:50.578827 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578763 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerName="maas-api" Jul 17 01:43:50.578827 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578772 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" containerName="revoke-keys" Jul 17 01:43:50.578827 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578777 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" containerName="revoke-keys" Jul 17 01:43:50.578827 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578785 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerName="maas-api" Jul 17 01:43:50.578827 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578792 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerName="maas-api" Jul 17 01:43:50.578827 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578807 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" containerName="revoke-keys" Jul 17 01:43:50.578827 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578812 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" containerName="revoke-keys" Jul 17 01:43:50.579253 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578874 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerName="maas-api" Jul 17 01:43:50.579253 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578882 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" containerName="revoke-keys" Jul 17 01:43:50.579253 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578889 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerName="maas-api" Jul 17 01:43:50.579253 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578894 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" containerName="revoke-keys" Jul 17 01:43:50.579253 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578899 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" containerName="revoke-keys" Jul 17 01:43:50.579253 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578905 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerName="maas-api" Jul 17 01:43:50.579253 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.578909 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd2ace9-e626-4acd-8d88-786b7bc1890a" containerName="revoke-keys" Jul 17 01:43:50.583473 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.583454 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:43:50.585844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.585825 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 01:43:50.586651 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.586630 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5l8p8\"" Jul 17 01:43:50.586651 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.586645 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2ef2c65d-serving-cert\"" Jul 17 01:43:50.586809 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.586659 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:43:50.593193 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.593167 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh"] Jul 17 01:43:50.617993 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.617958 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:50.713444 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.713357 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dae2011-0f79-44d3-9eec-8a1e57171dc0-maas-api-tls\") pod \"maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh\" (UID: \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:43:50.713444 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.713432 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dae2011-0f79-44d3-9eec-8a1e57171dc0-openshift-service-ca\") pod \"maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh\" (UID: \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:43:50.713632 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.713475 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrbxr\" (UniqueName: \"kubernetes.io/projected/3dae2011-0f79-44d3-9eec-8a1e57171dc0-kube-api-access-zrbxr\") pod \"maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh\" (UID: \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:43:50.814948 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.814914 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dae2011-0f79-44d3-9eec-8a1e57171dc0-maas-api-tls\") pod \"maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh\" (UID: \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:43:50.815137 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.814980 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dae2011-0f79-44d3-9eec-8a1e57171dc0-openshift-service-ca\") pod \"maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh\" (UID: \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:43:50.815137 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.815027 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrbxr\" (UniqueName: \"kubernetes.io/projected/3dae2011-0f79-44d3-9eec-8a1e57171dc0-kube-api-access-zrbxr\") pod \"maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh\" (UID: \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:43:50.815137 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:43:50.815079 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-serving-cert: secret "maas-api-e2e-ait-2ef2c65d-serving-cert" not found Jul 17 01:43:50.815255 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:43:50.815189 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dae2011-0f79-44d3-9eec-8a1e57171dc0-maas-api-tls podName:3dae2011-0f79-44d3-9eec-8a1e57171dc0 nodeName:}" failed. No retries permitted until 2026-07-17 01:43:51.315167865 +0000 UTC m=+1788.710782693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3dae2011-0f79-44d3-9eec-8a1e57171dc0-maas-api-tls") pod "maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" (UID: "3dae2011-0f79-44d3-9eec-8a1e57171dc0") : secret "maas-api-e2e-ait-2ef2c65d-serving-cert" not found Jul 17 01:43:50.815687 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.815663 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dae2011-0f79-44d3-9eec-8a1e57171dc0-openshift-service-ca\") pod \"maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh\" (UID: \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:43:50.826271 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.826242 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrbxr\" (UniqueName: \"kubernetes.io/projected/3dae2011-0f79-44d3-9eec-8a1e57171dc0-kube-api-access-zrbxr\") pod \"maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh\" (UID: \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:43:50.886112 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:50.886054 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:51.041844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:51.041814 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:51.145499 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:51.145472 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:51.167939 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:51.167911 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f8aecfa-9826-4794-99d5-950bb9ab1b94" path="/var/lib/kubelet/pods/7f8aecfa-9826-4794-99d5-950bb9ab1b94/volumes" Jul 17 01:43:51.319510 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:51.319422 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dae2011-0f79-44d3-9eec-8a1e57171dc0-maas-api-tls\") pod \"maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh\" (UID: \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:43:51.322001 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:51.321978 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dae2011-0f79-44d3-9eec-8a1e57171dc0-maas-api-tls\") pod \"maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh\" (UID: \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:43:51.494702 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:51.494657 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:43:51.670669 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:51.670633 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh"] Jul 17 01:43:51.673652 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:43:51.673620 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dae2011_0f79_44d3_9eec_8a1e57171dc0.slice/crio-539c6afc8603667313c880bd2851de0930033caa04832e57db529bb5adc016a8 WatchSource:0}: Error finding container 539c6afc8603667313c880bd2851de0930033caa04832e57db529bb5adc016a8: Status 404 returned error can't find the container with id 539c6afc8603667313c880bd2851de0930033caa04832e57db529bb5adc016a8 Jul 17 01:43:52.204212 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:52.204169 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" event={"ID":"3dae2011-0f79-44d3-9eec-8a1e57171dc0","Type":"ContainerStarted","Data":"be313ac7d99aae1de2f7ade18742dbf078a1c5a7837548bde0a29c5fd9bd84ca"} Jul 17 01:43:52.204357 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:52.204235 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" event={"ID":"3dae2011-0f79-44d3-9eec-8a1e57171dc0","Type":"ContainerStarted","Data":"539c6afc8603667313c880bd2851de0930033caa04832e57db529bb5adc016a8"} Jul 17 01:43:52.204357 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:52.204300 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:43:52.222351 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:52.222289 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podStartSLOduration=1.809845326 podStartE2EDuration="2.222265687s" totalImagesPullingTime="412.420361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:43:52.218753594 +0000 UTC m=+1789.614368449" watchObservedRunningTime="2026-07-17 01:43:52.222265687 +0000 UTC m=+1789.617880540" Jul 17 01:43:52.579506 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:52.579470 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:43:53.210455 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:53.210373 2583 generic.go:356] "Generic (PLEG): container finished" podID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerID="be313ac7d99aae1de2f7ade18742dbf078a1c5a7837548bde0a29c5fd9bd84ca" exitCode=1 Jul 17 01:43:53.210455 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:53.210426 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" event={"ID":"3dae2011-0f79-44d3-9eec-8a1e57171dc0","Type":"ContainerDied","Data":"be313ac7d99aae1de2f7ade18742dbf078a1c5a7837548bde0a29c5fd9bd84ca"} Jul 17 01:43:53.210714 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:53.210697 2583 scope.go:117] "RemoveContainer" containerID="be313ac7d99aae1de2f7ade18742dbf078a1c5a7837548bde0a29c5fd9bd84ca" Jul 17 01:43:54.219018 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:54.218912 2583 generic.go:356] "Generic (PLEG): container finished" podID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerID="42f11ae45a7cac22ba78a21e8ac5cd139141c441d77436741c3dfa91a6426c1f" exitCode=1 Jul 17 01:43:54.219018 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:54.218987 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" event={"ID":"3dae2011-0f79-44d3-9eec-8a1e57171dc0","Type":"ContainerDied","Data":"42f11ae45a7cac22ba78a21e8ac5cd139141c441d77436741c3dfa91a6426c1f"} Jul 17 01:43:54.219551 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:54.219046 2583 scope.go:117] "RemoveContainer" containerID="be313ac7d99aae1de2f7ade18742dbf078a1c5a7837548bde0a29c5fd9bd84ca" Jul 17 01:43:54.219551 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:54.219324 2583 scope.go:117] "RemoveContainer" containerID="42f11ae45a7cac22ba78a21e8ac5cd139141c441d77436741c3dfa91a6426c1f" Jul 17 01:43:54.219648 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:43:54.219569 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:43:55.225708 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:55.225684 2583 scope.go:117] "RemoveContainer" containerID="42f11ae45a7cac22ba78a21e8ac5cd139141c441d77436741c3dfa91a6426c1f" Jul 17 01:43:55.226124 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:43:55.225847 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:43:57.204608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:57.204566 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:43:57.204992 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:43:57.204955 2583 scope.go:117] "RemoveContainer" containerID="42f11ae45a7cac22ba78a21e8ac5cd139141c441d77436741c3dfa91a6426c1f" Jul 17 01:43:57.205185 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:43:57.205164 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:44:01.495914 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:01.495873 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:44:01.496405 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:01.496301 2583 scope.go:117] "RemoveContainer" containerID="42f11ae45a7cac22ba78a21e8ac5cd139141c441d77436741c3dfa91a6426c1f" Jul 17 01:44:01.496484 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:44:01.496461 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:44:03.169699 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:03.169670 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:44:03.171991 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:03.171971 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:44:04.866085 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:04.866048 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:44:04.966215 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:04.966182 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:44:16.162342 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:16.162312 2583 scope.go:117] "RemoveContainer" containerID="42f11ae45a7cac22ba78a21e8ac5cd139141c441d77436741c3dfa91a6426c1f" Jul 17 01:44:16.162760 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:16.162627 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 01:44:19.327498 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:19.327466 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" event={"ID":"3dae2011-0f79-44d3-9eec-8a1e57171dc0","Type":"ContainerStarted","Data":"e622b0b76f4fd694b3efe29dfe6d401916d84490de2f00ce59f042a810f5e801"} Jul 17 01:44:19.327936 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:19.327676 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:44:20.333814 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:20.333779 2583 generic.go:356] "Generic (PLEG): container finished" podID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerID="e622b0b76f4fd694b3efe29dfe6d401916d84490de2f00ce59f042a810f5e801" exitCode=1 Jul 17 01:44:20.334275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:20.333833 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" event={"ID":"3dae2011-0f79-44d3-9eec-8a1e57171dc0","Type":"ContainerDied","Data":"e622b0b76f4fd694b3efe29dfe6d401916d84490de2f00ce59f042a810f5e801"} Jul 17 01:44:20.334275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:20.333878 2583 scope.go:117] "RemoveContainer" containerID="42f11ae45a7cac22ba78a21e8ac5cd139141c441d77436741c3dfa91a6426c1f" Jul 17 01:44:20.334275 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:20.334048 2583 scope.go:117] "RemoveContainer" containerID="e622b0b76f4fd694b3efe29dfe6d401916d84490de2f00ce59f042a810f5e801" Jul 17 01:44:20.334275 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:44:20.334263 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:44:21.340316 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:21.340289 2583 scope.go:117] "RemoveContainer" containerID="e622b0b76f4fd694b3efe29dfe6d401916d84490de2f00ce59f042a810f5e801" Jul 17 01:44:21.340684 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:44:21.340462 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:44:21.495344 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:21.495311 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:44:22.350156 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:22.350129 2583 scope.go:117] "RemoveContainer" containerID="e622b0b76f4fd694b3efe29dfe6d401916d84490de2f00ce59f042a810f5e801" Jul 17 01:44:22.350531 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:44:22.350292 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:44:34.162963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:34.162923 2583 scope.go:117] "RemoveContainer" containerID="e622b0b76f4fd694b3efe29dfe6d401916d84490de2f00ce59f042a810f5e801" Jul 17 01:44:34.163541 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:44:34.163210 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:44:49.163196 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:49.163162 2583 scope.go:117] "RemoveContainer" containerID="e622b0b76f4fd694b3efe29dfe6d401916d84490de2f00ce59f042a810f5e801" Jul 17 01:44:50.475840 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:50.475805 2583 generic.go:356] "Generic (PLEG): container finished" podID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerID="3315b81dff7a31c7b2c46930bc7cf59cae2877d508cd618a5bedd755a32fcba0" exitCode=1 Jul 17 01:44:50.475840 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:50.475845 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" event={"ID":"3dae2011-0f79-44d3-9eec-8a1e57171dc0","Type":"ContainerDied","Data":"3315b81dff7a31c7b2c46930bc7cf59cae2877d508cd618a5bedd755a32fcba0"} Jul 17 01:44:50.476343 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:50.475916 2583 scope.go:117] "RemoveContainer" containerID="e622b0b76f4fd694b3efe29dfe6d401916d84490de2f00ce59f042a810f5e801" Jul 17 01:44:50.476343 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:50.476274 2583 scope.go:117] "RemoveContainer" containerID="3315b81dff7a31c7b2c46930bc7cf59cae2877d508cd618a5bedd755a32fcba0" Jul 17 01:44:50.476481 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:44:50.476458 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:44:51.495189 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:51.495160 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:44:51.495576 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:51.495544 2583 scope.go:117] "RemoveContainer" containerID="3315b81dff7a31c7b2c46930bc7cf59cae2877d508cd618a5bedd755a32fcba0" Jul 17 01:44:51.495758 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:44:51.495738 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:44:54.327969 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:54.327938 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:44:54.328360 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:44:54.328350 2583 scope.go:117] "RemoveContainer" containerID="3315b81dff7a31c7b2c46930bc7cf59cae2877d508cd618a5bedd755a32fcba0" Jul 17 01:44:54.328524 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:44:54.328506 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:45:00.140493 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:00.140452 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737545-rdh5q"] Jul 17 01:45:00.140882 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:00.140854 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1572ee5-f099-41e2-a964-8f52e7aeddd4" containerName="maas-api" Jul 17 01:45:00.145232 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:00.145204 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737545-rdh5q" Jul 17 01:45:00.147565 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:00.147528 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:45:00.153245 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:00.153216 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737545-rdh5q"] Jul 17 01:45:00.160316 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:00.160274 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737545-rdh5q" Jul 17 01:45:00.301272 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:00.301245 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737545-rdh5q"] Jul 17 01:45:00.303690 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:45:00.303667 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4eacf93_c3e3_425c_9522_862425ddd03b.slice/crio-2b7625152648b4067334709ee82b85f0bbff8d00bca60a8fc9b0bf6a693df6bc WatchSource:0}: Error finding container 2b7625152648b4067334709ee82b85f0bbff8d00bca60a8fc9b0bf6a693df6bc: Status 404 returned error can't find the container with id 2b7625152648b4067334709ee82b85f0bbff8d00bca60a8fc9b0bf6a693df6bc Jul 17 01:45:00.522312 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:00.522232 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737545-rdh5q" event={"ID":"a4eacf93-c3e3-425c-9522-862425ddd03b","Type":"ContainerStarted","Data":"2b7625152648b4067334709ee82b85f0bbff8d00bca60a8fc9b0bf6a693df6bc"} Jul 17 01:45:01.527621 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:01.527584 2583 generic.go:356] "Generic (PLEG): container finished" podID="a4eacf93-c3e3-425c-9522-862425ddd03b" containerID="e056422ecfc4add23c5d736150a59554426a6b18ff55dbcccd1c90716d9a7397" exitCode=0 Jul 17 01:45:01.528069 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:01.527644 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737545-rdh5q" event={"ID":"a4eacf93-c3e3-425c-9522-862425ddd03b","Type":"ContainerDied","Data":"e056422ecfc4add23c5d736150a59554426a6b18ff55dbcccd1c90716d9a7397"} Jul 17 01:45:02.669902 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:02.669873 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737545-rdh5q" Jul 17 01:45:03.539529 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:03.539492 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737545-rdh5q" event={"ID":"a4eacf93-c3e3-425c-9522-862425ddd03b","Type":"ContainerDied","Data":"2b7625152648b4067334709ee82b85f0bbff8d00bca60a8fc9b0bf6a693df6bc"} Jul 17 01:45:03.539730 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:03.539550 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b7625152648b4067334709ee82b85f0bbff8d00bca60a8fc9b0bf6a693df6bc" Jul 17 01:45:03.539730 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:03.539508 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737545-rdh5q" Jul 17 01:45:07.162675 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:07.162641 2583 scope.go:117] "RemoveContainer" containerID="3315b81dff7a31c7b2c46930bc7cf59cae2877d508cd618a5bedd755a32fcba0" Jul 17 01:45:07.163190 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:45:07.162864 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:45:17.508336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:17.508292 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7"] Jul 17 01:45:17.508798 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:17.508644 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4eacf93-c3e3-425c-9522-862425ddd03b" containerName="cleanup" Jul 17 01:45:17.508798 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:17.508655 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4eacf93-c3e3-425c-9522-862425ddd03b" containerName="cleanup" Jul 17 01:45:17.508798 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:17.508706 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4eacf93-c3e3-425c-9522-862425ddd03b" containerName="cleanup" Jul 17 01:45:17.511717 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:17.511701 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" Jul 17 01:45:17.514035 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:17.514015 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:45:17.518444 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:17.518419 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7"] Jul 17 01:45:17.572961 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:17.572917 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34038833-1ecd-4b11-90b0-c50b8a4585ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7\" (UID: \"34038833-1ecd-4b11-90b0-c50b8a4585ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" Jul 17 01:45:17.674354 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:17.674317 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34038833-1ecd-4b11-90b0-c50b8a4585ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7\" (UID: \"34038833-1ecd-4b11-90b0-c50b8a4585ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" Jul 17 01:45:17.674962 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:17.674944 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34038833-1ecd-4b11-90b0-c50b8a4585ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7\" (UID: \"34038833-1ecd-4b11-90b0-c50b8a4585ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" Jul 17 01:45:17.823323 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:17.823290 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" Jul 17 01:45:17.956118 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:17.956057 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7"] Jul 17 01:45:17.959744 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:45:17.959716 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34038833_1ecd_4b11_90b0_c50b8a4585ff.slice/crio-5da52953bd4647fe901dc96bfd365ead31045c6032b5a7f00262e5ef771060bb WatchSource:0}: Error finding container 5da52953bd4647fe901dc96bfd365ead31045c6032b5a7f00262e5ef771060bb: Status 404 returned error can't find the container with id 5da52953bd4647fe901dc96bfd365ead31045c6032b5a7f00262e5ef771060bb Jul 17 01:45:18.608660 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:18.608621 2583 generic.go:356] "Generic (PLEG): container finished" podID="34038833-1ecd-4b11-90b0-c50b8a4585ff" containerID="5f538a1c0ccd82ef890277ad2f49619256b0a51b39f3bd6bc4fab3bfc12095cc" exitCode=6 Jul 17 01:45:18.609035 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:18.608697 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" event={"ID":"34038833-1ecd-4b11-90b0-c50b8a4585ff","Type":"ContainerDied","Data":"5f538a1c0ccd82ef890277ad2f49619256b0a51b39f3bd6bc4fab3bfc12095cc"} Jul 17 01:45:18.609035 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:18.608778 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" event={"ID":"34038833-1ecd-4b11-90b0-c50b8a4585ff","Type":"ContainerStarted","Data":"5da52953bd4647fe901dc96bfd365ead31045c6032b5a7f00262e5ef771060bb"} Jul 17 01:45:18.609035 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:18.608976 2583 scope.go:117] "RemoveContainer" containerID="5f538a1c0ccd82ef890277ad2f49619256b0a51b39f3bd6bc4fab3bfc12095cc" Jul 17 01:45:19.615733 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:19.615699 2583 generic.go:356] "Generic (PLEG): container finished" podID="34038833-1ecd-4b11-90b0-c50b8a4585ff" containerID="25550b97ecbe6814911b50ea7fa7f36aec6b6e7317c57b904c70102c4da81f0c" exitCode=6 Jul 17 01:45:19.616221 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:19.615774 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" event={"ID":"34038833-1ecd-4b11-90b0-c50b8a4585ff","Type":"ContainerDied","Data":"25550b97ecbe6814911b50ea7fa7f36aec6b6e7317c57b904c70102c4da81f0c"} Jul 17 01:45:19.616221 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:19.615838 2583 scope.go:117] "RemoveContainer" containerID="5f538a1c0ccd82ef890277ad2f49619256b0a51b39f3bd6bc4fab3bfc12095cc" Jul 17 01:45:19.616221 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:19.616125 2583 scope.go:117] "RemoveContainer" containerID="25550b97ecbe6814911b50ea7fa7f36aec6b6e7317c57b904c70102c4da81f0c" Jul 17 01:45:19.616354 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:45:19.616325 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7_odh-ai-gateway-infra(34038833-1ecd-4b11-90b0-c50b8a4585ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" podUID="34038833-1ecd-4b11-90b0-c50b8a4585ff" Jul 17 01:45:20.621347 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:20.621317 2583 scope.go:117] "RemoveContainer" containerID="25550b97ecbe6814911b50ea7fa7f36aec6b6e7317c57b904c70102c4da81f0c" Jul 17 01:45:20.621723 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:45:20.621499 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7_odh-ai-gateway-infra(34038833-1ecd-4b11-90b0-c50b8a4585ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" podUID="34038833-1ecd-4b11-90b0-c50b8a4585ff" Jul 17 01:45:22.162727 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:22.162694 2583 scope.go:117] "RemoveContainer" containerID="3315b81dff7a31c7b2c46930bc7cf59cae2877d508cd618a5bedd755a32fcba0" Jul 17 01:45:22.163235 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:45:22.162889 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:45:34.162993 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:34.162955 2583 scope.go:117] "RemoveContainer" containerID="3315b81dff7a31c7b2c46930bc7cf59cae2877d508cd618a5bedd755a32fcba0" Jul 17 01:45:34.691187 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:34.691153 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" event={"ID":"3dae2011-0f79-44d3-9eec-8a1e57171dc0","Type":"ContainerStarted","Data":"39ac0d1f53ed5902ce9245ae594d3de78c997aafe20ede9f95e918288e98d78c"} Jul 17 01:45:34.691390 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:34.691375 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:45:35.162329 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:35.162299 2583 scope.go:117] "RemoveContainer" containerID="25550b97ecbe6814911b50ea7fa7f36aec6b6e7317c57b904c70102c4da81f0c" Jul 17 01:45:35.577746 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:35.577705 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8"] Jul 17 01:45:35.581144 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:35.581128 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" Jul 17 01:45:35.588721 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:35.588689 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8"] Jul 17 01:45:35.697326 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:35.697290 2583 generic.go:356] "Generic (PLEG): container finished" podID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerID="39ac0d1f53ed5902ce9245ae594d3de78c997aafe20ede9f95e918288e98d78c" exitCode=1 Jul 17 01:45:35.697508 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:35.697367 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" event={"ID":"3dae2011-0f79-44d3-9eec-8a1e57171dc0","Type":"ContainerDied","Data":"39ac0d1f53ed5902ce9245ae594d3de78c997aafe20ede9f95e918288e98d78c"} Jul 17 01:45:35.697508 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:35.697424 2583 scope.go:117] "RemoveContainer" containerID="3315b81dff7a31c7b2c46930bc7cf59cae2877d508cd618a5bedd755a32fcba0" Jul 17 01:45:35.697687 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:35.697663 2583 scope.go:117] "RemoveContainer" containerID="39ac0d1f53ed5902ce9245ae594d3de78c997aafe20ede9f95e918288e98d78c" Jul 17 01:45:35.697912 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:45:35.697873 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:45:35.699389 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:35.699369 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" event={"ID":"34038833-1ecd-4b11-90b0-c50b8a4585ff","Type":"ContainerStarted","Data":"181837b69c6aa41740a5dd457e64d9dcf9df5f896e3a1055b4655bff17daef43"} Jul 17 01:45:35.726170 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:35.726091 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" podStartSLOduration=18.726076534 podStartE2EDuration="18.726076534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:45:35.72500169 +0000 UTC m=+1893.120616542" watchObservedRunningTime="2026-07-17 01:45:35.726076534 +0000 UTC m=+1893.121691385" Jul 17 01:45:35.747206 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:35.747165 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48dc7ec-b437-4aac-b6aa-e92509c48378-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8\" (UID: \"c48dc7ec-b437-4aac-b6aa-e92509c48378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" Jul 17 01:45:35.848070 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:35.847983 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48dc7ec-b437-4aac-b6aa-e92509c48378-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8\" (UID: \"c48dc7ec-b437-4aac-b6aa-e92509c48378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" Jul 17 01:45:35.848643 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:35.848619 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48dc7ec-b437-4aac-b6aa-e92509c48378-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8\" (UID: \"c48dc7ec-b437-4aac-b6aa-e92509c48378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" Jul 17 01:45:35.892546 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:35.892506 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" Jul 17 01:45:36.033215 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:36.033180 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8"] Jul 17 01:45:36.035379 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:45:36.035347 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48dc7ec_b437_4aac_b6aa_e92509c48378.slice/crio-3ef75d0acaa60554d0838844ece52ba712aa0dc7e4d50352cb3615c07282907c WatchSource:0}: Error finding container 3ef75d0acaa60554d0838844ece52ba712aa0dc7e4d50352cb3615c07282907c: Status 404 returned error can't find the container with id 3ef75d0acaa60554d0838844ece52ba712aa0dc7e4d50352cb3615c07282907c Jul 17 01:45:36.710692 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:36.710662 2583 scope.go:117] "RemoveContainer" containerID="39ac0d1f53ed5902ce9245ae594d3de78c997aafe20ede9f95e918288e98d78c" Jul 17 01:45:36.711176 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:45:36.710908 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:45:36.712293 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:36.712271 2583 generic.go:356] "Generic (PLEG): container finished" podID="34038833-1ecd-4b11-90b0-c50b8a4585ff" containerID="181837b69c6aa41740a5dd457e64d9dcf9df5f896e3a1055b4655bff17daef43" exitCode=6 Jul 17 01:45:36.712399 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:36.712342 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" event={"ID":"34038833-1ecd-4b11-90b0-c50b8a4585ff","Type":"ContainerDied","Data":"181837b69c6aa41740a5dd457e64d9dcf9df5f896e3a1055b4655bff17daef43"} Jul 17 01:45:36.712458 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:36.712444 2583 scope.go:117] "RemoveContainer" containerID="25550b97ecbe6814911b50ea7fa7f36aec6b6e7317c57b904c70102c4da81f0c" Jul 17 01:45:36.712743 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:36.712715 2583 scope.go:117] "RemoveContainer" containerID="181837b69c6aa41740a5dd457e64d9dcf9df5f896e3a1055b4655bff17daef43" Jul 17 01:45:36.712959 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:45:36.712937 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7_odh-ai-gateway-infra(34038833-1ecd-4b11-90b0-c50b8a4585ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" podUID="34038833-1ecd-4b11-90b0-c50b8a4585ff" Jul 17 01:45:36.714025 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:36.714002 2583 generic.go:356] "Generic (PLEG): container finished" podID="c48dc7ec-b437-4aac-b6aa-e92509c48378" containerID="319b561807d0e7d90b2e63bfe647ba895c88594e8b3b62c55e6e16a79787f516" exitCode=7 Jul 17 01:45:36.714130 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:36.714038 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" event={"ID":"c48dc7ec-b437-4aac-b6aa-e92509c48378","Type":"ContainerDied","Data":"319b561807d0e7d90b2e63bfe647ba895c88594e8b3b62c55e6e16a79787f516"} Jul 17 01:45:36.714130 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:36.714076 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" event={"ID":"c48dc7ec-b437-4aac-b6aa-e92509c48378","Type":"ContainerStarted","Data":"3ef75d0acaa60554d0838844ece52ba712aa0dc7e4d50352cb3615c07282907c"} Jul 17 01:45:36.714266 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:36.714253 2583 scope.go:117] "RemoveContainer" containerID="319b561807d0e7d90b2e63bfe647ba895c88594e8b3b62c55e6e16a79787f516" Jul 17 01:45:36.734409 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:36.734372 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7"] Jul 17 01:45:37.721514 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:37.721483 2583 generic.go:356] "Generic (PLEG): container finished" podID="c48dc7ec-b437-4aac-b6aa-e92509c48378" containerID="5ceabd22deb6588c7f15b358d0c132526cf8bfdcfd3a93274fc0d6cdf5484a65" exitCode=7 Jul 17 01:45:37.721881 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:37.721544 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" event={"ID":"c48dc7ec-b437-4aac-b6aa-e92509c48378","Type":"ContainerDied","Data":"5ceabd22deb6588c7f15b358d0c132526cf8bfdcfd3a93274fc0d6cdf5484a65"} Jul 17 01:45:37.721881 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:37.721601 2583 scope.go:117] "RemoveContainer" containerID="319b561807d0e7d90b2e63bfe647ba895c88594e8b3b62c55e6e16a79787f516" Jul 17 01:45:37.721881 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:37.721811 2583 scope.go:117] "RemoveContainer" containerID="5ceabd22deb6588c7f15b358d0c132526cf8bfdcfd3a93274fc0d6cdf5484a65" Jul 17 01:45:37.722069 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:45:37.722045 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8_odh-ai-gateway-infra(c48dc7ec-b437-4aac-b6aa-e92509c48378)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" podUID="c48dc7ec-b437-4aac-b6aa-e92509c48378" Jul 17 01:45:37.853639 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:37.853614 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" Jul 17 01:45:37.967136 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:37.967077 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34038833-1ecd-4b11-90b0-c50b8a4585ff-maas-api-service-ca\") pod \"34038833-1ecd-4b11-90b0-c50b8a4585ff\" (UID: \"34038833-1ecd-4b11-90b0-c50b8a4585ff\") " Jul 17 01:45:37.967463 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:37.967436 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34038833-1ecd-4b11-90b0-c50b8a4585ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34038833-1ecd-4b11-90b0-c50b8a4585ff" (UID: "34038833-1ecd-4b11-90b0-c50b8a4585ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:45:38.068436 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:38.068391 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34038833-1ecd-4b11-90b0-c50b8a4585ff-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:45:38.727669 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:38.727639 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" Jul 17 01:45:38.728170 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:38.727671 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7" event={"ID":"34038833-1ecd-4b11-90b0-c50b8a4585ff","Type":"ContainerDied","Data":"5da52953bd4647fe901dc96bfd365ead31045c6032b5a7f00262e5ef771060bb"} Jul 17 01:45:38.728170 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:38.727743 2583 scope.go:117] "RemoveContainer" containerID="181837b69c6aa41740a5dd457e64d9dcf9df5f896e3a1055b4655bff17daef43" Jul 17 01:45:38.729558 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:38.729539 2583 scope.go:117] "RemoveContainer" containerID="5ceabd22deb6588c7f15b358d0c132526cf8bfdcfd3a93274fc0d6cdf5484a65" Jul 17 01:45:38.729771 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:45:38.729749 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8_odh-ai-gateway-infra(c48dc7ec-b437-4aac-b6aa-e92509c48378)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" podUID="c48dc7ec-b437-4aac-b6aa-e92509c48378" Jul 17 01:45:38.756945 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:38.756902 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7"] Jul 17 01:45:38.758472 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:38.758450 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nddz7"] Jul 17 01:45:39.168084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:39.168048 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34038833-1ecd-4b11-90b0-c50b8a4585ff" path="/var/lib/kubelet/pods/34038833-1ecd-4b11-90b0-c50b8a4585ff/volumes" Jul 17 01:45:41.495569 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:41.495512 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:45:41.495980 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:41.495945 2583 scope.go:117] "RemoveContainer" containerID="39ac0d1f53ed5902ce9245ae594d3de78c997aafe20ede9f95e918288e98d78c" Jul 17 01:45:41.496178 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:45:41.496158 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:45:50.162536 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:50.162506 2583 scope.go:117] "RemoveContainer" containerID="5ceabd22deb6588c7f15b358d0c132526cf8bfdcfd3a93274fc0d6cdf5484a65" Jul 17 01:45:50.787132 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:50.787075 2583 generic.go:356] "Generic (PLEG): container finished" podID="c48dc7ec-b437-4aac-b6aa-e92509c48378" containerID="191fe9bbbf5709dd17d456eeb3584a5b216355d041d9f9a8728219c7bb82a741" exitCode=7 Jul 17 01:45:50.787132 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:50.787129 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" event={"ID":"c48dc7ec-b437-4aac-b6aa-e92509c48378","Type":"ContainerDied","Data":"191fe9bbbf5709dd17d456eeb3584a5b216355d041d9f9a8728219c7bb82a741"} Jul 17 01:45:50.787358 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:50.787188 2583 scope.go:117] "RemoveContainer" containerID="5ceabd22deb6588c7f15b358d0c132526cf8bfdcfd3a93274fc0d6cdf5484a65" Jul 17 01:45:50.787555 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:50.787534 2583 scope.go:117] "RemoveContainer" containerID="191fe9bbbf5709dd17d456eeb3584a5b216355d041d9f9a8728219c7bb82a741" Jul 17 01:45:50.787764 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:45:50.787745 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8_odh-ai-gateway-infra(c48dc7ec-b437-4aac-b6aa-e92509c48378)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" podUID="c48dc7ec-b437-4aac-b6aa-e92509c48378" Jul 17 01:45:51.810634 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:51.810597 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8"] Jul 17 01:45:51.946759 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:51.946734 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" Jul 17 01:45:51.987939 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:51.987906 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48dc7ec-b437-4aac-b6aa-e92509c48378-maas-api-service-ca\") pod \"c48dc7ec-b437-4aac-b6aa-e92509c48378\" (UID: \"c48dc7ec-b437-4aac-b6aa-e92509c48378\") " Jul 17 01:45:51.988296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:51.988276 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48dc7ec-b437-4aac-b6aa-e92509c48378-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c48dc7ec-b437-4aac-b6aa-e92509c48378" (UID: "c48dc7ec-b437-4aac-b6aa-e92509c48378"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:45:52.088969 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:52.088863 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c48dc7ec-b437-4aac-b6aa-e92509c48378-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:45:52.800142 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:52.800083 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" event={"ID":"c48dc7ec-b437-4aac-b6aa-e92509c48378","Type":"ContainerDied","Data":"3ef75d0acaa60554d0838844ece52ba712aa0dc7e4d50352cb3615c07282907c"} Jul 17 01:45:52.800142 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:52.800134 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8" Jul 17 01:45:52.800395 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:52.800181 2583 scope.go:117] "RemoveContainer" containerID="191fe9bbbf5709dd17d456eeb3584a5b216355d041d9f9a8728219c7bb82a741" Jul 17 01:45:52.821050 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:52.821009 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8"] Jul 17 01:45:52.826281 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:52.826251 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-4dpz8"] Jul 17 01:45:53.165789 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:53.165715 2583 scope.go:117] "RemoveContainer" containerID="39ac0d1f53ed5902ce9245ae594d3de78c997aafe20ede9f95e918288e98d78c" Jul 17 01:45:53.165949 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:45:53.165913 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:45:53.169820 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:45:53.169798 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c48dc7ec-b437-4aac-b6aa-e92509c48378" path="/var/lib/kubelet/pods/c48dc7ec-b437-4aac-b6aa-e92509c48378/volumes" Jul 17 01:46:04.162553 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:04.162506 2583 scope.go:117] "RemoveContainer" containerID="39ac0d1f53ed5902ce9245ae594d3de78c997aafe20ede9f95e918288e98d78c" Jul 17 01:46:04.163018 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:46:04.162805 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:46:19.163228 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:19.163196 2583 scope.go:117] "RemoveContainer" containerID="39ac0d1f53ed5902ce9245ae594d3de78c997aafe20ede9f95e918288e98d78c" Jul 17 01:46:19.163710 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:46:19.163419 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:46:25.613762 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.613730 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg"] Jul 17 01:46:25.614174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.614039 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34038833-1ecd-4b11-90b0-c50b8a4585ff" containerName="revoke-keys" Jul 17 01:46:25.614174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.614050 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="34038833-1ecd-4b11-90b0-c50b8a4585ff" containerName="revoke-keys" Jul 17 01:46:25.614174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.614057 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c48dc7ec-b437-4aac-b6aa-e92509c48378" containerName="revoke-keys" Jul 17 01:46:25.614174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.614063 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c48dc7ec-b437-4aac-b6aa-e92509c48378" containerName="revoke-keys" Jul 17 01:46:25.614174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.614074 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c48dc7ec-b437-4aac-b6aa-e92509c48378" containerName="revoke-keys" Jul 17 01:46:25.614174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.614080 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c48dc7ec-b437-4aac-b6aa-e92509c48378" containerName="revoke-keys" Jul 17 01:46:25.614174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.614093 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34038833-1ecd-4b11-90b0-c50b8a4585ff" containerName="revoke-keys" Jul 17 01:46:25.614174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.614113 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="34038833-1ecd-4b11-90b0-c50b8a4585ff" containerName="revoke-keys" Jul 17 01:46:25.614174 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.614173 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c48dc7ec-b437-4aac-b6aa-e92509c48378" containerName="revoke-keys" Jul 17 01:46:25.614452 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.614181 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c48dc7ec-b437-4aac-b6aa-e92509c48378" containerName="revoke-keys" Jul 17 01:46:25.614452 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.614188 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="34038833-1ecd-4b11-90b0-c50b8a4585ff" containerName="revoke-keys" Jul 17 01:46:25.614452 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.614194 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="34038833-1ecd-4b11-90b0-c50b8a4585ff" containerName="revoke-keys" Jul 17 01:46:25.614452 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.614200 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="34038833-1ecd-4b11-90b0-c50b8a4585ff" containerName="revoke-keys" Jul 17 01:46:25.616905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.616890 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" Jul 17 01:46:25.619221 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.619202 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:46:25.624456 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.624432 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg"] Jul 17 01:46:25.784971 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.784936 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24a4fae-2209-4bda-a193-f9e7a4f318d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg\" (UID: \"e24a4fae-2209-4bda-a193-f9e7a4f318d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" Jul 17 01:46:25.885877 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.885799 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24a4fae-2209-4bda-a193-f9e7a4f318d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg\" (UID: \"e24a4fae-2209-4bda-a193-f9e7a4f318d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" Jul 17 01:46:25.886469 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.886450 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24a4fae-2209-4bda-a193-f9e7a4f318d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg\" (UID: \"e24a4fae-2209-4bda-a193-f9e7a4f318d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" Jul 17 01:46:25.928017 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:25.927988 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" Jul 17 01:46:26.056489 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:26.056466 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg"] Jul 17 01:46:26.058864 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:46:26.058838 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24a4fae_2209_4bda_a193_f9e7a4f318d0.slice/crio-bed9f63c539cd64c7e3c824c7650ef4e6951ad01828a8507dce4b4bb1b4c04f2 WatchSource:0}: Error finding container bed9f63c539cd64c7e3c824c7650ef4e6951ad01828a8507dce4b4bb1b4c04f2: Status 404 returned error can't find the container with id bed9f63c539cd64c7e3c824c7650ef4e6951ad01828a8507dce4b4bb1b4c04f2 Jul 17 01:46:26.954552 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:26.954519 2583 generic.go:356] "Generic (PLEG): container finished" podID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" containerID="82edd729baf6281bd3d6642d9f7d8991ebe88d6f49daeb7d531f8ef76b0de47c" exitCode=7 Jul 17 01:46:26.954926 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:26.954562 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" event={"ID":"e24a4fae-2209-4bda-a193-f9e7a4f318d0","Type":"ContainerDied","Data":"82edd729baf6281bd3d6642d9f7d8991ebe88d6f49daeb7d531f8ef76b0de47c"} Jul 17 01:46:26.954926 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:26.954600 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" event={"ID":"e24a4fae-2209-4bda-a193-f9e7a4f318d0","Type":"ContainerStarted","Data":"bed9f63c539cd64c7e3c824c7650ef4e6951ad01828a8507dce4b4bb1b4c04f2"} Jul 17 01:46:26.954926 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:26.954799 2583 scope.go:117] "RemoveContainer" containerID="82edd729baf6281bd3d6642d9f7d8991ebe88d6f49daeb7d531f8ef76b0de47c" Jul 17 01:46:27.961043 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:27.961010 2583 generic.go:356] "Generic (PLEG): container finished" podID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" containerID="2e3a0cb1ae5d233c630ddc521210fb2764ee6d3ecafcb7abc547225e5bd6fe8f" exitCode=7 Jul 17 01:46:27.961475 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:27.961088 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" event={"ID":"e24a4fae-2209-4bda-a193-f9e7a4f318d0","Type":"ContainerDied","Data":"2e3a0cb1ae5d233c630ddc521210fb2764ee6d3ecafcb7abc547225e5bd6fe8f"} Jul 17 01:46:27.961475 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:27.961179 2583 scope.go:117] "RemoveContainer" containerID="82edd729baf6281bd3d6642d9f7d8991ebe88d6f49daeb7d531f8ef76b0de47c" Jul 17 01:46:27.961475 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:27.961415 2583 scope.go:117] "RemoveContainer" containerID="2e3a0cb1ae5d233c630ddc521210fb2764ee6d3ecafcb7abc547225e5bd6fe8f" Jul 17 01:46:27.961638 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:46:27.961619 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg_odh-ai-gateway-infra(e24a4fae-2209-4bda-a193-f9e7a4f318d0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" podUID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" Jul 17 01:46:28.967580 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:28.967551 2583 scope.go:117] "RemoveContainer" containerID="2e3a0cb1ae5d233c630ddc521210fb2764ee6d3ecafcb7abc547225e5bd6fe8f" Jul 17 01:46:28.967963 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:46:28.967739 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg_odh-ai-gateway-infra(e24a4fae-2209-4bda-a193-f9e7a4f318d0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" podUID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" Jul 17 01:46:34.162305 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:34.162274 2583 scope.go:117] "RemoveContainer" containerID="39ac0d1f53ed5902ce9245ae594d3de78c997aafe20ede9f95e918288e98d78c" Jul 17 01:46:34.162692 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:46:34.162432 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:46:40.162844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:40.162818 2583 scope.go:117] "RemoveContainer" containerID="2e3a0cb1ae5d233c630ddc521210fb2764ee6d3ecafcb7abc547225e5bd6fe8f" Jul 17 01:46:41.021554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:41.021469 2583 generic.go:356] "Generic (PLEG): container finished" podID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" containerID="6e0b4f3dd8bb4d2957def7dff75b2ef5a5c3e9ee011bb02f6ade0e0fc178e87e" exitCode=7 Jul 17 01:46:41.021554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:41.021516 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" event={"ID":"e24a4fae-2209-4bda-a193-f9e7a4f318d0","Type":"ContainerDied","Data":"6e0b4f3dd8bb4d2957def7dff75b2ef5a5c3e9ee011bb02f6ade0e0fc178e87e"} Jul 17 01:46:41.021732 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:41.021563 2583 scope.go:117] "RemoveContainer" containerID="2e3a0cb1ae5d233c630ddc521210fb2764ee6d3ecafcb7abc547225e5bd6fe8f" Jul 17 01:46:41.021832 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:41.021814 2583 scope.go:117] "RemoveContainer" containerID="6e0b4f3dd8bb4d2957def7dff75b2ef5a5c3e9ee011bb02f6ade0e0fc178e87e" Jul 17 01:46:41.022046 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:46:41.022026 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg_odh-ai-gateway-infra(e24a4fae-2209-4bda-a193-f9e7a4f318d0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" podUID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" Jul 17 01:46:42.047998 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:42.047962 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg"] Jul 17 01:46:42.185957 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:42.185931 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" Jul 17 01:46:42.229831 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:42.229803 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24a4fae-2209-4bda-a193-f9e7a4f318d0-maas-api-service-ca\") pod \"e24a4fae-2209-4bda-a193-f9e7a4f318d0\" (UID: \"e24a4fae-2209-4bda-a193-f9e7a4f318d0\") " Jul 17 01:46:42.230144 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:42.230121 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e24a4fae-2209-4bda-a193-f9e7a4f318d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e24a4fae-2209-4bda-a193-f9e7a4f318d0" (UID: "e24a4fae-2209-4bda-a193-f9e7a4f318d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:46:42.331313 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:42.331238 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e24a4fae-2209-4bda-a193-f9e7a4f318d0-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:46:43.032910 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:43.032879 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" Jul 17 01:46:43.032910 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:43.032890 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg" event={"ID":"e24a4fae-2209-4bda-a193-f9e7a4f318d0","Type":"ContainerDied","Data":"bed9f63c539cd64c7e3c824c7650ef4e6951ad01828a8507dce4b4bb1b4c04f2"} Jul 17 01:46:43.033276 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:43.032952 2583 scope.go:117] "RemoveContainer" containerID="6e0b4f3dd8bb4d2957def7dff75b2ef5a5c3e9ee011bb02f6ade0e0fc178e87e" Jul 17 01:46:43.051672 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:43.051648 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg"] Jul 17 01:46:43.055944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:43.055921 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-mz2hg"] Jul 17 01:46:43.167232 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:43.167197 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" path="/var/lib/kubelet/pods/e24a4fae-2209-4bda-a193-f9e7a4f318d0/volumes" Jul 17 01:46:47.163302 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:47.163276 2583 scope.go:117] "RemoveContainer" containerID="39ac0d1f53ed5902ce9245ae594d3de78c997aafe20ede9f95e918288e98d78c" Jul 17 01:46:47.163692 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:46:47.163454 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:46:59.162733 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:46:59.162649 2583 scope.go:117] "RemoveContainer" containerID="39ac0d1f53ed5902ce9245ae594d3de78c997aafe20ede9f95e918288e98d78c" Jul 17 01:47:00.109027 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:00.108984 2583 generic.go:356] "Generic (PLEG): container finished" podID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" exitCode=1 Jul 17 01:47:00.109231 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:00.109054 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" event={"ID":"3dae2011-0f79-44d3-9eec-8a1e57171dc0","Type":"ContainerDied","Data":"640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c"} Jul 17 01:47:00.109231 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:00.109135 2583 scope.go:117] "RemoveContainer" containerID="39ac0d1f53ed5902ce9245ae594d3de78c997aafe20ede9f95e918288e98d78c" Jul 17 01:47:00.109454 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:00.109438 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:47:00.109684 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:47:00.109663 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:47:01.495414 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:01.495379 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:47:01.495859 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:01.495759 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:47:01.495992 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:47:01.495971 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:47:04.692181 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:04.692127 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:47:04.692589 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:04.692518 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:47:04.692716 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:47:04.692695 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:47:08.916143 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:08.916089 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:47:10.317567 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:10.317535 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:47:15.518698 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.518657 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:47:15.616913 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.616882 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:47:15.654607 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.654570 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh"] Jul 17 01:47:15.654901 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.654888 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34038833-1ecd-4b11-90b0-c50b8a4585ff" containerName="revoke-keys" Jul 17 01:47:15.654956 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.654903 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="34038833-1ecd-4b11-90b0-c50b8a4585ff" containerName="revoke-keys" Jul 17 01:47:15.654956 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.654918 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" containerName="revoke-keys" Jul 17 01:47:15.654956 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.654923 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" containerName="revoke-keys" Jul 17 01:47:15.654956 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.654933 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c48dc7ec-b437-4aac-b6aa-e92509c48378" containerName="revoke-keys" Jul 17 01:47:15.654956 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.654939 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c48dc7ec-b437-4aac-b6aa-e92509c48378" containerName="revoke-keys" Jul 17 01:47:15.654956 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.654951 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" containerName="revoke-keys" Jul 17 01:47:15.654956 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.654957 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" containerName="revoke-keys" Jul 17 01:47:15.655214 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.655012 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" containerName="revoke-keys" Jul 17 01:47:15.655214 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.655022 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c48dc7ec-b437-4aac-b6aa-e92509c48378" containerName="revoke-keys" Jul 17 01:47:15.655214 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.655029 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" containerName="revoke-keys" Jul 17 01:47:15.657779 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.657764 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" Jul 17 01:47:15.660107 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.660074 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:47:15.665770 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.665439 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh"] Jul 17 01:47:15.830867 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.830825 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ae7e10-0439-4ffa-9dfd-9f6ad46655da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh\" (UID: \"83ae7e10-0439-4ffa-9dfd-9f6ad46655da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" Jul 17 01:47:15.932170 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.932133 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ae7e10-0439-4ffa-9dfd-9f6ad46655da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh\" (UID: \"83ae7e10-0439-4ffa-9dfd-9f6ad46655da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" Jul 17 01:47:15.932735 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.932715 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ae7e10-0439-4ffa-9dfd-9f6ad46655da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh\" (UID: \"83ae7e10-0439-4ffa-9dfd-9f6ad46655da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" Jul 17 01:47:15.969823 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:15.969789 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" Jul 17 01:47:16.100619 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:16.100592 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh"] Jul 17 01:47:16.102813 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:47:16.102788 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ae7e10_0439_4ffa_9dfd_9f6ad46655da.slice/crio-67395c0c3a29842dee604ea90eada7a41609064d6b75ac1ac9324349c097e69a WatchSource:0}: Error finding container 67395c0c3a29842dee604ea90eada7a41609064d6b75ac1ac9324349c097e69a: Status 404 returned error can't find the container with id 67395c0c3a29842dee604ea90eada7a41609064d6b75ac1ac9324349c097e69a Jul 17 01:47:16.180713 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:16.180676 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" event={"ID":"83ae7e10-0439-4ffa-9dfd-9f6ad46655da","Type":"ContainerStarted","Data":"67395c0c3a29842dee604ea90eada7a41609064d6b75ac1ac9324349c097e69a"} Jul 17 01:47:17.162921 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:17.162894 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:47:17.163330 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:47:17.163082 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:47:17.186427 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:17.186392 2583 generic.go:356] "Generic (PLEG): container finished" podID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" containerID="886c4b48a4d1324157fdc5b83d3ece2fe0d11e428e26feaa842ef9d664495cc3" exitCode=7 Jul 17 01:47:17.186616 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:17.186476 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" event={"ID":"83ae7e10-0439-4ffa-9dfd-9f6ad46655da","Type":"ContainerDied","Data":"886c4b48a4d1324157fdc5b83d3ece2fe0d11e428e26feaa842ef9d664495cc3"} Jul 17 01:47:17.186733 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:17.186717 2583 scope.go:117] "RemoveContainer" containerID="886c4b48a4d1324157fdc5b83d3ece2fe0d11e428e26feaa842ef9d664495cc3" Jul 17 01:47:18.193053 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:18.192964 2583 generic.go:356] "Generic (PLEG): container finished" podID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" containerID="59f2682f8c2a7ca44e73c9a6bc8a0ccefe4c5e2b9602338c4d6e34d68451d094" exitCode=7 Jul 17 01:47:18.193578 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:18.193048 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" event={"ID":"83ae7e10-0439-4ffa-9dfd-9f6ad46655da","Type":"ContainerDied","Data":"59f2682f8c2a7ca44e73c9a6bc8a0ccefe4c5e2b9602338c4d6e34d68451d094"} Jul 17 01:47:18.193578 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:18.193136 2583 scope.go:117] "RemoveContainer" containerID="886c4b48a4d1324157fdc5b83d3ece2fe0d11e428e26feaa842ef9d664495cc3" Jul 17 01:47:18.193578 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:18.193284 2583 scope.go:117] "RemoveContainer" containerID="59f2682f8c2a7ca44e73c9a6bc8a0ccefe4c5e2b9602338c4d6e34d68451d094" Jul 17 01:47:18.193578 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:47:18.193492 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh_odh-ai-gateway-infra(83ae7e10-0439-4ffa-9dfd-9f6ad46655da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" podUID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" Jul 17 01:47:19.199513 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:19.199482 2583 scope.go:117] "RemoveContainer" containerID="59f2682f8c2a7ca44e73c9a6bc8a0ccefe4c5e2b9602338c4d6e34d68451d094" Jul 17 01:47:19.199909 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:47:19.199698 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh_odh-ai-gateway-infra(83ae7e10-0439-4ffa-9dfd-9f6ad46655da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" podUID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" Jul 17 01:47:30.163030 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:30.162997 2583 scope.go:117] "RemoveContainer" containerID="59f2682f8c2a7ca44e73c9a6bc8a0ccefe4c5e2b9602338c4d6e34d68451d094" Jul 17 01:47:31.252863 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:31.252832 2583 generic.go:356] "Generic (PLEG): container finished" podID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" containerID="12097e07a65da1b62b1677a351dd0da12b7eaecce81f7cae4a95eca70ab42e30" exitCode=7 Jul 17 01:47:31.253292 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:31.252880 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" event={"ID":"83ae7e10-0439-4ffa-9dfd-9f6ad46655da","Type":"ContainerDied","Data":"12097e07a65da1b62b1677a351dd0da12b7eaecce81f7cae4a95eca70ab42e30"} Jul 17 01:47:31.253292 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:31.252925 2583 scope.go:117] "RemoveContainer" containerID="59f2682f8c2a7ca44e73c9a6bc8a0ccefe4c5e2b9602338c4d6e34d68451d094" Jul 17 01:47:31.253292 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:31.253225 2583 scope.go:117] "RemoveContainer" containerID="12097e07a65da1b62b1677a351dd0da12b7eaecce81f7cae4a95eca70ab42e30" Jul 17 01:47:31.253467 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:47:31.253442 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh_odh-ai-gateway-infra(83ae7e10-0439-4ffa-9dfd-9f6ad46655da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" podUID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" Jul 17 01:47:32.162963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:32.162931 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:47:32.163142 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:47:32.163125 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:47:32.278518 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:32.278482 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh"] Jul 17 01:47:32.408784 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:32.408753 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" Jul 17 01:47:32.456189 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:32.456090 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ae7e10-0439-4ffa-9dfd-9f6ad46655da-maas-api-service-ca\") pod \"83ae7e10-0439-4ffa-9dfd-9f6ad46655da\" (UID: \"83ae7e10-0439-4ffa-9dfd-9f6ad46655da\") " Jul 17 01:47:32.456425 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:32.456403 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ae7e10-0439-4ffa-9dfd-9f6ad46655da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83ae7e10-0439-4ffa-9dfd-9f6ad46655da" (UID: "83ae7e10-0439-4ffa-9dfd-9f6ad46655da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:47:32.557660 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:32.557614 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ae7e10-0439-4ffa-9dfd-9f6ad46655da-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:47:33.264287 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:33.264254 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" Jul 17 01:47:33.264460 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:33.264290 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh" event={"ID":"83ae7e10-0439-4ffa-9dfd-9f6ad46655da","Type":"ContainerDied","Data":"67395c0c3a29842dee604ea90eada7a41609064d6b75ac1ac9324349c097e69a"} Jul 17 01:47:33.264460 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:33.264353 2583 scope.go:117] "RemoveContainer" containerID="12097e07a65da1b62b1677a351dd0da12b7eaecce81f7cae4a95eca70ab42e30" Jul 17 01:47:33.279072 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:33.279047 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh"] Jul 17 01:47:33.282574 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:33.282551 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-tqpbh"] Jul 17 01:47:35.167845 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:35.167798 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" path="/var/lib/kubelet/pods/83ae7e10-0439-4ffa-9dfd-9f6ad46655da/volumes" Jul 17 01:47:46.162266 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:46.162238 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:47:46.162762 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:47:46.162409 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:47:59.163250 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:47:59.163220 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:47:59.163655 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:47:59.163394 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:48:07.634593 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.634556 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr"] Jul 17 01:48:07.635002 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.634927 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" containerName="revoke-keys" Jul 17 01:48:07.635002 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.634940 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" containerName="revoke-keys" Jul 17 01:48:07.635002 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.634953 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" containerName="revoke-keys" Jul 17 01:48:07.635002 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.634959 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" containerName="revoke-keys" Jul 17 01:48:07.635002 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.634965 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" containerName="revoke-keys" Jul 17 01:48:07.635002 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.634971 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" containerName="revoke-keys" Jul 17 01:48:07.635002 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.634986 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" containerName="revoke-keys" Jul 17 01:48:07.635002 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.634994 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" containerName="revoke-keys" Jul 17 01:48:07.635301 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.635056 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e24a4fae-2209-4bda-a193-f9e7a4f318d0" containerName="revoke-keys" Jul 17 01:48:07.635301 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.635067 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" containerName="revoke-keys" Jul 17 01:48:07.635301 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.635079 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" containerName="revoke-keys" Jul 17 01:48:07.635301 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.635083 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ae7e10-0439-4ffa-9dfd-9f6ad46655da" containerName="revoke-keys" Jul 17 01:48:07.638000 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.637983 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" Jul 17 01:48:07.640224 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.640200 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:48:07.645135 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.645081 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr"] Jul 17 01:48:07.753363 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.753314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b057ea-453d-4647-a164-7f7dae35470d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr\" (UID: \"59b057ea-453d-4647-a164-7f7dae35470d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" Jul 17 01:48:07.854245 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.854212 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b057ea-453d-4647-a164-7f7dae35470d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr\" (UID: \"59b057ea-453d-4647-a164-7f7dae35470d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" Jul 17 01:48:07.854869 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.854845 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b057ea-453d-4647-a164-7f7dae35470d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr\" (UID: \"59b057ea-453d-4647-a164-7f7dae35470d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" Jul 17 01:48:07.950419 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:07.950311 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" Jul 17 01:48:08.085033 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:08.084998 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr"] Jul 17 01:48:08.088256 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:48:08.088225 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b057ea_453d_4647_a164_7f7dae35470d.slice/crio-b6d2b1e77e977f4dcf0bc5a766d390317d10caf433e6a9c8d8823e2d663eaef3 WatchSource:0}: Error finding container b6d2b1e77e977f4dcf0bc5a766d390317d10caf433e6a9c8d8823e2d663eaef3: Status 404 returned error can't find the container with id b6d2b1e77e977f4dcf0bc5a766d390317d10caf433e6a9c8d8823e2d663eaef3 Jul 17 01:48:08.422138 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:08.422075 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" event={"ID":"59b057ea-453d-4647-a164-7f7dae35470d","Type":"ContainerStarted","Data":"71873775df494b295da0c77854d1d508967db98155fc375183f53bbfcdd127d5"} Jul 17 01:48:08.422310 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:08.422162 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" event={"ID":"59b057ea-453d-4647-a164-7f7dae35470d","Type":"ContainerStarted","Data":"b6d2b1e77e977f4dcf0bc5a766d390317d10caf433e6a9c8d8823e2d663eaef3"} Jul 17 01:48:08.438597 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:08.438540 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" podStartSLOduration=1.43852105 podStartE2EDuration="1.43852105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:48:08.436767461 +0000 UTC m=+2045.832382313" watchObservedRunningTime="2026-07-17 01:48:08.43852105 +0000 UTC m=+2045.834135901" Jul 17 01:48:09.428395 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:09.428358 2583 generic.go:356] "Generic (PLEG): container finished" podID="59b057ea-453d-4647-a164-7f7dae35470d" containerID="71873775df494b295da0c77854d1d508967db98155fc375183f53bbfcdd127d5" exitCode=6 Jul 17 01:48:09.428788 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:09.428427 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" event={"ID":"59b057ea-453d-4647-a164-7f7dae35470d","Type":"ContainerDied","Data":"71873775df494b295da0c77854d1d508967db98155fc375183f53bbfcdd127d5"} Jul 17 01:48:09.428788 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:09.428668 2583 scope.go:117] "RemoveContainer" containerID="71873775df494b295da0c77854d1d508967db98155fc375183f53bbfcdd127d5" Jul 17 01:48:10.435482 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:10.435390 2583 generic.go:356] "Generic (PLEG): container finished" podID="59b057ea-453d-4647-a164-7f7dae35470d" containerID="288ddbcf5155fcb0758993e7cac8c9844fd4c6ec7567b740a832fc45a3a827f3" exitCode=6 Jul 17 01:48:10.435482 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:10.435455 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" event={"ID":"59b057ea-453d-4647-a164-7f7dae35470d","Type":"ContainerDied","Data":"288ddbcf5155fcb0758993e7cac8c9844fd4c6ec7567b740a832fc45a3a827f3"} Jul 17 01:48:10.435945 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:10.435498 2583 scope.go:117] "RemoveContainer" containerID="71873775df494b295da0c77854d1d508967db98155fc375183f53bbfcdd127d5" Jul 17 01:48:10.435945 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:10.435642 2583 scope.go:117] "RemoveContainer" containerID="288ddbcf5155fcb0758993e7cac8c9844fd4c6ec7567b740a832fc45a3a827f3" Jul 17 01:48:10.435945 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:48:10.435891 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr_odh-ai-gateway-infra(59b057ea-453d-4647-a164-7f7dae35470d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" podUID="59b057ea-453d-4647-a164-7f7dae35470d" Jul 17 01:48:11.162747 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:11.162706 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:48:11.162966 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:48:11.162854 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:48:11.442255 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:11.442178 2583 scope.go:117] "RemoveContainer" containerID="288ddbcf5155fcb0758993e7cac8c9844fd4c6ec7567b740a832fc45a3a827f3" Jul 17 01:48:11.442608 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:48:11.442368 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr_odh-ai-gateway-infra(59b057ea-453d-4647-a164-7f7dae35470d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" podUID="59b057ea-453d-4647-a164-7f7dae35470d" Jul 17 01:48:22.163075 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:22.163039 2583 scope.go:117] "RemoveContainer" containerID="288ddbcf5155fcb0758993e7cac8c9844fd4c6ec7567b740a832fc45a3a827f3" Jul 17 01:48:22.494663 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:22.494578 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" event={"ID":"59b057ea-453d-4647-a164-7f7dae35470d","Type":"ContainerStarted","Data":"fed02eebbd921c2b8cd02460380c56224f8e81b68dc8a2871c63fbe63f16f6ab"} Jul 17 01:48:23.164904 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:23.164870 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:48:23.165312 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:48:23.165050 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:48:23.501171 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:23.501139 2583 generic.go:356] "Generic (PLEG): container finished" podID="59b057ea-453d-4647-a164-7f7dae35470d" containerID="fed02eebbd921c2b8cd02460380c56224f8e81b68dc8a2871c63fbe63f16f6ab" exitCode=6 Jul 17 01:48:23.501359 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:23.501194 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" event={"ID":"59b057ea-453d-4647-a164-7f7dae35470d","Type":"ContainerDied","Data":"fed02eebbd921c2b8cd02460380c56224f8e81b68dc8a2871c63fbe63f16f6ab"} Jul 17 01:48:23.501359 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:23.501244 2583 scope.go:117] "RemoveContainer" containerID="288ddbcf5155fcb0758993e7cac8c9844fd4c6ec7567b740a832fc45a3a827f3" Jul 17 01:48:23.501555 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:23.501536 2583 scope.go:117] "RemoveContainer" containerID="fed02eebbd921c2b8cd02460380c56224f8e81b68dc8a2871c63fbe63f16f6ab" Jul 17 01:48:23.501774 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:48:23.501749 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr_odh-ai-gateway-infra(59b057ea-453d-4647-a164-7f7dae35470d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" podUID="59b057ea-453d-4647-a164-7f7dae35470d" Jul 17 01:48:23.523266 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:23.523234 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr"] Jul 17 01:48:24.646251 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:24.646223 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" Jul 17 01:48:24.703185 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:24.703146 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b057ea-453d-4647-a164-7f7dae35470d-maas-api-service-ca\") pod \"59b057ea-453d-4647-a164-7f7dae35470d\" (UID: \"59b057ea-453d-4647-a164-7f7dae35470d\") " Jul 17 01:48:24.703555 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:24.703525 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b057ea-453d-4647-a164-7f7dae35470d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59b057ea-453d-4647-a164-7f7dae35470d" (UID: "59b057ea-453d-4647-a164-7f7dae35470d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:48:24.803703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:24.803671 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59b057ea-453d-4647-a164-7f7dae35470d-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:48:25.512185 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:25.512069 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" event={"ID":"59b057ea-453d-4647-a164-7f7dae35470d","Type":"ContainerDied","Data":"b6d2b1e77e977f4dcf0bc5a766d390317d10caf433e6a9c8d8823e2d663eaef3"} Jul 17 01:48:25.512185 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:25.512160 2583 scope.go:117] "RemoveContainer" containerID="fed02eebbd921c2b8cd02460380c56224f8e81b68dc8a2871c63fbe63f16f6ab" Jul 17 01:48:25.512440 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:25.512082 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr" Jul 17 01:48:25.527595 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:25.527564 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr"] Jul 17 01:48:25.533415 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:25.533387 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-mjggr"] Jul 17 01:48:27.168093 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:27.168054 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59b057ea-453d-4647-a164-7f7dae35470d" path="/var/lib/kubelet/pods/59b057ea-453d-4647-a164-7f7dae35470d/volumes" Jul 17 01:48:36.163232 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:36.163200 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:48:36.163652 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:48:36.163382 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:48:50.162471 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:50.162443 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:48:50.162863 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:48:50.162629 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:48:57.677930 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:57.677885 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89"] Jul 17 01:48:57.678558 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:57.678381 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b057ea-453d-4647-a164-7f7dae35470d" containerName="revoke-keys" Jul 17 01:48:57.678558 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:57.678401 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b057ea-453d-4647-a164-7f7dae35470d" containerName="revoke-keys" Jul 17 01:48:57.678558 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:57.678425 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b057ea-453d-4647-a164-7f7dae35470d" containerName="revoke-keys" Jul 17 01:48:57.678558 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:57.678435 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b057ea-453d-4647-a164-7f7dae35470d" containerName="revoke-keys" Jul 17 01:48:57.678558 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:57.678539 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b057ea-453d-4647-a164-7f7dae35470d" containerName="revoke-keys" Jul 17 01:48:57.678558 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:57.678550 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b057ea-453d-4647-a164-7f7dae35470d" containerName="revoke-keys" Jul 17 01:48:57.681432 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:57.681411 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" Jul 17 01:48:57.683840 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:57.683821 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:48:57.688025 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:57.688000 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89"] Jul 17 01:48:57.794131 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:57.794061 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24dde6b-91b2-41cb-ac18-4b7436cce94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89\" (UID: \"a24dde6b-91b2-41cb-ac18-4b7436cce94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" Jul 17 01:48:57.895501 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:57.895464 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24dde6b-91b2-41cb-ac18-4b7436cce94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89\" (UID: \"a24dde6b-91b2-41cb-ac18-4b7436cce94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" Jul 17 01:48:57.896091 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:57.896069 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24dde6b-91b2-41cb-ac18-4b7436cce94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89\" (UID: \"a24dde6b-91b2-41cb-ac18-4b7436cce94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" Jul 17 01:48:57.994268 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:57.994172 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" Jul 17 01:48:58.125695 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:58.125671 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89"] Jul 17 01:48:58.128380 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:48:58.128355 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24dde6b_91b2_41cb_ac18_4b7436cce94f.slice/crio-f90ce36fe278fece337b4b981dc7c5807b9a848a44d0091a406e9c5704b991f4 WatchSource:0}: Error finding container f90ce36fe278fece337b4b981dc7c5807b9a848a44d0091a406e9c5704b991f4: Status 404 returned error can't find the container with id f90ce36fe278fece337b4b981dc7c5807b9a848a44d0091a406e9c5704b991f4 Jul 17 01:48:58.658305 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:58.658213 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" event={"ID":"a24dde6b-91b2-41cb-ac18-4b7436cce94f","Type":"ContainerStarted","Data":"7be113cfbbda781639f65c9f2fbdff99f20de6ac0456d1795c654405d2b3ff9e"} Jul 17 01:48:58.658305 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:58.658266 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" event={"ID":"a24dde6b-91b2-41cb-ac18-4b7436cce94f","Type":"ContainerStarted","Data":"f90ce36fe278fece337b4b981dc7c5807b9a848a44d0091a406e9c5704b991f4"} Jul 17 01:48:58.673910 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:58.673855 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" podStartSLOduration=1.673838141 podStartE2EDuration="1.673838141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:48:58.672137623 +0000 UTC m=+2096.067752469" watchObservedRunningTime="2026-07-17 01:48:58.673838141 +0000 UTC m=+2096.069452992" Jul 17 01:48:59.649994 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:59.649957 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c"] Jul 17 01:48:59.650410 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:59.650330 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b057ea-453d-4647-a164-7f7dae35470d" containerName="revoke-keys" Jul 17 01:48:59.650410 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:59.650347 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b057ea-453d-4647-a164-7f7dae35470d" containerName="revoke-keys" Jul 17 01:48:59.650498 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:59.650432 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b057ea-453d-4647-a164-7f7dae35470d" containerName="revoke-keys" Jul 17 01:48:59.653403 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:59.653385 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" Jul 17 01:48:59.660285 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:59.660258 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c"] Jul 17 01:48:59.665378 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:59.665344 2583 generic.go:356] "Generic (PLEG): container finished" podID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" containerID="7be113cfbbda781639f65c9f2fbdff99f20de6ac0456d1795c654405d2b3ff9e" exitCode=6 Jul 17 01:48:59.665509 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:59.665423 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" event={"ID":"a24dde6b-91b2-41cb-ac18-4b7436cce94f","Type":"ContainerDied","Data":"7be113cfbbda781639f65c9f2fbdff99f20de6ac0456d1795c654405d2b3ff9e"} Jul 17 01:48:59.665601 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:59.665586 2583 scope.go:117] "RemoveContainer" containerID="7be113cfbbda781639f65c9f2fbdff99f20de6ac0456d1795c654405d2b3ff9e" Jul 17 01:48:59.811571 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:59.811528 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c\" (UID: \"8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" Jul 17 01:48:59.912055 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:59.911962 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c\" (UID: \"8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" Jul 17 01:48:59.912665 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:59.912640 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c\" (UID: \"8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" Jul 17 01:48:59.965393 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:48:59.965357 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" Jul 17 01:49:00.100328 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:00.100292 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c"] Jul 17 01:49:00.103312 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:49:00.103282 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff901fb_6a11_41f9_af8e_f8b0b8e79cf9.slice/crio-84c222ec1a37bcea3f49f400da56a90fccaa5e51b88d7fff69f563a07410822f WatchSource:0}: Error finding container 84c222ec1a37bcea3f49f400da56a90fccaa5e51b88d7fff69f563a07410822f: Status 404 returned error can't find the container with id 84c222ec1a37bcea3f49f400da56a90fccaa5e51b88d7fff69f563a07410822f Jul 17 01:49:00.671704 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:00.671657 2583 generic.go:356] "Generic (PLEG): container finished" podID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" containerID="ada0bd3ef9b7c79f647310e7aa070ba4898aaae2eababeeafe84b8aa3b6a5dae" exitCode=6 Jul 17 01:49:00.672187 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:00.671704 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" event={"ID":"a24dde6b-91b2-41cb-ac18-4b7436cce94f","Type":"ContainerDied","Data":"ada0bd3ef9b7c79f647310e7aa070ba4898aaae2eababeeafe84b8aa3b6a5dae"} Jul 17 01:49:00.672187 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:00.671774 2583 scope.go:117] "RemoveContainer" containerID="7be113cfbbda781639f65c9f2fbdff99f20de6ac0456d1795c654405d2b3ff9e" Jul 17 01:49:00.672187 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:00.671962 2583 scope.go:117] "RemoveContainer" containerID="ada0bd3ef9b7c79f647310e7aa070ba4898aaae2eababeeafe84b8aa3b6a5dae" Jul 17 01:49:00.672435 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:49:00.672211 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89_odh-ai-gateway-infra(a24dde6b-91b2-41cb-ac18-4b7436cce94f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" podUID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" Jul 17 01:49:00.673472 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:00.673444 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" event={"ID":"8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9","Type":"ContainerStarted","Data":"672d473a161beb158edadca4281a0947cade939bfdd94d2ef0de3606c939719c"} Jul 17 01:49:00.673568 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:00.673484 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" event={"ID":"8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9","Type":"ContainerStarted","Data":"84c222ec1a37bcea3f49f400da56a90fccaa5e51b88d7fff69f563a07410822f"} Jul 17 01:49:00.701081 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:00.701028 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" podStartSLOduration=1.7010113100000002 podStartE2EDuration="1.70101131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:49:00.699169586 +0000 UTC m=+2098.094784437" watchObservedRunningTime="2026-07-17 01:49:00.70101131 +0000 UTC m=+2098.096626161" Jul 17 01:49:01.162839 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:01.162809 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:49:01.163025 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:49:01.162986 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:49:01.679003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:01.678970 2583 scope.go:117] "RemoveContainer" containerID="ada0bd3ef9b7c79f647310e7aa070ba4898aaae2eababeeafe84b8aa3b6a5dae" Jul 17 01:49:01.679429 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:49:01.679214 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89_odh-ai-gateway-infra(a24dde6b-91b2-41cb-ac18-4b7436cce94f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" podUID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" Jul 17 01:49:01.680224 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:01.680203 2583 generic.go:356] "Generic (PLEG): container finished" podID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" containerID="672d473a161beb158edadca4281a0947cade939bfdd94d2ef0de3606c939719c" exitCode=6 Jul 17 01:49:01.680322 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:01.680255 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" event={"ID":"8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9","Type":"ContainerDied","Data":"672d473a161beb158edadca4281a0947cade939bfdd94d2ef0de3606c939719c"} Jul 17 01:49:01.680484 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:01.680469 2583 scope.go:117] "RemoveContainer" containerID="672d473a161beb158edadca4281a0947cade939bfdd94d2ef0de3606c939719c" Jul 17 01:49:02.687877 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:02.687779 2583 generic.go:356] "Generic (PLEG): container finished" podID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" containerID="f4984fccd5b05bd1e2b12e783c4dcef67d02173e31e1528f717a90470412a137" exitCode=6 Jul 17 01:49:02.687877 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:02.687841 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" event={"ID":"8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9","Type":"ContainerDied","Data":"f4984fccd5b05bd1e2b12e783c4dcef67d02173e31e1528f717a90470412a137"} Jul 17 01:49:02.688433 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:02.687904 2583 scope.go:117] "RemoveContainer" containerID="672d473a161beb158edadca4281a0947cade939bfdd94d2ef0de3606c939719c" Jul 17 01:49:02.688433 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:02.688088 2583 scope.go:117] "RemoveContainer" containerID="f4984fccd5b05bd1e2b12e783c4dcef67d02173e31e1528f717a90470412a137" Jul 17 01:49:02.688433 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:49:02.688378 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c_odh-ai-gateway-infra(8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" podUID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" Jul 17 01:49:03.196432 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:03.196407 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:49:03.200477 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:03.200447 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:49:03.694402 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:03.694372 2583 scope.go:117] "RemoveContainer" containerID="f4984fccd5b05bd1e2b12e783c4dcef67d02173e31e1528f717a90470412a137" Jul 17 01:49:03.694826 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:49:03.694550 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c_odh-ai-gateway-infra(8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" podUID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" Jul 17 01:49:13.164357 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:13.164330 2583 scope.go:117] "RemoveContainer" containerID="ada0bd3ef9b7c79f647310e7aa070ba4898aaae2eababeeafe84b8aa3b6a5dae" Jul 17 01:49:13.742395 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:13.742358 2583 generic.go:356] "Generic (PLEG): container finished" podID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" containerID="35a945f895a9e0efb3a28e1670f5463c27f804db769d628beab6c6d12cca65ba" exitCode=6 Jul 17 01:49:13.742575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:13.742418 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" event={"ID":"a24dde6b-91b2-41cb-ac18-4b7436cce94f","Type":"ContainerDied","Data":"35a945f895a9e0efb3a28e1670f5463c27f804db769d628beab6c6d12cca65ba"} Jul 17 01:49:13.742575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:13.742498 2583 scope.go:117] "RemoveContainer" containerID="ada0bd3ef9b7c79f647310e7aa070ba4898aaae2eababeeafe84b8aa3b6a5dae" Jul 17 01:49:13.742869 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:13.742847 2583 scope.go:117] "RemoveContainer" containerID="35a945f895a9e0efb3a28e1670f5463c27f804db769d628beab6c6d12cca65ba" Jul 17 01:49:13.743092 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:49:13.743072 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89_odh-ai-gateway-infra(a24dde6b-91b2-41cb-ac18-4b7436cce94f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" podUID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" Jul 17 01:49:14.766563 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:14.766529 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89"] Jul 17 01:49:14.908922 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:14.908892 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" Jul 17 01:49:14.944472 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:14.944434 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24dde6b-91b2-41cb-ac18-4b7436cce94f-maas-api-service-ca\") pod \"a24dde6b-91b2-41cb-ac18-4b7436cce94f\" (UID: \"a24dde6b-91b2-41cb-ac18-4b7436cce94f\") " Jul 17 01:49:14.944747 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:14.944725 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24dde6b-91b2-41cb-ac18-4b7436cce94f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a24dde6b-91b2-41cb-ac18-4b7436cce94f" (UID: "a24dde6b-91b2-41cb-ac18-4b7436cce94f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:49:15.045737 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:15.045703 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24dde6b-91b2-41cb-ac18-4b7436cce94f-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:49:15.754472 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:15.754440 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" Jul 17 01:49:15.754472 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:15.754455 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89" event={"ID":"a24dde6b-91b2-41cb-ac18-4b7436cce94f","Type":"ContainerDied","Data":"f90ce36fe278fece337b4b981dc7c5807b9a848a44d0091a406e9c5704b991f4"} Jul 17 01:49:15.754717 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:15.754538 2583 scope.go:117] "RemoveContainer" containerID="35a945f895a9e0efb3a28e1670f5463c27f804db769d628beab6c6d12cca65ba" Jul 17 01:49:15.769418 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:15.769387 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89"] Jul 17 01:49:15.772411 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:15.772385 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-r9m89"] Jul 17 01:49:16.162591 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:16.162559 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:49:16.162785 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:49:16.162734 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:49:17.162634 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:17.162602 2583 scope.go:117] "RemoveContainer" containerID="f4984fccd5b05bd1e2b12e783c4dcef67d02173e31e1528f717a90470412a137" Jul 17 01:49:17.167911 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:17.167883 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" path="/var/lib/kubelet/pods/a24dde6b-91b2-41cb-ac18-4b7436cce94f/volumes" Jul 17 01:49:17.767236 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:17.767151 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" event={"ID":"8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9","Type":"ContainerStarted","Data":"dae5bd326ac07b692a597a0741943342c33094033b8668e1955df3546769d803"} Jul 17 01:49:18.773628 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:18.773591 2583 generic.go:356] "Generic (PLEG): container finished" podID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" containerID="dae5bd326ac07b692a597a0741943342c33094033b8668e1955df3546769d803" exitCode=6 Jul 17 01:49:18.774027 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:18.773673 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" event={"ID":"8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9","Type":"ContainerDied","Data":"dae5bd326ac07b692a597a0741943342c33094033b8668e1955df3546769d803"} Jul 17 01:49:18.774027 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:18.773771 2583 scope.go:117] "RemoveContainer" containerID="f4984fccd5b05bd1e2b12e783c4dcef67d02173e31e1528f717a90470412a137" Jul 17 01:49:18.774164 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:18.774078 2583 scope.go:117] "RemoveContainer" containerID="dae5bd326ac07b692a597a0741943342c33094033b8668e1955df3546769d803" Jul 17 01:49:18.774314 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:49:18.774293 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c_odh-ai-gateway-infra(8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" podUID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" Jul 17 01:49:18.793218 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:18.793192 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c"] Jul 17 01:49:19.920884 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:19.920855 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" Jul 17 01:49:19.992522 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:19.992482 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9-maas-api-service-ca\") pod \"8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9\" (UID: \"8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9\") " Jul 17 01:49:19.992847 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:19.992819 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" (UID: "8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:49:20.093462 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:20.093355 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:49:20.785419 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:20.785391 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" Jul 17 01:49:20.785419 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:20.785394 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c" event={"ID":"8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9","Type":"ContainerDied","Data":"84c222ec1a37bcea3f49f400da56a90fccaa5e51b88d7fff69f563a07410822f"} Jul 17 01:49:20.785670 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:20.785453 2583 scope.go:117] "RemoveContainer" containerID="dae5bd326ac07b692a597a0741943342c33094033b8668e1955df3546769d803" Jul 17 01:49:20.807910 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:20.807877 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c"] Jul 17 01:49:20.815265 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:20.815235 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-fcq4c"] Jul 17 01:49:21.168397 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:21.168314 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" path="/var/lib/kubelet/pods/8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9/volumes" Jul 17 01:49:22.233003 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:22.232964 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:49:29.162302 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:29.162272 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:49:29.162766 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:49:29.162461 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:49:44.163078 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:44.163048 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:49:44.163529 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:44.163356 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 01:49:44.895844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:44.895750 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" event={"ID":"3dae2011-0f79-44d3-9eec-8a1e57171dc0","Type":"ContainerStarted","Data":"ebd4c189cbd30f3d732b946df7de2a6ca3cbc4fb2f67b2a9cb7c11c98a9cfbfb"} Jul 17 01:49:44.895993 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:44.895960 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:49:45.901422 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:45.901381 2583 generic.go:356] "Generic (PLEG): container finished" podID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerID="ebd4c189cbd30f3d732b946df7de2a6ca3cbc4fb2f67b2a9cb7c11c98a9cfbfb" exitCode=1 Jul 17 01:49:45.901845 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:45.901479 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" event={"ID":"3dae2011-0f79-44d3-9eec-8a1e57171dc0","Type":"ContainerDied","Data":"ebd4c189cbd30f3d732b946df7de2a6ca3cbc4fb2f67b2a9cb7c11c98a9cfbfb"} Jul 17 01:49:45.901845 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:45.901534 2583 scope.go:117] "RemoveContainer" containerID="640559eb3042739c0447fd2760fbcc2c9674e4b14326516203190801cce11a3c" Jul 17 01:49:45.901845 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:45.901681 2583 scope.go:117] "RemoveContainer" containerID="ebd4c189cbd30f3d732b946df7de2a6ca3cbc4fb2f67b2a9cb7c11c98a9cfbfb" Jul 17 01:49:45.901976 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:49:45.901901 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:49:46.907269 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:46.907240 2583 scope.go:117] "RemoveContainer" containerID="ebd4c189cbd30f3d732b946df7de2a6ca3cbc4fb2f67b2a9cb7c11c98a9cfbfb" Jul 17 01:49:46.907705 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:49:46.907460 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:49:51.495939 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:51.495900 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:49:51.496336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:51.496307 2583 scope.go:117] "RemoveContainer" containerID="ebd4c189cbd30f3d732b946df7de2a6ca3cbc4fb2f67b2a9cb7c11c98a9cfbfb" Jul 17 01:49:51.496515 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:49:51.496492 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh_odh-ai-gateway-infra(3dae2011-0f79-44d3-9eec-8a1e57171dc0)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" Jul 17 01:49:55.769913 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.769800 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:49:55.775566 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775533 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf"] Jul 17 01:49:55.775853 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775841 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" containerName="revoke-keys" Jul 17 01:49:55.775895 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775855 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" containerName="revoke-keys" Jul 17 01:49:55.775895 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775867 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" containerName="revoke-keys" Jul 17 01:49:55.775895 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775873 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" containerName="revoke-keys" Jul 17 01:49:55.775895 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775881 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" containerName="revoke-keys" Jul 17 01:49:55.775895 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775886 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" containerName="revoke-keys" Jul 17 01:49:55.776054 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775901 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" containerName="revoke-keys" Jul 17 01:49:55.776054 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775906 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" containerName="revoke-keys" Jul 17 01:49:55.776054 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775948 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" containerName="revoke-keys" Jul 17 01:49:55.776054 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775956 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" containerName="revoke-keys" Jul 17 01:49:55.776054 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775964 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" containerName="revoke-keys" Jul 17 01:49:55.776054 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775971 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" containerName="revoke-keys" Jul 17 01:49:55.776054 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775976 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" containerName="revoke-keys" Jul 17 01:49:55.776054 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.775983 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" containerName="revoke-keys" Jul 17 01:49:55.780204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.780180 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" Jul 17 01:49:55.782625 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.782608 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:49:55.785664 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.785633 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf"] Jul 17 01:49:55.871896 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.871830 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:49:55.909146 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:55.909078 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53922e9b-80b1-4a33-9da2-36a259ce5002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf\" (UID: \"53922e9b-80b1-4a33-9da2-36a259ce5002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" Jul 17 01:49:56.010215 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:56.010176 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53922e9b-80b1-4a33-9da2-36a259ce5002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf\" (UID: \"53922e9b-80b1-4a33-9da2-36a259ce5002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" Jul 17 01:49:56.010755 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:56.010732 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53922e9b-80b1-4a33-9da2-36a259ce5002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf\" (UID: \"53922e9b-80b1-4a33-9da2-36a259ce5002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" Jul 17 01:49:56.091676 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:56.091637 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" Jul 17 01:49:56.221470 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:56.221344 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf"] Jul 17 01:49:56.223596 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:49:56.223568 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53922e9b_80b1_4a33_9da2_36a259ce5002.slice/crio-8c3574db264be5af5a879c1d275c87f29ae70aa8abb3a7ec128e3172ad5df3df WatchSource:0}: Error finding container 8c3574db264be5af5a879c1d275c87f29ae70aa8abb3a7ec128e3172ad5df3df: Status 404 returned error can't find the container with id 8c3574db264be5af5a879c1d275c87f29ae70aa8abb3a7ec128e3172ad5df3df Jul 17 01:49:56.952010 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:56.951974 2583 generic.go:356] "Generic (PLEG): container finished" podID="53922e9b-80b1-4a33-9da2-36a259ce5002" containerID="98df2558b8b868032481c4b2d8a0da3e7d7fe16d89de39a2a143f9d8d9742c71" exitCode=7 Jul 17 01:49:56.952466 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:56.952055 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" event={"ID":"53922e9b-80b1-4a33-9da2-36a259ce5002","Type":"ContainerDied","Data":"98df2558b8b868032481c4b2d8a0da3e7d7fe16d89de39a2a143f9d8d9742c71"} Jul 17 01:49:56.952466 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:56.952118 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" event={"ID":"53922e9b-80b1-4a33-9da2-36a259ce5002","Type":"ContainerStarted","Data":"8c3574db264be5af5a879c1d275c87f29ae70aa8abb3a7ec128e3172ad5df3df"} Jul 17 01:49:56.952466 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:56.952264 2583 scope.go:117] "RemoveContainer" containerID="98df2558b8b868032481c4b2d8a0da3e7d7fe16d89de39a2a143f9d8d9742c71" Jul 17 01:49:57.958591 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:57.958557 2583 generic.go:356] "Generic (PLEG): container finished" podID="53922e9b-80b1-4a33-9da2-36a259ce5002" containerID="2b147b9602f8b7eed4adad0584a1593b8dd43458d4938932775bd5093fce4eeb" exitCode=7 Jul 17 01:49:57.959042 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:57.958640 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" event={"ID":"53922e9b-80b1-4a33-9da2-36a259ce5002","Type":"ContainerDied","Data":"2b147b9602f8b7eed4adad0584a1593b8dd43458d4938932775bd5093fce4eeb"} Jul 17 01:49:57.959042 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:57.958699 2583 scope.go:117] "RemoveContainer" containerID="98df2558b8b868032481c4b2d8a0da3e7d7fe16d89de39a2a143f9d8d9742c71" Jul 17 01:49:57.959042 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:57.958804 2583 scope.go:117] "RemoveContainer" containerID="2b147b9602f8b7eed4adad0584a1593b8dd43458d4938932775bd5093fce4eeb" Jul 17 01:49:57.959042 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:49:57.959034 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf_odh-ai-gateway-infra(53922e9b-80b1-4a33-9da2-36a259ce5002)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" podUID="53922e9b-80b1-4a33-9da2-36a259ce5002" Jul 17 01:49:58.964438 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:49:58.964410 2583 scope.go:117] "RemoveContainer" containerID="2b147b9602f8b7eed4adad0584a1593b8dd43458d4938932775bd5093fce4eeb" Jul 17 01:49:58.964794 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:49:58.964619 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf_odh-ai-gateway-infra(53922e9b-80b1-4a33-9da2-36a259ce5002)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" podUID="53922e9b-80b1-4a33-9da2-36a259ce5002" Jul 17 01:50:01.252520 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.252482 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh"] Jul 17 01:50:01.277132 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.277079 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:50:01.386261 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.386237 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:50:01.457840 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.457806 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dae2011-0f79-44d3-9eec-8a1e57171dc0-openshift-service-ca\") pod \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\" (UID: \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\") " Jul 17 01:50:01.458030 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.457875 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dae2011-0f79-44d3-9eec-8a1e57171dc0-maas-api-tls\") pod \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\" (UID: \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\") " Jul 17 01:50:01.458030 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.457909 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrbxr\" (UniqueName: \"kubernetes.io/projected/3dae2011-0f79-44d3-9eec-8a1e57171dc0-kube-api-access-zrbxr\") pod \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\" (UID: \"3dae2011-0f79-44d3-9eec-8a1e57171dc0\") " Jul 17 01:50:01.458322 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.458291 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dae2011-0f79-44d3-9eec-8a1e57171dc0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3dae2011-0f79-44d3-9eec-8a1e57171dc0" (UID: "3dae2011-0f79-44d3-9eec-8a1e57171dc0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:50:01.460235 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.460211 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dae2011-0f79-44d3-9eec-8a1e57171dc0-kube-api-access-zrbxr" (OuterVolumeSpecName: "kube-api-access-zrbxr") pod "3dae2011-0f79-44d3-9eec-8a1e57171dc0" (UID: "3dae2011-0f79-44d3-9eec-8a1e57171dc0"). InnerVolumeSpecName "kube-api-access-zrbxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:50:01.460329 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.460252 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dae2011-0f79-44d3-9eec-8a1e57171dc0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3dae2011-0f79-44d3-9eec-8a1e57171dc0" (UID: "3dae2011-0f79-44d3-9eec-8a1e57171dc0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:50:01.558826 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.558796 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dae2011-0f79-44d3-9eec-8a1e57171dc0-openshift-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:50:01.558826 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.558822 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dae2011-0f79-44d3-9eec-8a1e57171dc0-maas-api-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:50:01.558826 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.558833 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrbxr\" (UniqueName: \"kubernetes.io/projected/3dae2011-0f79-44d3-9eec-8a1e57171dc0-kube-api-access-zrbxr\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:50:01.978235 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.978204 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" Jul 17 01:50:01.978449 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.978246 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh" event={"ID":"3dae2011-0f79-44d3-9eec-8a1e57171dc0","Type":"ContainerDied","Data":"539c6afc8603667313c880bd2851de0930033caa04832e57db529bb5adc016a8"} Jul 17 01:50:01.978449 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:01.978309 2583 scope.go:117] "RemoveContainer" containerID="ebd4c189cbd30f3d732b946df7de2a6ca3cbc4fb2f67b2a9cb7c11c98a9cfbfb" Jul 17 01:50:02.017766 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:02.017729 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh"] Jul 17 01:50:02.025230 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:02.025199 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ef2c65d-678b8d647c-gdvjh"] Jul 17 01:50:02.077253 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:02.077218 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:50:03.167844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:03.167811 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" path="/var/lib/kubelet/pods/3dae2011-0f79-44d3-9eec-8a1e57171dc0/volumes" Jul 17 01:50:07.394646 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:07.394613 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:50:07.495449 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:07.495408 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:50:07.657701 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:07.657629 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:50:08.926145 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:08.926110 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:50:09.025216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025177 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv"] Jul 17 01:50:09.025500 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025488 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.025546 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025501 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.025546 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025510 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.025546 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025517 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.025546 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025524 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.025546 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025530 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.025546 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025542 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.025546 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025548 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.025750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025555 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.025750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025560 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.025750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025566 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" containerName="revoke-keys" Jul 17 01:50:09.025750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025571 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24dde6b-91b2-41cb-ac18-4b7436cce94f" containerName="revoke-keys" Jul 17 01:50:09.025750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025586 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" containerName="revoke-keys" Jul 17 01:50:09.025750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025591 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff901fb-6a11-41f9-af8e-f8b0b8e79cf9" containerName="revoke-keys" Jul 17 01:50:09.025750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025635 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.025750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025642 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.025750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025648 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.025750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025653 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.025750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.025661 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:09.028535 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.028510 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:09.031306 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.031276 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1e02ab63-serving-cert\"" Jul 17 01:50:09.032269 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.032248 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5l8p8\"" Jul 17 01:50:09.032407 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.032272 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 01:50:09.044699 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.044667 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv"] Jul 17 01:50:09.061561 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.061533 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:50:09.125543 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.125501 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244c0c1d-f71c-4bf3-80d5-e655fa015bda-openshift-service-ca\") pod \"maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv\" (UID: \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:09.125748 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.125559 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244c0c1d-f71c-4bf3-80d5-e655fa015bda-maas-api-tls\") pod \"maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv\" (UID: \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:09.125748 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.125585 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps8f9\" (UniqueName: \"kubernetes.io/projected/244c0c1d-f71c-4bf3-80d5-e655fa015bda-kube-api-access-ps8f9\") pod \"maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv\" (UID: \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:09.161234 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.161201 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:50:09.226374 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.226282 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps8f9\" (UniqueName: \"kubernetes.io/projected/244c0c1d-f71c-4bf3-80d5-e655fa015bda-kube-api-access-ps8f9\") pod \"maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv\" (UID: \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:09.226374 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.226369 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244c0c1d-f71c-4bf3-80d5-e655fa015bda-openshift-service-ca\") pod \"maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv\" (UID: \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:09.226618 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.226405 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244c0c1d-f71c-4bf3-80d5-e655fa015bda-maas-api-tls\") pod \"maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv\" (UID: \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:09.226618 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:09.226494 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-serving-cert: secret "maas-api-e2e-derive-1e02ab63-serving-cert" not found Jul 17 01:50:09.226618 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:09.226564 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244c0c1d-f71c-4bf3-80d5-e655fa015bda-maas-api-tls podName:244c0c1d-f71c-4bf3-80d5-e655fa015bda nodeName:}" failed. No retries permitted until 2026-07-17 01:50:09.726545755 +0000 UTC m=+2167.122160586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/244c0c1d-f71c-4bf3-80d5-e655fa015bda-maas-api-tls") pod "maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" (UID: "244c0c1d-f71c-4bf3-80d5-e655fa015bda") : secret "maas-api-e2e-derive-1e02ab63-serving-cert" not found Jul 17 01:50:09.227146 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.227082 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244c0c1d-f71c-4bf3-80d5-e655fa015bda-openshift-service-ca\") pod \"maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv\" (UID: \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:09.234768 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.234745 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps8f9\" (UniqueName: \"kubernetes.io/projected/244c0c1d-f71c-4bf3-80d5-e655fa015bda-kube-api-access-ps8f9\") pod \"maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv\" (UID: \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:09.286663 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.286632 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:50:09.559709 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.559674 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:50:09.660964 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.660928 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:50:09.730970 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.730921 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244c0c1d-f71c-4bf3-80d5-e655fa015bda-maas-api-tls\") pod \"maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv\" (UID: \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:09.733646 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.733620 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244c0c1d-f71c-4bf3-80d5-e655fa015bda-maas-api-tls\") pod \"maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv\" (UID: \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:09.941816 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:09.941714 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:10.074504 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:10.074436 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv"] Jul 17 01:50:10.077026 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:50:10.076997 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244c0c1d_f71c_4bf3_80d5_e655fa015bda.slice/crio-3882646e201822debc0c4c597d17ea78b707f0433d40f625d1c2875f154e078d WatchSource:0}: Error finding container 3882646e201822debc0c4c597d17ea78b707f0433d40f625d1c2875f154e078d: Status 404 returned error can't find the container with id 3882646e201822debc0c4c597d17ea78b707f0433d40f625d1c2875f154e078d Jul 17 01:50:11.019074 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:11.018991 2583 generic.go:356] "Generic (PLEG): container finished" podID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerID="a23382157ef01d07eb683b6bf2f1cb0f0ce539deab6e6d3ebaf9c5bb5f0eee0d" exitCode=1 Jul 17 01:50:11.019458 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:11.019081 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" event={"ID":"244c0c1d-f71c-4bf3-80d5-e655fa015bda","Type":"ContainerDied","Data":"a23382157ef01d07eb683b6bf2f1cb0f0ce539deab6e6d3ebaf9c5bb5f0eee0d"} Jul 17 01:50:11.019458 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:11.019166 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" event={"ID":"244c0c1d-f71c-4bf3-80d5-e655fa015bda","Type":"ContainerStarted","Data":"3882646e201822debc0c4c597d17ea78b707f0433d40f625d1c2875f154e078d"} Jul 17 01:50:11.019458 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:11.019275 2583 scope.go:117] "RemoveContainer" containerID="a23382157ef01d07eb683b6bf2f1cb0f0ce539deab6e6d3ebaf9c5bb5f0eee0d" Jul 17 01:50:12.031027 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:12.030993 2583 generic.go:356] "Generic (PLEG): container finished" podID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerID="052bbedcdfd6d4c6aaf388b467b5974be859b195320d3a16e127d46196bed39e" exitCode=1 Jul 17 01:50:12.031438 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:12.031041 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" event={"ID":"244c0c1d-f71c-4bf3-80d5-e655fa015bda","Type":"ContainerDied","Data":"052bbedcdfd6d4c6aaf388b467b5974be859b195320d3a16e127d46196bed39e"} Jul 17 01:50:12.031438 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:12.031152 2583 scope.go:117] "RemoveContainer" containerID="a23382157ef01d07eb683b6bf2f1cb0f0ce539deab6e6d3ebaf9c5bb5f0eee0d" Jul 17 01:50:12.031438 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:12.031369 2583 scope.go:117] "RemoveContainer" containerID="052bbedcdfd6d4c6aaf388b467b5974be859b195320d3a16e127d46196bed39e" Jul 17 01:50:12.031583 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:12.031552 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:50:13.037119 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:13.037082 2583 scope.go:117] "RemoveContainer" containerID="052bbedcdfd6d4c6aaf388b467b5974be859b195320d3a16e127d46196bed39e" Jul 17 01:50:13.037578 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:13.037317 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:50:13.205219 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:13.205185 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8"] Jul 17 01:50:13.205524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:13.205513 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:13.205562 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:13.205526 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:13.205616 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:13.205597 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:13.205616 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:13.205607 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:13.209725 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:13.209708 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" Jul 17 01:50:13.214893 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:13.214849 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8"] Jul 17 01:50:13.259930 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:13.259884 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46015628-8c9a-413e-aeb9-46872ffdf0ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-727b8\" (UID: \"46015628-8c9a-413e-aeb9-46872ffdf0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" Jul 17 01:50:13.360488 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:13.360402 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46015628-8c9a-413e-aeb9-46872ffdf0ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-727b8\" (UID: \"46015628-8c9a-413e-aeb9-46872ffdf0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" Jul 17 01:50:13.361008 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:13.360989 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46015628-8c9a-413e-aeb9-46872ffdf0ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-727b8\" (UID: \"46015628-8c9a-413e-aeb9-46872ffdf0ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" Jul 17 01:50:13.521247 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:13.521202 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" Jul 17 01:50:13.654978 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:13.654783 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8"] Jul 17 01:50:13.657591 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:50:13.657558 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46015628_8c9a_413e_aeb9_46872ffdf0ec.slice/crio-772ad279c49b1b5bd025d90dd88d77c72966293698e2218bb6af9614f5f6667c WatchSource:0}: Error finding container 772ad279c49b1b5bd025d90dd88d77c72966293698e2218bb6af9614f5f6667c: Status 404 returned error can't find the container with id 772ad279c49b1b5bd025d90dd88d77c72966293698e2218bb6af9614f5f6667c Jul 17 01:50:14.043211 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:14.043171 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" event={"ID":"46015628-8c9a-413e-aeb9-46872ffdf0ec","Type":"ContainerStarted","Data":"34c7cd74170c13f1ef5a91284ce53efb24fae889270c30d122e5cb47bc755699"} Jul 17 01:50:14.043572 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:14.043237 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" event={"ID":"46015628-8c9a-413e-aeb9-46872ffdf0ec","Type":"ContainerStarted","Data":"772ad279c49b1b5bd025d90dd88d77c72966293698e2218bb6af9614f5f6667c"} Jul 17 01:50:14.057897 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:14.057842 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" podStartSLOduration=1.057826694 podStartE2EDuration="1.057826694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:50:14.056254681 +0000 UTC m=+2171.451869524" watchObservedRunningTime="2026-07-17 01:50:14.057826694 +0000 UTC m=+2171.453441584" Jul 17 01:50:14.162185 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:14.162159 2583 scope.go:117] "RemoveContainer" containerID="2b147b9602f8b7eed4adad0584a1593b8dd43458d4938932775bd5093fce4eeb" Jul 17 01:50:14.942930 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:14.942834 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:14.943287 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:14.943274 2583 scope.go:117] "RemoveContainer" containerID="052bbedcdfd6d4c6aaf388b467b5974be859b195320d3a16e127d46196bed39e" Jul 17 01:50:14.943459 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:14.943441 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:50:15.049725 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:15.049690 2583 generic.go:356] "Generic (PLEG): container finished" podID="46015628-8c9a-413e-aeb9-46872ffdf0ec" containerID="34c7cd74170c13f1ef5a91284ce53efb24fae889270c30d122e5cb47bc755699" exitCode=7 Jul 17 01:50:15.050175 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:15.049777 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" event={"ID":"46015628-8c9a-413e-aeb9-46872ffdf0ec","Type":"ContainerDied","Data":"34c7cd74170c13f1ef5a91284ce53efb24fae889270c30d122e5cb47bc755699"} Jul 17 01:50:15.050175 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:15.049984 2583 scope.go:117] "RemoveContainer" containerID="34c7cd74170c13f1ef5a91284ce53efb24fae889270c30d122e5cb47bc755699" Jul 17 01:50:15.051732 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:15.051711 2583 generic.go:356] "Generic (PLEG): container finished" podID="53922e9b-80b1-4a33-9da2-36a259ce5002" containerID="b79aeace58e55c3c5777d9a633ceac16e25a4642a236597cca18139671f640d5" exitCode=6 Jul 17 01:50:15.051832 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:15.051744 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" event={"ID":"53922e9b-80b1-4a33-9da2-36a259ce5002","Type":"ContainerDied","Data":"b79aeace58e55c3c5777d9a633ceac16e25a4642a236597cca18139671f640d5"} Jul 17 01:50:15.051832 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:15.051781 2583 scope.go:117] "RemoveContainer" containerID="2b147b9602f8b7eed4adad0584a1593b8dd43458d4938932775bd5093fce4eeb" Jul 17 01:50:15.052156 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:15.052139 2583 scope.go:117] "RemoveContainer" containerID="b79aeace58e55c3c5777d9a633ceac16e25a4642a236597cca18139671f640d5" Jul 17 01:50:15.052386 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:15.052342 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf_odh-ai-gateway-infra(53922e9b-80b1-4a33-9da2-36a259ce5002)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" podUID="53922e9b-80b1-4a33-9da2-36a259ce5002" Jul 17 01:50:16.059634 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:16.059601 2583 generic.go:356] "Generic (PLEG): container finished" podID="46015628-8c9a-413e-aeb9-46872ffdf0ec" containerID="1f18a431146e4913b7c806f6574adbb83797927aa4f002fd4fd37c8c4303051c" exitCode=7 Jul 17 01:50:16.060080 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:16.059661 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" event={"ID":"46015628-8c9a-413e-aeb9-46872ffdf0ec","Type":"ContainerDied","Data":"1f18a431146e4913b7c806f6574adbb83797927aa4f002fd4fd37c8c4303051c"} Jul 17 01:50:16.060080 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:16.059708 2583 scope.go:117] "RemoveContainer" containerID="34c7cd74170c13f1ef5a91284ce53efb24fae889270c30d122e5cb47bc755699" Jul 17 01:50:16.060080 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:16.059891 2583 scope.go:117] "RemoveContainer" containerID="1f18a431146e4913b7c806f6574adbb83797927aa4f002fd4fd37c8c4303051c" Jul 17 01:50:16.060240 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:16.060131 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-727b8_odh-ai-gateway-infra(46015628-8c9a-413e-aeb9-46872ffdf0ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" podUID="46015628-8c9a-413e-aeb9-46872ffdf0ec" Jul 17 01:50:16.086394 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:16.086363 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf"] Jul 17 01:50:16.221672 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:16.221648 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" Jul 17 01:50:16.283039 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:16.283005 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53922e9b-80b1-4a33-9da2-36a259ce5002-maas-api-service-ca\") pod \"53922e9b-80b1-4a33-9da2-36a259ce5002\" (UID: \"53922e9b-80b1-4a33-9da2-36a259ce5002\") " Jul 17 01:50:16.283426 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:16.283400 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53922e9b-80b1-4a33-9da2-36a259ce5002-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53922e9b-80b1-4a33-9da2-36a259ce5002" (UID: "53922e9b-80b1-4a33-9da2-36a259ce5002"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:50:16.383906 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:16.383823 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53922e9b-80b1-4a33-9da2-36a259ce5002-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:50:17.065170 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:17.065139 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" Jul 17 01:50:17.065170 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:17.065139 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf" event={"ID":"53922e9b-80b1-4a33-9da2-36a259ce5002","Type":"ContainerDied","Data":"8c3574db264be5af5a879c1d275c87f29ae70aa8abb3a7ec128e3172ad5df3df"} Jul 17 01:50:17.065676 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:17.065234 2583 scope.go:117] "RemoveContainer" containerID="b79aeace58e55c3c5777d9a633ceac16e25a4642a236597cca18139671f640d5" Jul 17 01:50:17.066895 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:17.066879 2583 scope.go:117] "RemoveContainer" containerID="1f18a431146e4913b7c806f6574adbb83797927aa4f002fd4fd37c8c4303051c" Jul 17 01:50:17.067298 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:17.067250 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-727b8_odh-ai-gateway-infra(46015628-8c9a-413e-aeb9-46872ffdf0ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" podUID="46015628-8c9a-413e-aeb9-46872ffdf0ec" Jul 17 01:50:17.096845 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:17.096811 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf"] Jul 17 01:50:17.101714 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:17.101687 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vs9mf"] Jul 17 01:50:17.168319 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:17.168286 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53922e9b-80b1-4a33-9da2-36a259ce5002" path="/var/lib/kubelet/pods/53922e9b-80b1-4a33-9da2-36a259ce5002/volumes" Jul 17 01:50:19.942274 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:19.942238 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:19.942675 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:19.942627 2583 scope.go:117] "RemoveContainer" containerID="052bbedcdfd6d4c6aaf388b467b5974be859b195320d3a16e127d46196bed39e" Jul 17 01:50:19.942813 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:19.942784 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:50:22.445694 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:22.445659 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:50:22.496011 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:22.495974 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:50:26.252606 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:26.252574 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:50:28.163118 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:28.163063 2583 scope.go:117] "RemoveContainer" containerID="1f18a431146e4913b7c806f6574adbb83797927aa4f002fd4fd37c8c4303051c" Jul 17 01:50:29.132071 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:29.132040 2583 generic.go:356] "Generic (PLEG): container finished" podID="46015628-8c9a-413e-aeb9-46872ffdf0ec" containerID="7039675bfffd329575fb2b6990b498d18c7d192883fd7bd05458d0c8e251fd92" exitCode=7 Jul 17 01:50:29.132288 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:29.132119 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" event={"ID":"46015628-8c9a-413e-aeb9-46872ffdf0ec","Type":"ContainerDied","Data":"7039675bfffd329575fb2b6990b498d18c7d192883fd7bd05458d0c8e251fd92"} Jul 17 01:50:29.132288 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:29.132192 2583 scope.go:117] "RemoveContainer" containerID="1f18a431146e4913b7c806f6574adbb83797927aa4f002fd4fd37c8c4303051c" Jul 17 01:50:29.132521 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:29.132503 2583 scope.go:117] "RemoveContainer" containerID="7039675bfffd329575fb2b6990b498d18c7d192883fd7bd05458d0c8e251fd92" Jul 17 01:50:29.132733 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:29.132709 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-727b8_odh-ai-gateway-infra(46015628-8c9a-413e-aeb9-46872ffdf0ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" podUID="46015628-8c9a-413e-aeb9-46872ffdf0ec" Jul 17 01:50:30.156464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:30.156432 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8"] Jul 17 01:50:30.292164 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:30.292139 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" Jul 17 01:50:30.415299 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:30.415213 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46015628-8c9a-413e-aeb9-46872ffdf0ec-maas-api-service-ca\") pod \"46015628-8c9a-413e-aeb9-46872ffdf0ec\" (UID: \"46015628-8c9a-413e-aeb9-46872ffdf0ec\") " Jul 17 01:50:30.415565 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:30.415542 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46015628-8c9a-413e-aeb9-46872ffdf0ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "46015628-8c9a-413e-aeb9-46872ffdf0ec" (UID: "46015628-8c9a-413e-aeb9-46872ffdf0ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:50:30.515976 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:30.515936 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46015628-8c9a-413e-aeb9-46872ffdf0ec-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:50:31.145034 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:31.144995 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" event={"ID":"46015628-8c9a-413e-aeb9-46872ffdf0ec","Type":"ContainerDied","Data":"772ad279c49b1b5bd025d90dd88d77c72966293698e2218bb6af9614f5f6667c"} Jul 17 01:50:31.145034 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:31.145013 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8" Jul 17 01:50:31.145303 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:31.145055 2583 scope.go:117] "RemoveContainer" containerID="7039675bfffd329575fb2b6990b498d18c7d192883fd7bd05458d0c8e251fd92" Jul 17 01:50:31.167566 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:31.167539 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8"] Jul 17 01:50:31.170820 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:31.170789 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-727b8"] Jul 17 01:50:33.167504 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:33.167469 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46015628-8c9a-413e-aeb9-46872ffdf0ec" path="/var/lib/kubelet/pods/46015628-8c9a-413e-aeb9-46872ffdf0ec/volumes" Jul 17 01:50:34.162213 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:34.162183 2583 scope.go:117] "RemoveContainer" containerID="052bbedcdfd6d4c6aaf388b467b5974be859b195320d3a16e127d46196bed39e" Jul 17 01:50:35.170328 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:35.170290 2583 generic.go:356] "Generic (PLEG): container finished" podID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerID="f2ebfd8bf06f5a44304bb1226e652e2f3914db10429f79d983ef82dd487e6f69" exitCode=1 Jul 17 01:50:35.172858 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:35.172825 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" event={"ID":"244c0c1d-f71c-4bf3-80d5-e655fa015bda","Type":"ContainerDied","Data":"f2ebfd8bf06f5a44304bb1226e652e2f3914db10429f79d983ef82dd487e6f69"} Jul 17 01:50:35.172998 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:35.172898 2583 scope.go:117] "RemoveContainer" containerID="052bbedcdfd6d4c6aaf388b467b5974be859b195320d3a16e127d46196bed39e" Jul 17 01:50:35.173295 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:35.173267 2583 scope.go:117] "RemoveContainer" containerID="f2ebfd8bf06f5a44304bb1226e652e2f3914db10429f79d983ef82dd487e6f69" Jul 17 01:50:35.173457 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:35.173432 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:50:39.942087 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:39.942045 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:39.942499 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:39.942122 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:50:39.942499 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:39.942472 2583 scope.go:117] "RemoveContainer" containerID="f2ebfd8bf06f5a44304bb1226e652e2f3914db10429f79d983ef82dd487e6f69" Jul 17 01:50:39.942678 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:39.942656 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:50:49.734621 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.734583 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm"] Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.734927 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53922e9b-80b1-4a33-9da2-36a259ce5002" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.734937 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="53922e9b-80b1-4a33-9da2-36a259ce5002" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.734949 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53922e9b-80b1-4a33-9da2-36a259ce5002" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.734955 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="53922e9b-80b1-4a33-9da2-36a259ce5002" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.734967 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53922e9b-80b1-4a33-9da2-36a259ce5002" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.734972 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="53922e9b-80b1-4a33-9da2-36a259ce5002" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.734979 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.734984 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dae2011-0f79-44d3-9eec-8a1e57171dc0" containerName="maas-api" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.734991 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46015628-8c9a-413e-aeb9-46872ffdf0ec" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.734996 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="46015628-8c9a-413e-aeb9-46872ffdf0ec" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.735006 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46015628-8c9a-413e-aeb9-46872ffdf0ec" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.735011 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="46015628-8c9a-413e-aeb9-46872ffdf0ec" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.735016 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46015628-8c9a-413e-aeb9-46872ffdf0ec" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.735021 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="46015628-8c9a-413e-aeb9-46872ffdf0ec" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.735070 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="53922e9b-80b1-4a33-9da2-36a259ce5002" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.735076 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="46015628-8c9a-413e-aeb9-46872ffdf0ec" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.735082 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="46015628-8c9a-413e-aeb9-46872ffdf0ec" containerName="revoke-keys" Jul 17 01:50:49.735084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.735088 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="46015628-8c9a-413e-aeb9-46872ffdf0ec" containerName="revoke-keys" Jul 17 01:50:49.735702 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.735113 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="53922e9b-80b1-4a33-9da2-36a259ce5002" containerName="revoke-keys" Jul 17 01:50:49.738247 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.738226 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" Jul 17 01:50:49.740742 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.740722 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:50:49.744513 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.744489 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm"] Jul 17 01:50:49.798634 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.798596 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a57bb-4c8c-419a-b114-ac91f3714dd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm\" (UID: \"ae6a57bb-4c8c-419a-b114-ac91f3714dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" Jul 17 01:50:49.899985 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.899947 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a57bb-4c8c-419a-b114-ac91f3714dd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm\" (UID: \"ae6a57bb-4c8c-419a-b114-ac91f3714dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" Jul 17 01:50:49.900584 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:49.900562 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a57bb-4c8c-419a-b114-ac91f3714dd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm\" (UID: \"ae6a57bb-4c8c-419a-b114-ac91f3714dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" Jul 17 01:50:50.050213 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:50.050168 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" Jul 17 01:50:50.194367 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:50.182157 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm"] Jul 17 01:50:50.194367 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:50:50.184593 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6a57bb_4c8c_419a_b114_ac91f3714dd2.slice/crio-ccfcd47a429417cf2a9c1770ff4971656e587dd6de15fe4f84e42b27446e99cc WatchSource:0}: Error finding container ccfcd47a429417cf2a9c1770ff4971656e587dd6de15fe4f84e42b27446e99cc: Status 404 returned error can't find the container with id ccfcd47a429417cf2a9c1770ff4971656e587dd6de15fe4f84e42b27446e99cc Jul 17 01:50:50.240022 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:50.239985 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" event={"ID":"ae6a57bb-4c8c-419a-b114-ac91f3714dd2","Type":"ContainerStarted","Data":"ccfcd47a429417cf2a9c1770ff4971656e587dd6de15fe4f84e42b27446e99cc"} Jul 17 01:50:51.162561 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:51.162470 2583 scope.go:117] "RemoveContainer" containerID="f2ebfd8bf06f5a44304bb1226e652e2f3914db10429f79d983ef82dd487e6f69" Jul 17 01:50:51.162981 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:51.162642 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:50:51.245950 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:51.245915 2583 generic.go:356] "Generic (PLEG): container finished" podID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" containerID="daf64ca32a48e8a686a004f379abd27dea4d9cbb7369cf5ece845cf17677d762" exitCode=6 Jul 17 01:50:51.246133 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:51.246010 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" event={"ID":"ae6a57bb-4c8c-419a-b114-ac91f3714dd2","Type":"ContainerDied","Data":"daf64ca32a48e8a686a004f379abd27dea4d9cbb7369cf5ece845cf17677d762"} Jul 17 01:50:51.246229 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:51.246216 2583 scope.go:117] "RemoveContainer" containerID="daf64ca32a48e8a686a004f379abd27dea4d9cbb7369cf5ece845cf17677d762" Jul 17 01:50:52.251989 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:52.251891 2583 generic.go:356] "Generic (PLEG): container finished" podID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" containerID="0ddda9d154542e4137e68e61a858cdc43be139e2261646b6e8a7419f93ed9117" exitCode=6 Jul 17 01:50:52.252442 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:52.251983 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" event={"ID":"ae6a57bb-4c8c-419a-b114-ac91f3714dd2","Type":"ContainerDied","Data":"0ddda9d154542e4137e68e61a858cdc43be139e2261646b6e8a7419f93ed9117"} Jul 17 01:50:52.252442 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:52.252044 2583 scope.go:117] "RemoveContainer" containerID="daf64ca32a48e8a686a004f379abd27dea4d9cbb7369cf5ece845cf17677d762" Jul 17 01:50:52.252442 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:52.252259 2583 scope.go:117] "RemoveContainer" containerID="0ddda9d154542e4137e68e61a858cdc43be139e2261646b6e8a7419f93ed9117" Jul 17 01:50:52.252564 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:52.252485 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm_odh-ai-gateway-infra(ae6a57bb-4c8c-419a-b114-ac91f3714dd2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" podUID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" Jul 17 01:50:53.258512 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:50:53.258484 2583 scope.go:117] "RemoveContainer" containerID="0ddda9d154542e4137e68e61a858cdc43be139e2261646b6e8a7419f93ed9117" Jul 17 01:50:53.258923 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:50:53.258674 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm_odh-ai-gateway-infra(ae6a57bb-4c8c-419a-b114-ac91f3714dd2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" podUID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" Jul 17 01:51:03.166271 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:03.166243 2583 scope.go:117] "RemoveContainer" containerID="f2ebfd8bf06f5a44304bb1226e652e2f3914db10429f79d983ef82dd487e6f69" Jul 17 01:51:03.244292 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:03.244256 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz"] Jul 17 01:51:03.244637 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:03.244625 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="53922e9b-80b1-4a33-9da2-36a259ce5002" containerName="revoke-keys" Jul 17 01:51:03.248753 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:03.248736 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" Jul 17 01:51:03.253887 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:03.253859 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz"] Jul 17 01:51:03.318067 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:03.318038 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f490c66f-d27a-4745-bcbf-26e79ec593c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz\" (UID: \"f490c66f-d27a-4745-bcbf-26e79ec593c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" Jul 17 01:51:03.418950 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:03.418860 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f490c66f-d27a-4745-bcbf-26e79ec593c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz\" (UID: \"f490c66f-d27a-4745-bcbf-26e79ec593c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" Jul 17 01:51:03.419539 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:03.419513 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f490c66f-d27a-4745-bcbf-26e79ec593c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz\" (UID: \"f490c66f-d27a-4745-bcbf-26e79ec593c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" Jul 17 01:51:03.560124 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:03.560080 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" Jul 17 01:51:03.699553 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:03.699531 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz"] Jul 17 01:51:03.701674 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:51:03.701651 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf490c66f_d27a_4745_bcbf_26e79ec593c5.slice/crio-c688d3a8b69b0e742dbacae16189138b03ffa4fec639d711eaaff4911840cd81 WatchSource:0}: Error finding container c688d3a8b69b0e742dbacae16189138b03ffa4fec639d711eaaff4911840cd81: Status 404 returned error can't find the container with id c688d3a8b69b0e742dbacae16189138b03ffa4fec639d711eaaff4911840cd81 Jul 17 01:51:04.163085 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:04.163059 2583 scope.go:117] "RemoveContainer" containerID="0ddda9d154542e4137e68e61a858cdc43be139e2261646b6e8a7419f93ed9117" Jul 17 01:51:04.308517 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:04.308484 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" event={"ID":"f490c66f-d27a-4745-bcbf-26e79ec593c5","Type":"ContainerStarted","Data":"fcfd7a5b6acbb108709e03da23ace0c16a820c36135dfefe3867c9b787c3d5e1"} Jul 17 01:51:04.308959 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:04.308538 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" event={"ID":"f490c66f-d27a-4745-bcbf-26e79ec593c5","Type":"ContainerStarted","Data":"c688d3a8b69b0e742dbacae16189138b03ffa4fec639d711eaaff4911840cd81"} Jul 17 01:51:04.310333 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:04.310309 2583 generic.go:356] "Generic (PLEG): container finished" podID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerID="3e763092a88a90ee1f5bc059049d7f588edbb4b73f5084049924232231adbd57" exitCode=1 Jul 17 01:51:04.310448 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:04.310349 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" event={"ID":"244c0c1d-f71c-4bf3-80d5-e655fa015bda","Type":"ContainerDied","Data":"3e763092a88a90ee1f5bc059049d7f588edbb4b73f5084049924232231adbd57"} Jul 17 01:51:04.310448 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:04.310389 2583 scope.go:117] "RemoveContainer" containerID="f2ebfd8bf06f5a44304bb1226e652e2f3914db10429f79d983ef82dd487e6f69" Jul 17 01:51:04.310676 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:04.310651 2583 scope.go:117] "RemoveContainer" containerID="3e763092a88a90ee1f5bc059049d7f588edbb4b73f5084049924232231adbd57" Jul 17 01:51:04.310876 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:04.310857 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:51:04.324403 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:04.324359 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" podStartSLOduration=1.3243206490000001 podStartE2EDuration="1.324320649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:51:04.32251958 +0000 UTC m=+2221.718134430" watchObservedRunningTime="2026-07-17 01:51:04.324320649 +0000 UTC m=+2221.719935501" Jul 17 01:51:04.942529 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:04.942497 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:51:05.316734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:05.316702 2583 generic.go:356] "Generic (PLEG): container finished" podID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" containerID="b2a999e8fbe5bd9634c9b8df965b8872acf01a01ee896356ef22c3b341fb030c" exitCode=6 Jul 17 01:51:05.317200 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:05.316780 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" event={"ID":"ae6a57bb-4c8c-419a-b114-ac91f3714dd2","Type":"ContainerDied","Data":"b2a999e8fbe5bd9634c9b8df965b8872acf01a01ee896356ef22c3b341fb030c"} Jul 17 01:51:05.317200 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:05.316847 2583 scope.go:117] "RemoveContainer" containerID="0ddda9d154542e4137e68e61a858cdc43be139e2261646b6e8a7419f93ed9117" Jul 17 01:51:05.317200 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:05.317175 2583 scope.go:117] "RemoveContainer" containerID="b2a999e8fbe5bd9634c9b8df965b8872acf01a01ee896356ef22c3b341fb030c" Jul 17 01:51:05.317418 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:05.317398 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm_odh-ai-gateway-infra(ae6a57bb-4c8c-419a-b114-ac91f3714dd2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" podUID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" Jul 17 01:51:05.318295 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:05.318271 2583 generic.go:356] "Generic (PLEG): container finished" podID="f490c66f-d27a-4745-bcbf-26e79ec593c5" containerID="fcfd7a5b6acbb108709e03da23ace0c16a820c36135dfefe3867c9b787c3d5e1" exitCode=7 Jul 17 01:51:05.318399 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:05.318321 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" event={"ID":"f490c66f-d27a-4745-bcbf-26e79ec593c5","Type":"ContainerDied","Data":"fcfd7a5b6acbb108709e03da23ace0c16a820c36135dfefe3867c9b787c3d5e1"} Jul 17 01:51:05.318505 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:05.318491 2583 scope.go:117] "RemoveContainer" containerID="fcfd7a5b6acbb108709e03da23ace0c16a820c36135dfefe3867c9b787c3d5e1" Jul 17 01:51:05.320138 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:05.320119 2583 scope.go:117] "RemoveContainer" containerID="3e763092a88a90ee1f5bc059049d7f588edbb4b73f5084049924232231adbd57" Jul 17 01:51:05.320301 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:05.320281 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:51:06.328211 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:06.328173 2583 generic.go:356] "Generic (PLEG): container finished" podID="f490c66f-d27a-4745-bcbf-26e79ec593c5" containerID="62e860c4c2861017bae0782cbeeeb01b6d78456a1397e3a01abde3f6f3be35f4" exitCode=7 Jul 17 01:51:06.328211 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:06.328210 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" event={"ID":"f490c66f-d27a-4745-bcbf-26e79ec593c5","Type":"ContainerDied","Data":"62e860c4c2861017bae0782cbeeeb01b6d78456a1397e3a01abde3f6f3be35f4"} Jul 17 01:51:06.328654 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:06.328272 2583 scope.go:117] "RemoveContainer" containerID="fcfd7a5b6acbb108709e03da23ace0c16a820c36135dfefe3867c9b787c3d5e1" Jul 17 01:51:06.328654 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:06.328476 2583 scope.go:117] "RemoveContainer" containerID="62e860c4c2861017bae0782cbeeeb01b6d78456a1397e3a01abde3f6f3be35f4" Jul 17 01:51:06.328731 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:06.328681 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz_odh-ai-gateway-infra(f490c66f-d27a-4745-bcbf-26e79ec593c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" podUID="f490c66f-d27a-4745-bcbf-26e79ec593c5" Jul 17 01:51:06.341542 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:06.341517 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm"] Jul 17 01:51:06.473798 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:06.473776 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" Jul 17 01:51:06.543484 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:06.543443 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a57bb-4c8c-419a-b114-ac91f3714dd2-maas-api-service-ca\") pod \"ae6a57bb-4c8c-419a-b114-ac91f3714dd2\" (UID: \"ae6a57bb-4c8c-419a-b114-ac91f3714dd2\") " Jul 17 01:51:06.543777 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:06.543755 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6a57bb-4c8c-419a-b114-ac91f3714dd2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae6a57bb-4c8c-419a-b114-ac91f3714dd2" (UID: "ae6a57bb-4c8c-419a-b114-ac91f3714dd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:51:06.644750 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:06.644662 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6a57bb-4c8c-419a-b114-ac91f3714dd2-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:51:07.335000 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:07.334978 2583 scope.go:117] "RemoveContainer" containerID="62e860c4c2861017bae0782cbeeeb01b6d78456a1397e3a01abde3f6f3be35f4" Jul 17 01:51:07.335430 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:07.335209 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz_odh-ai-gateway-infra(f490c66f-d27a-4745-bcbf-26e79ec593c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" podUID="f490c66f-d27a-4745-bcbf-26e79ec593c5" Jul 17 01:51:07.336048 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:07.336029 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" event={"ID":"ae6a57bb-4c8c-419a-b114-ac91f3714dd2","Type":"ContainerDied","Data":"ccfcd47a429417cf2a9c1770ff4971656e587dd6de15fe4f84e42b27446e99cc"} Jul 17 01:51:07.336134 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:07.336049 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm" Jul 17 01:51:07.336134 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:07.336072 2583 scope.go:117] "RemoveContainer" containerID="b2a999e8fbe5bd9634c9b8df965b8872acf01a01ee896356ef22c3b341fb030c" Jul 17 01:51:07.365106 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:07.365064 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm"] Jul 17 01:51:07.368234 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:07.368210 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-87qbm"] Jul 17 01:51:09.167603 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:09.167566 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" path="/var/lib/kubelet/pods/ae6a57bb-4c8c-419a-b114-ac91f3714dd2/volumes" Jul 17 01:51:09.942718 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:09.942680 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:51:09.943146 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:09.943132 2583 scope.go:117] "RemoveContainer" containerID="3e763092a88a90ee1f5bc059049d7f588edbb4b73f5084049924232231adbd57" Jul 17 01:51:09.943340 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:09.943319 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:51:18.163155 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:18.163089 2583 scope.go:117] "RemoveContainer" containerID="62e860c4c2861017bae0782cbeeeb01b6d78456a1397e3a01abde3f6f3be35f4" Jul 17 01:51:18.388692 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:18.388651 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" event={"ID":"f490c66f-d27a-4745-bcbf-26e79ec593c5","Type":"ContainerStarted","Data":"d95bce9ae641a8cfc909c22130fcecb0ee42af631d287293d7c5cb9a28fa20b5"} Jul 17 01:51:19.394536 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:19.394502 2583 generic.go:356] "Generic (PLEG): container finished" podID="f490c66f-d27a-4745-bcbf-26e79ec593c5" containerID="d95bce9ae641a8cfc909c22130fcecb0ee42af631d287293d7c5cb9a28fa20b5" exitCode=7 Jul 17 01:51:19.394953 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:19.394568 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" event={"ID":"f490c66f-d27a-4745-bcbf-26e79ec593c5","Type":"ContainerDied","Data":"d95bce9ae641a8cfc909c22130fcecb0ee42af631d287293d7c5cb9a28fa20b5"} Jul 17 01:51:19.394953 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:19.394623 2583 scope.go:117] "RemoveContainer" containerID="62e860c4c2861017bae0782cbeeeb01b6d78456a1397e3a01abde3f6f3be35f4" Jul 17 01:51:19.394953 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:19.394940 2583 scope.go:117] "RemoveContainer" containerID="d95bce9ae641a8cfc909c22130fcecb0ee42af631d287293d7c5cb9a28fa20b5" Jul 17 01:51:19.395234 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:19.395212 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz_odh-ai-gateway-infra(f490c66f-d27a-4745-bcbf-26e79ec593c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" podUID="f490c66f-d27a-4745-bcbf-26e79ec593c5" Jul 17 01:51:19.419294 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:19.419262 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz"] Jul 17 01:51:20.527006 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:20.526984 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" Jul 17 01:51:20.671474 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:20.671381 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f490c66f-d27a-4745-bcbf-26e79ec593c5-maas-api-service-ca\") pod \"f490c66f-d27a-4745-bcbf-26e79ec593c5\" (UID: \"f490c66f-d27a-4745-bcbf-26e79ec593c5\") " Jul 17 01:51:20.671778 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:20.671757 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f490c66f-d27a-4745-bcbf-26e79ec593c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f490c66f-d27a-4745-bcbf-26e79ec593c5" (UID: "f490c66f-d27a-4745-bcbf-26e79ec593c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:51:20.772280 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:20.772240 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f490c66f-d27a-4745-bcbf-26e79ec593c5-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:51:21.162425 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:21.162392 2583 scope.go:117] "RemoveContainer" containerID="3e763092a88a90ee1f5bc059049d7f588edbb4b73f5084049924232231adbd57" Jul 17 01:51:21.162601 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:21.162565 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:51:21.406949 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:21.406914 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" event={"ID":"f490c66f-d27a-4745-bcbf-26e79ec593c5","Type":"ContainerDied","Data":"c688d3a8b69b0e742dbacae16189138b03ffa4fec639d711eaaff4911840cd81"} Jul 17 01:51:21.407139 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:21.406980 2583 scope.go:117] "RemoveContainer" containerID="d95bce9ae641a8cfc909c22130fcecb0ee42af631d287293d7c5cb9a28fa20b5" Jul 17 01:51:21.407139 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:21.406925 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz" Jul 17 01:51:21.425153 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:21.425124 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz"] Jul 17 01:51:21.426488 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:21.426465 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-l2ndz"] Jul 17 01:51:23.167720 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:23.167686 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f490c66f-d27a-4745-bcbf-26e79ec593c5" path="/var/lib/kubelet/pods/f490c66f-d27a-4745-bcbf-26e79ec593c5/volumes" Jul 17 01:51:34.162696 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:34.162661 2583 scope.go:117] "RemoveContainer" containerID="3e763092a88a90ee1f5bc059049d7f588edbb4b73f5084049924232231adbd57" Jul 17 01:51:34.163141 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:34.162860 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:51:42.268880 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:42.268838 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:51:49.162818 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:49.162783 2583 scope.go:117] "RemoveContainer" containerID="3e763092a88a90ee1f5bc059049d7f588edbb4b73f5084049924232231adbd57" Jul 17 01:51:50.537039 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:50.537002 2583 generic.go:356] "Generic (PLEG): container finished" podID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerID="37cf04191f5b2027b54fcf429fef8ed7a17caa5fba0e77c93e950ba880337ba1" exitCode=1 Jul 17 01:51:50.537527 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:50.537078 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" event={"ID":"244c0c1d-f71c-4bf3-80d5-e655fa015bda","Type":"ContainerDied","Data":"37cf04191f5b2027b54fcf429fef8ed7a17caa5fba0e77c93e950ba880337ba1"} Jul 17 01:51:50.537527 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:50.537152 2583 scope.go:117] "RemoveContainer" containerID="3e763092a88a90ee1f5bc059049d7f588edbb4b73f5084049924232231adbd57" Jul 17 01:51:50.537527 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:50.537509 2583 scope.go:117] "RemoveContainer" containerID="37cf04191f5b2027b54fcf429fef8ed7a17caa5fba0e77c93e950ba880337ba1" Jul 17 01:51:50.537716 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:50.537694 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:51:53.279608 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.279569 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6"] Jul 17 01:51:53.279996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.279892 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" containerName="revoke-keys" Jul 17 01:51:53.279996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.279903 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" containerName="revoke-keys" Jul 17 01:51:53.279996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.279913 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" containerName="revoke-keys" Jul 17 01:51:53.279996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.279919 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" containerName="revoke-keys" Jul 17 01:51:53.279996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.279928 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" containerName="revoke-keys" Jul 17 01:51:53.279996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.279934 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" containerName="revoke-keys" Jul 17 01:51:53.279996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.279945 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f490c66f-d27a-4745-bcbf-26e79ec593c5" containerName="revoke-keys" Jul 17 01:51:53.279996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.279949 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f490c66f-d27a-4745-bcbf-26e79ec593c5" containerName="revoke-keys" Jul 17 01:51:53.279996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.279967 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f490c66f-d27a-4745-bcbf-26e79ec593c5" containerName="revoke-keys" Jul 17 01:51:53.279996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.279972 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f490c66f-d27a-4745-bcbf-26e79ec593c5" containerName="revoke-keys" Jul 17 01:51:53.279996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.279977 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f490c66f-d27a-4745-bcbf-26e79ec593c5" containerName="revoke-keys" Jul 17 01:51:53.279996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.279983 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f490c66f-d27a-4745-bcbf-26e79ec593c5" containerName="revoke-keys" Jul 17 01:51:53.280388 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.280032 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" containerName="revoke-keys" Jul 17 01:51:53.280388 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.280040 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f490c66f-d27a-4745-bcbf-26e79ec593c5" containerName="revoke-keys" Jul 17 01:51:53.280388 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.280061 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f490c66f-d27a-4745-bcbf-26e79ec593c5" containerName="revoke-keys" Jul 17 01:51:53.280388 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.280068 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f490c66f-d27a-4745-bcbf-26e79ec593c5" containerName="revoke-keys" Jul 17 01:51:53.280388 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.280074 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" containerName="revoke-keys" Jul 17 01:51:53.284329 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.284308 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" Jul 17 01:51:53.286733 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.286711 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:51:53.290158 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.290127 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6"] Jul 17 01:51:53.354854 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.354816 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb62f37b-c95e-4973-b69e-7405f59ec800-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6\" (UID: \"bb62f37b-c95e-4973-b69e-7405f59ec800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" Jul 17 01:51:53.455764 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.455719 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb62f37b-c95e-4973-b69e-7405f59ec800-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6\" (UID: \"bb62f37b-c95e-4973-b69e-7405f59ec800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" Jul 17 01:51:53.456379 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.456359 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb62f37b-c95e-4973-b69e-7405f59ec800-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6\" (UID: \"bb62f37b-c95e-4973-b69e-7405f59ec800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" Jul 17 01:51:53.597616 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.597575 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" Jul 17 01:51:53.735962 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:53.735928 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6"] Jul 17 01:51:53.738707 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:51:53.738679 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb62f37b_c95e_4973_b69e_7405f59ec800.slice/crio-6ca1ea1b5b5a9064a4c66253b88b0195e0c722efd11a916ddc4020975bf65913 WatchSource:0}: Error finding container 6ca1ea1b5b5a9064a4c66253b88b0195e0c722efd11a916ddc4020975bf65913: Status 404 returned error can't find the container with id 6ca1ea1b5b5a9064a4c66253b88b0195e0c722efd11a916ddc4020975bf65913 Jul 17 01:51:54.556471 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:54.556433 2583 generic.go:356] "Generic (PLEG): container finished" podID="bb62f37b-c95e-4973-b69e-7405f59ec800" containerID="81cc22f290df360dfb3e0206582452e4e53d4b62fa2de61e4a112d710e5047b8" exitCode=7 Jul 17 01:51:54.556875 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:54.556507 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" event={"ID":"bb62f37b-c95e-4973-b69e-7405f59ec800","Type":"ContainerDied","Data":"81cc22f290df360dfb3e0206582452e4e53d4b62fa2de61e4a112d710e5047b8"} Jul 17 01:51:54.556875 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:54.556554 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" event={"ID":"bb62f37b-c95e-4973-b69e-7405f59ec800","Type":"ContainerStarted","Data":"6ca1ea1b5b5a9064a4c66253b88b0195e0c722efd11a916ddc4020975bf65913"} Jul 17 01:51:54.556875 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:54.556663 2583 scope.go:117] "RemoveContainer" containerID="81cc22f290df360dfb3e0206582452e4e53d4b62fa2de61e4a112d710e5047b8" Jul 17 01:51:54.942041 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:54.941940 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:51:54.942372 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:54.942358 2583 scope.go:117] "RemoveContainer" containerID="37cf04191f5b2027b54fcf429fef8ed7a17caa5fba0e77c93e950ba880337ba1" Jul 17 01:51:54.942565 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:54.942546 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:51:55.562703 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:55.562667 2583 generic.go:356] "Generic (PLEG): container finished" podID="bb62f37b-c95e-4973-b69e-7405f59ec800" containerID="83a1d7dea652b7a9fbbb6eb2a885d48aedd9f365ea7f5a55a9649c0224f08974" exitCode=7 Jul 17 01:51:55.563124 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:55.562726 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" event={"ID":"bb62f37b-c95e-4973-b69e-7405f59ec800","Type":"ContainerDied","Data":"83a1d7dea652b7a9fbbb6eb2a885d48aedd9f365ea7f5a55a9649c0224f08974"} Jul 17 01:51:55.563124 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:55.562785 2583 scope.go:117] "RemoveContainer" containerID="81cc22f290df360dfb3e0206582452e4e53d4b62fa2de61e4a112d710e5047b8" Jul 17 01:51:55.563124 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:55.563071 2583 scope.go:117] "RemoveContainer" containerID="83a1d7dea652b7a9fbbb6eb2a885d48aedd9f365ea7f5a55a9649c0224f08974" Jul 17 01:51:55.563290 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:55.563270 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6_odh-ai-gateway-infra(bb62f37b-c95e-4973-b69e-7405f59ec800)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" podUID="bb62f37b-c95e-4973-b69e-7405f59ec800" Jul 17 01:51:55.871350 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:55.871268 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch"] Jul 17 01:51:55.871677 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:55.871664 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae6a57bb-4c8c-419a-b114-ac91f3714dd2" containerName="revoke-keys" Jul 17 01:51:55.876154 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:55.876132 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" Jul 17 01:51:55.881259 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:55.881227 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch"] Jul 17 01:51:55.974366 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:55.974333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cc9cac-e06e-4381-83af-d1d9b3da88d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch\" (UID: \"05cc9cac-e06e-4381-83af-d1d9b3da88d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" Jul 17 01:51:56.075230 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:56.075184 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cc9cac-e06e-4381-83af-d1d9b3da88d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch\" (UID: \"05cc9cac-e06e-4381-83af-d1d9b3da88d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" Jul 17 01:51:56.075814 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:56.075794 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cc9cac-e06e-4381-83af-d1d9b3da88d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch\" (UID: \"05cc9cac-e06e-4381-83af-d1d9b3da88d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" Jul 17 01:51:56.188214 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:56.188118 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" Jul 17 01:51:56.322412 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:56.322385 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch"] Jul 17 01:51:56.324964 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:51:56.324928 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05cc9cac_e06e_4381_83af_d1d9b3da88d8.slice/crio-3be39dc4fc6c58feafc60c9a35e182f9670ea59727465072d9fad3ae07f81658 WatchSource:0}: Error finding container 3be39dc4fc6c58feafc60c9a35e182f9670ea59727465072d9fad3ae07f81658: Status 404 returned error can't find the container with id 3be39dc4fc6c58feafc60c9a35e182f9670ea59727465072d9fad3ae07f81658 Jul 17 01:51:56.568699 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:56.568659 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" event={"ID":"05cc9cac-e06e-4381-83af-d1d9b3da88d8","Type":"ContainerStarted","Data":"3be39dc4fc6c58feafc60c9a35e182f9670ea59727465072d9fad3ae07f81658"} Jul 17 01:51:56.570300 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:56.570281 2583 scope.go:117] "RemoveContainer" containerID="83a1d7dea652b7a9fbbb6eb2a885d48aedd9f365ea7f5a55a9649c0224f08974" Jul 17 01:51:56.570499 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:56.570480 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6_odh-ai-gateway-infra(bb62f37b-c95e-4973-b69e-7405f59ec800)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" podUID="bb62f37b-c95e-4973-b69e-7405f59ec800" Jul 17 01:51:57.575391 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:57.575355 2583 generic.go:356] "Generic (PLEG): container finished" podID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" containerID="a57245925246654c1844a869eea8f7d5268a6b27b4b3e2fb5ca09b7a496f25c5" exitCode=6 Jul 17 01:51:57.575846 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:57.575416 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" event={"ID":"05cc9cac-e06e-4381-83af-d1d9b3da88d8","Type":"ContainerDied","Data":"a57245925246654c1844a869eea8f7d5268a6b27b4b3e2fb5ca09b7a496f25c5"} Jul 17 01:51:57.575846 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:57.575610 2583 scope.go:117] "RemoveContainer" containerID="a57245925246654c1844a869eea8f7d5268a6b27b4b3e2fb5ca09b7a496f25c5" Jul 17 01:51:58.582062 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:58.582027 2583 generic.go:356] "Generic (PLEG): container finished" podID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" containerID="0ad7eea9e6a5d273e0bafb454e570926b768f7616e3899103b3f9fcb346337cc" exitCode=6 Jul 17 01:51:58.582503 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:58.582138 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" event={"ID":"05cc9cac-e06e-4381-83af-d1d9b3da88d8","Type":"ContainerDied","Data":"0ad7eea9e6a5d273e0bafb454e570926b768f7616e3899103b3f9fcb346337cc"} Jul 17 01:51:58.582503 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:58.582197 2583 scope.go:117] "RemoveContainer" containerID="a57245925246654c1844a869eea8f7d5268a6b27b4b3e2fb5ca09b7a496f25c5" Jul 17 01:51:58.582503 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:58.582459 2583 scope.go:117] "RemoveContainer" containerID="0ad7eea9e6a5d273e0bafb454e570926b768f7616e3899103b3f9fcb346337cc" Jul 17 01:51:58.582674 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:58.582653 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch_odh-ai-gateway-infra(05cc9cac-e06e-4381-83af-d1d9b3da88d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" podUID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" Jul 17 01:51:59.588858 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:59.588829 2583 scope.go:117] "RemoveContainer" containerID="0ad7eea9e6a5d273e0bafb454e570926b768f7616e3899103b3f9fcb346337cc" Jul 17 01:51:59.589290 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:59.589006 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch_odh-ai-gateway-infra(05cc9cac-e06e-4381-83af-d1d9b3da88d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" podUID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" Jul 17 01:51:59.942408 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:59.942316 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:51:59.942734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:51:59.942721 2583 scope.go:117] "RemoveContainer" containerID="37cf04191f5b2027b54fcf429fef8ed7a17caa5fba0e77c93e950ba880337ba1" Jul 17 01:51:59.942923 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:51:59.942901 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:52:04.106788 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:04.106744 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:52:09.580302 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:09.580263 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:52:09.682205 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:09.682177 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:52:11.162523 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:11.162485 2583 scope.go:117] "RemoveContainer" containerID="83a1d7dea652b7a9fbbb6eb2a885d48aedd9f365ea7f5a55a9649c0224f08974" Jul 17 01:52:11.647342 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:11.647309 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" event={"ID":"bb62f37b-c95e-4973-b69e-7405f59ec800","Type":"ContainerStarted","Data":"625d6cb52367a0b58a9cfb771e3f1ff6a39f0b4b25877cf17f960d5a4f2cd517"} Jul 17 01:52:11.662315 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:11.662262 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" podStartSLOduration=18.662246171 podStartE2EDuration="18.662246171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:52:11.660629523 +0000 UTC m=+2289.056244375" watchObservedRunningTime="2026-07-17 01:52:11.662246171 +0000 UTC m=+2289.057861022" Jul 17 01:52:12.163288 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:12.163192 2583 scope.go:117] "RemoveContainer" containerID="0ad7eea9e6a5d273e0bafb454e570926b768f7616e3899103b3f9fcb346337cc" Jul 17 01:52:12.410001 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:12.409969 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:52:12.654351 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:12.654313 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" event={"ID":"05cc9cac-e06e-4381-83af-d1d9b3da88d8","Type":"ContainerStarted","Data":"8769c7b818066c3470226d4e96ed15507c1a2be21c6c0566697e07f67e4f00bd"} Jul 17 01:52:12.656052 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:12.656028 2583 generic.go:356] "Generic (PLEG): container finished" podID="bb62f37b-c95e-4973-b69e-7405f59ec800" containerID="625d6cb52367a0b58a9cfb771e3f1ff6a39f0b4b25877cf17f960d5a4f2cd517" exitCode=7 Jul 17 01:52:12.656180 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:12.656062 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" event={"ID":"bb62f37b-c95e-4973-b69e-7405f59ec800","Type":"ContainerDied","Data":"625d6cb52367a0b58a9cfb771e3f1ff6a39f0b4b25877cf17f960d5a4f2cd517"} Jul 17 01:52:12.656180 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:12.656118 2583 scope.go:117] "RemoveContainer" containerID="83a1d7dea652b7a9fbbb6eb2a885d48aedd9f365ea7f5a55a9649c0224f08974" Jul 17 01:52:12.656327 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:12.656312 2583 scope.go:117] "RemoveContainer" containerID="625d6cb52367a0b58a9cfb771e3f1ff6a39f0b4b25877cf17f960d5a4f2cd517" Jul 17 01:52:12.656544 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:52:12.656519 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6_odh-ai-gateway-infra(bb62f37b-c95e-4973-b69e-7405f59ec800)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" podUID="bb62f37b-c95e-4973-b69e-7405f59ec800" Jul 17 01:52:12.670436 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:12.670410 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6"] Jul 17 01:52:12.671812 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:12.671775 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" podStartSLOduration=17.671762192 podStartE2EDuration="17.671762192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:52:12.670980934 +0000 UTC m=+2290.066595782" watchObservedRunningTime="2026-07-17 01:52:12.671762192 +0000 UTC m=+2290.067377042" Jul 17 01:52:13.164894 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:13.164871 2583 scope.go:117] "RemoveContainer" containerID="37cf04191f5b2027b54fcf429fef8ed7a17caa5fba0e77c93e950ba880337ba1" Jul 17 01:52:13.165384 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:52:13.165044 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:52:13.662563 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:13.662530 2583 generic.go:356] "Generic (PLEG): container finished" podID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" containerID="8769c7b818066c3470226d4e96ed15507c1a2be21c6c0566697e07f67e4f00bd" exitCode=6 Jul 17 01:52:13.662845 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:13.662568 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" event={"ID":"05cc9cac-e06e-4381-83af-d1d9b3da88d8","Type":"ContainerDied","Data":"8769c7b818066c3470226d4e96ed15507c1a2be21c6c0566697e07f67e4f00bd"} Jul 17 01:52:13.662845 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:13.662639 2583 scope.go:117] "RemoveContainer" containerID="0ad7eea9e6a5d273e0bafb454e570926b768f7616e3899103b3f9fcb346337cc" Jul 17 01:52:13.662992 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:13.662922 2583 scope.go:117] "RemoveContainer" containerID="8769c7b818066c3470226d4e96ed15507c1a2be21c6c0566697e07f67e4f00bd" Jul 17 01:52:13.663207 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:52:13.663174 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch_odh-ai-gateway-infra(05cc9cac-e06e-4381-83af-d1d9b3da88d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" podUID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" Jul 17 01:52:13.682776 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:13.682727 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch"] Jul 17 01:52:13.687400 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:13.687375 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:52:13.805074 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:13.805044 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" Jul 17 01:52:13.941167 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:13.941053 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb62f37b-c95e-4973-b69e-7405f59ec800-maas-api-service-ca\") pod \"bb62f37b-c95e-4973-b69e-7405f59ec800\" (UID: \"bb62f37b-c95e-4973-b69e-7405f59ec800\") " Jul 17 01:52:13.941447 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:13.941423 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb62f37b-c95e-4973-b69e-7405f59ec800-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb62f37b-c95e-4973-b69e-7405f59ec800" (UID: "bb62f37b-c95e-4973-b69e-7405f59ec800"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:52:14.042187 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:14.042153 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb62f37b-c95e-4973-b69e-7405f59ec800-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:52:14.672623 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:14.672588 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" Jul 17 01:52:14.672623 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:14.672588 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6" event={"ID":"bb62f37b-c95e-4973-b69e-7405f59ec800","Type":"ContainerDied","Data":"6ca1ea1b5b5a9064a4c66253b88b0195e0c722efd11a916ddc4020975bf65913"} Jul 17 01:52:14.673056 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:14.672675 2583 scope.go:117] "RemoveContainer" containerID="625d6cb52367a0b58a9cfb771e3f1ff6a39f0b4b25877cf17f960d5a4f2cd517" Jul 17 01:52:14.697386 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:14.697340 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6"] Jul 17 01:52:14.704604 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:14.704570 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4v4w6"] Jul 17 01:52:14.813647 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:14.813620 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" Jul 17 01:52:14.949706 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:14.949606 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cc9cac-e06e-4381-83af-d1d9b3da88d8-maas-api-service-ca\") pod \"05cc9cac-e06e-4381-83af-d1d9b3da88d8\" (UID: \"05cc9cac-e06e-4381-83af-d1d9b3da88d8\") " Jul 17 01:52:14.950017 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:14.949990 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05cc9cac-e06e-4381-83af-d1d9b3da88d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05cc9cac-e06e-4381-83af-d1d9b3da88d8" (UID: "05cc9cac-e06e-4381-83af-d1d9b3da88d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:52:15.050693 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:15.050652 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cc9cac-e06e-4381-83af-d1d9b3da88d8-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:52:15.168293 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:15.168264 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb62f37b-c95e-4973-b69e-7405f59ec800" path="/var/lib/kubelet/pods/bb62f37b-c95e-4973-b69e-7405f59ec800/volumes" Jul 17 01:52:15.678749 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:15.678718 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" Jul 17 01:52:15.679261 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:15.678720 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch" event={"ID":"05cc9cac-e06e-4381-83af-d1d9b3da88d8","Type":"ContainerDied","Data":"3be39dc4fc6c58feafc60c9a35e182f9670ea59727465072d9fad3ae07f81658"} Jul 17 01:52:15.679261 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:15.678842 2583 scope.go:117] "RemoveContainer" containerID="8769c7b818066c3470226d4e96ed15507c1a2be21c6c0566697e07f67e4f00bd" Jul 17 01:52:15.693406 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:15.693375 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch"] Jul 17 01:52:15.697335 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:15.697310 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vk9ch"] Jul 17 01:52:17.167560 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:17.167527 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" path="/var/lib/kubelet/pods/05cc9cac-e06e-4381-83af-d1d9b3da88d8/volumes" Jul 17 01:52:26.162798 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:26.162756 2583 scope.go:117] "RemoveContainer" containerID="37cf04191f5b2027b54fcf429fef8ed7a17caa5fba0e77c93e950ba880337ba1" Jul 17 01:52:26.163225 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:52:26.162953 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:52:38.162877 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:38.162845 2583 scope.go:117] "RemoveContainer" containerID="37cf04191f5b2027b54fcf429fef8ed7a17caa5fba0e77c93e950ba880337ba1" Jul 17 01:52:38.163311 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:52:38.163030 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:52:45.907266 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.907229 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg"] Jul 17 01:52:45.907723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.907576 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb62f37b-c95e-4973-b69e-7405f59ec800" containerName="revoke-keys" Jul 17 01:52:45.907723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.907587 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb62f37b-c95e-4973-b69e-7405f59ec800" containerName="revoke-keys" Jul 17 01:52:45.907723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.907595 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" containerName="revoke-keys" Jul 17 01:52:45.907723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.907601 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" containerName="revoke-keys" Jul 17 01:52:45.907723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.907610 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb62f37b-c95e-4973-b69e-7405f59ec800" containerName="revoke-keys" Jul 17 01:52:45.907723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.907615 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb62f37b-c95e-4973-b69e-7405f59ec800" containerName="revoke-keys" Jul 17 01:52:45.907723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.907621 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" containerName="revoke-keys" Jul 17 01:52:45.907723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.907626 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" containerName="revoke-keys" Jul 17 01:52:45.907723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.907680 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb62f37b-c95e-4973-b69e-7405f59ec800" containerName="revoke-keys" Jul 17 01:52:45.907723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.907686 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" containerName="revoke-keys" Jul 17 01:52:45.907723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.907696 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb62f37b-c95e-4973-b69e-7405f59ec800" containerName="revoke-keys" Jul 17 01:52:45.907723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.907702 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" containerName="revoke-keys" Jul 17 01:52:45.910509 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.910494 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" Jul 17 01:52:45.912819 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.912798 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:52:45.916422 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:45.916396 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg"] Jul 17 01:52:46.030910 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:46.030874 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc02d150-651e-4e22-8d99-f615d796b3b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg\" (UID: \"bc02d150-651e-4e22-8d99-f615d796b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" Jul 17 01:52:46.131422 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:46.131387 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc02d150-651e-4e22-8d99-f615d796b3b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg\" (UID: \"bc02d150-651e-4e22-8d99-f615d796b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" Jul 17 01:52:46.132051 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:46.132031 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc02d150-651e-4e22-8d99-f615d796b3b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg\" (UID: \"bc02d150-651e-4e22-8d99-f615d796b3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" Jul 17 01:52:46.222300 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:46.222208 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" Jul 17 01:52:46.356740 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:46.356527 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg"] Jul 17 01:52:46.358894 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:52:46.358867 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc02d150_651e_4e22_8d99_f615d796b3b8.slice/crio-1a12bbe5e363e205451b87018aab061101e2bddedd61c3f8824d03d23220683e WatchSource:0}: Error finding container 1a12bbe5e363e205451b87018aab061101e2bddedd61c3f8824d03d23220683e: Status 404 returned error can't find the container with id 1a12bbe5e363e205451b87018aab061101e2bddedd61c3f8824d03d23220683e Jul 17 01:52:46.833065 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:46.832981 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" event={"ID":"bc02d150-651e-4e22-8d99-f615d796b3b8","Type":"ContainerStarted","Data":"36209ef383909ec078a8dd7a9466decb0d00640decb3566e40b43372d82ede52"} Jul 17 01:52:46.833065 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:46.833035 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" event={"ID":"bc02d150-651e-4e22-8d99-f615d796b3b8","Type":"ContainerStarted","Data":"1a12bbe5e363e205451b87018aab061101e2bddedd61c3f8824d03d23220683e"} Jul 17 01:52:46.847909 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:46.847851 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" podStartSLOduration=1.8478343860000002 podStartE2EDuration="1.847834386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:52:46.847231555 +0000 UTC m=+2324.242846406" watchObservedRunningTime="2026-07-17 01:52:46.847834386 +0000 UTC m=+2324.243449280" Jul 17 01:52:47.838442 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:47.838406 2583 generic.go:356] "Generic (PLEG): container finished" podID="bc02d150-651e-4e22-8d99-f615d796b3b8" containerID="36209ef383909ec078a8dd7a9466decb0d00640decb3566e40b43372d82ede52" exitCode=6 Jul 17 01:52:47.838915 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:47.838489 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" event={"ID":"bc02d150-651e-4e22-8d99-f615d796b3b8","Type":"ContainerDied","Data":"36209ef383909ec078a8dd7a9466decb0d00640decb3566e40b43372d82ede52"} Jul 17 01:52:47.838915 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:47.838626 2583 scope.go:117] "RemoveContainer" containerID="36209ef383909ec078a8dd7a9466decb0d00640decb3566e40b43372d82ede52" Jul 17 01:52:48.844386 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:48.844352 2583 generic.go:356] "Generic (PLEG): container finished" podID="bc02d150-651e-4e22-8d99-f615d796b3b8" containerID="48ff275986a175632f749757ee7b4af9964ffb12f28c8287f6aa73d6fb197df1" exitCode=6 Jul 17 01:52:48.844864 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:48.844429 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" event={"ID":"bc02d150-651e-4e22-8d99-f615d796b3b8","Type":"ContainerDied","Data":"48ff275986a175632f749757ee7b4af9964ffb12f28c8287f6aa73d6fb197df1"} Jul 17 01:52:48.844864 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:48.844484 2583 scope.go:117] "RemoveContainer" containerID="36209ef383909ec078a8dd7a9466decb0d00640decb3566e40b43372d82ede52" Jul 17 01:52:48.844864 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:48.844684 2583 scope.go:117] "RemoveContainer" containerID="48ff275986a175632f749757ee7b4af9964ffb12f28c8287f6aa73d6fb197df1" Jul 17 01:52:48.845030 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:52:48.844962 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg_odh-ai-gateway-infra(bc02d150-651e-4e22-8d99-f615d796b3b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" podUID="bc02d150-651e-4e22-8d99-f615d796b3b8" Jul 17 01:52:49.163006 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:49.162916 2583 scope.go:117] "RemoveContainer" containerID="37cf04191f5b2027b54fcf429fef8ed7a17caa5fba0e77c93e950ba880337ba1" Jul 17 01:52:49.163215 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:52:49.163079 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:52:49.850878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:52:49.850846 2583 scope.go:117] "RemoveContainer" containerID="48ff275986a175632f749757ee7b4af9964ffb12f28c8287f6aa73d6fb197df1" Jul 17 01:52:49.851272 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:52:49.851049 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg_odh-ai-gateway-infra(bc02d150-651e-4e22-8d99-f615d796b3b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" podUID="bc02d150-651e-4e22-8d99-f615d796b3b8" Jul 17 01:53:01.162895 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:01.162811 2583 scope.go:117] "RemoveContainer" containerID="37cf04191f5b2027b54fcf429fef8ed7a17caa5fba0e77c93e950ba880337ba1" Jul 17 01:53:01.162895 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:01.162882 2583 scope.go:117] "RemoveContainer" containerID="48ff275986a175632f749757ee7b4af9964ffb12f28c8287f6aa73d6fb197df1" Jul 17 01:53:01.163461 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:53:01.163157 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:53:01.907881 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:01.907851 2583 generic.go:356] "Generic (PLEG): container finished" podID="bc02d150-651e-4e22-8d99-f615d796b3b8" containerID="d49f8089be82a8b097c7ea93dc0a05f35805a591e49c6ac6f6518be7fa8329d0" exitCode=6 Jul 17 01:53:01.908039 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:01.907930 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" event={"ID":"bc02d150-651e-4e22-8d99-f615d796b3b8","Type":"ContainerDied","Data":"d49f8089be82a8b097c7ea93dc0a05f35805a591e49c6ac6f6518be7fa8329d0"} Jul 17 01:53:01.908039 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:01.908023 2583 scope.go:117] "RemoveContainer" containerID="48ff275986a175632f749757ee7b4af9964ffb12f28c8287f6aa73d6fb197df1" Jul 17 01:53:01.908373 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:01.908355 2583 scope.go:117] "RemoveContainer" containerID="d49f8089be82a8b097c7ea93dc0a05f35805a591e49c6ac6f6518be7fa8329d0" Jul 17 01:53:01.908592 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:53:01.908571 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg_odh-ai-gateway-infra(bc02d150-651e-4e22-8d99-f615d796b3b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" podUID="bc02d150-651e-4e22-8d99-f615d796b3b8" Jul 17 01:53:02.932438 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:02.932405 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg"] Jul 17 01:53:03.067919 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:03.067887 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" Jul 17 01:53:03.184466 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:03.184377 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc02d150-651e-4e22-8d99-f615d796b3b8-maas-api-service-ca\") pod \"bc02d150-651e-4e22-8d99-f615d796b3b8\" (UID: \"bc02d150-651e-4e22-8d99-f615d796b3b8\") " Jul 17 01:53:03.184756 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:03.184730 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc02d150-651e-4e22-8d99-f615d796b3b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc02d150-651e-4e22-8d99-f615d796b3b8" (UID: "bc02d150-651e-4e22-8d99-f615d796b3b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:53:03.285477 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:03.285433 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc02d150-651e-4e22-8d99-f615d796b3b8-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:53:03.920225 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:03.920189 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" Jul 17 01:53:03.920225 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:03.920204 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg" event={"ID":"bc02d150-651e-4e22-8d99-f615d796b3b8","Type":"ContainerDied","Data":"1a12bbe5e363e205451b87018aab061101e2bddedd61c3f8824d03d23220683e"} Jul 17 01:53:03.920494 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:03.920260 2583 scope.go:117] "RemoveContainer" containerID="d49f8089be82a8b097c7ea93dc0a05f35805a591e49c6ac6f6518be7fa8329d0" Jul 17 01:53:03.942871 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:03.942844 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg"] Jul 17 01:53:03.948024 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:03.948001 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-hx2tg"] Jul 17 01:53:05.168037 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:05.168006 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc02d150-651e-4e22-8d99-f615d796b3b8" path="/var/lib/kubelet/pods/bc02d150-651e-4e22-8d99-f615d796b3b8/volumes" Jul 17 01:53:12.162808 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:12.162762 2583 scope.go:117] "RemoveContainer" containerID="37cf04191f5b2027b54fcf429fef8ed7a17caa5fba0e77c93e950ba880337ba1" Jul 17 01:53:12.964058 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:12.964031 2583 generic.go:356] "Generic (PLEG): container finished" podID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerID="cb749f0233dd114b98a0f926ec0bd7bfb65e55a78a678c81166dbbd0a32af4c3" exitCode=1 Jul 17 01:53:12.964193 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:12.964088 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" event={"ID":"244c0c1d-f71c-4bf3-80d5-e655fa015bda","Type":"ContainerDied","Data":"cb749f0233dd114b98a0f926ec0bd7bfb65e55a78a678c81166dbbd0a32af4c3"} Jul 17 01:53:12.964193 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:12.964145 2583 scope.go:117] "RemoveContainer" containerID="37cf04191f5b2027b54fcf429fef8ed7a17caa5fba0e77c93e950ba880337ba1" Jul 17 01:53:12.964470 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:12.964447 2583 scope.go:117] "RemoveContainer" containerID="cb749f0233dd114b98a0f926ec0bd7bfb65e55a78a678c81166dbbd0a32af4c3" Jul 17 01:53:12.964657 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:53:12.964636 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:53:13.356380 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:13.356336 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:13.489704 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:13.489675 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:14.941976 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:14.941944 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:53:14.942386 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:14.942355 2583 scope.go:117] "RemoveContainer" containerID="cb749f0233dd114b98a0f926ec0bd7bfb65e55a78a678c81166dbbd0a32af4c3" Jul 17 01:53:14.942567 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:53:14.942549 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv_odh-ai-gateway-infra(244c0c1d-f71c-4bf3-80d5-e655fa015bda)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" Jul 17 01:53:18.642414 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.642376 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv"] Jul 17 01:53:18.669311 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.669277 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:18.773222 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.773185 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:53:18.811613 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.811588 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244c0c1d-f71c-4bf3-80d5-e655fa015bda-maas-api-tls\") pod \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\" (UID: \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\") " Jul 17 01:53:18.811761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.811645 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244c0c1d-f71c-4bf3-80d5-e655fa015bda-openshift-service-ca\") pod \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\" (UID: \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\") " Jul 17 01:53:18.811761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.811713 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps8f9\" (UniqueName: \"kubernetes.io/projected/244c0c1d-f71c-4bf3-80d5-e655fa015bda-kube-api-access-ps8f9\") pod \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\" (UID: \"244c0c1d-f71c-4bf3-80d5-e655fa015bda\") " Jul 17 01:53:18.811941 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.811917 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244c0c1d-f71c-4bf3-80d5-e655fa015bda-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "244c0c1d-f71c-4bf3-80d5-e655fa015bda" (UID: "244c0c1d-f71c-4bf3-80d5-e655fa015bda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:53:18.813859 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.813830 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244c0c1d-f71c-4bf3-80d5-e655fa015bda-kube-api-access-ps8f9" (OuterVolumeSpecName: "kube-api-access-ps8f9") pod "244c0c1d-f71c-4bf3-80d5-e655fa015bda" (UID: "244c0c1d-f71c-4bf3-80d5-e655fa015bda"). InnerVolumeSpecName "kube-api-access-ps8f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:53:18.813965 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.813874 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244c0c1d-f71c-4bf3-80d5-e655fa015bda-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "244c0c1d-f71c-4bf3-80d5-e655fa015bda" (UID: "244c0c1d-f71c-4bf3-80d5-e655fa015bda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:53:18.913232 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.913155 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244c0c1d-f71c-4bf3-80d5-e655fa015bda-maas-api-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:53:18.913232 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.913180 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244c0c1d-f71c-4bf3-80d5-e655fa015bda-openshift-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:53:18.913232 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.913190 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps8f9\" (UniqueName: \"kubernetes.io/projected/244c0c1d-f71c-4bf3-80d5-e655fa015bda-kube-api-access-ps8f9\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:53:18.994788 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.994753 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" Jul 17 01:53:18.994945 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.994754 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv" event={"ID":"244c0c1d-f71c-4bf3-80d5-e655fa015bda","Type":"ContainerDied","Data":"3882646e201822debc0c4c597d17ea78b707f0433d40f625d1c2875f154e078d"} Jul 17 01:53:18.994945 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:18.994920 2583 scope.go:117] "RemoveContainer" containerID="cb749f0233dd114b98a0f926ec0bd7bfb65e55a78a678c81166dbbd0a32af4c3" Jul 17 01:53:19.017657 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.017627 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv"] Jul 17 01:53:19.019883 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.019863 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1e02ab63-c89cd98f-lk2qv"] Jul 17 01:53:19.168378 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.168302 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" path="/var/lib/kubelet/pods/244c0c1d-f71c-4bf3-80d5-e655fa015bda/volumes" Jul 17 01:53:19.255469 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.255443 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:19.856575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856541 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s"] Jul 17 01:53:19.856944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856840 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:19.856944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856850 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:19.856944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856863 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc02d150-651e-4e22-8d99-f615d796b3b8" containerName="revoke-keys" Jul 17 01:53:19.856944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856868 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc02d150-651e-4e22-8d99-f615d796b3b8" containerName="revoke-keys" Jul 17 01:53:19.856944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856879 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:19.856944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856884 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:19.856944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856894 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:19.856944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856899 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:19.856944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856906 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc02d150-651e-4e22-8d99-f615d796b3b8" containerName="revoke-keys" Jul 17 01:53:19.856944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856911 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc02d150-651e-4e22-8d99-f615d796b3b8" containerName="revoke-keys" Jul 17 01:53:19.856944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856922 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb62f37b-c95e-4973-b69e-7405f59ec800" containerName="revoke-keys" Jul 17 01:53:19.856944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856926 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb62f37b-c95e-4973-b69e-7405f59ec800" containerName="revoke-keys" Jul 17 01:53:19.856944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856931 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" containerName="revoke-keys" Jul 17 01:53:19.856944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856936 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" containerName="revoke-keys" Jul 17 01:53:19.857392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856981 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:19.857392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856988 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb62f37b-c95e-4973-b69e-7405f59ec800" containerName="revoke-keys" Jul 17 01:53:19.857392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.856997 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc02d150-651e-4e22-8d99-f615d796b3b8" containerName="revoke-keys" Jul 17 01:53:19.857392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.857002 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="05cc9cac-e06e-4381-83af-d1d9b3da88d8" containerName="revoke-keys" Jul 17 01:53:19.857392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.857006 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:19.857392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.857012 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc02d150-651e-4e22-8d99-f615d796b3b8" containerName="revoke-keys" Jul 17 01:53:19.857392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.857016 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:19.857392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.857022 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:19.857392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.857028 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:19.857392 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.857033 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc02d150-651e-4e22-8d99-f615d796b3b8" containerName="revoke-keys" Jul 17 01:53:19.861281 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.861265 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" Jul 17 01:53:19.863706 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.863598 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:53:19.864468 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.864448 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:53:19.866049 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.866028 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s"] Jul 17 01:53:19.921434 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:19.921398 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78194a12-c627-4312-9091-914cf6db8d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s\" (UID: \"78194a12-c627-4312-9091-914cf6db8d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" Jul 17 01:53:20.022094 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:20.022062 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78194a12-c627-4312-9091-914cf6db8d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s\" (UID: \"78194a12-c627-4312-9091-914cf6db8d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" Jul 17 01:53:20.022711 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:20.022688 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78194a12-c627-4312-9091-914cf6db8d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s\" (UID: \"78194a12-c627-4312-9091-914cf6db8d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" Jul 17 01:53:20.172901 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:20.172827 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" Jul 17 01:53:20.303452 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:20.303426 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s"] Jul 17 01:53:20.305883 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:53:20.305855 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78194a12_c627_4312_9091_914cf6db8d37.slice/crio-a1c1b16a854da800b72463d1c3de05cd405b62b0c67a0aa5c89cba362d261723 WatchSource:0}: Error finding container a1c1b16a854da800b72463d1c3de05cd405b62b0c67a0aa5c89cba362d261723: Status 404 returned error can't find the container with id a1c1b16a854da800b72463d1c3de05cd405b62b0c67a0aa5c89cba362d261723 Jul 17 01:53:21.007867 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:21.007837 2583 generic.go:356] "Generic (PLEG): container finished" podID="78194a12-c627-4312-9091-914cf6db8d37" containerID="37d81dddb9e573d74a25a5392c3b23d21b6bdfc689fe64d752416411b8e30340" exitCode=6 Jul 17 01:53:21.008327 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:21.007929 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" event={"ID":"78194a12-c627-4312-9091-914cf6db8d37","Type":"ContainerDied","Data":"37d81dddb9e573d74a25a5392c3b23d21b6bdfc689fe64d752416411b8e30340"} Jul 17 01:53:21.008327 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:21.007996 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" event={"ID":"78194a12-c627-4312-9091-914cf6db8d37","Type":"ContainerStarted","Data":"a1c1b16a854da800b72463d1c3de05cd405b62b0c67a0aa5c89cba362d261723"} Jul 17 01:53:21.008327 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:21.008173 2583 scope.go:117] "RemoveContainer" containerID="37d81dddb9e573d74a25a5392c3b23d21b6bdfc689fe64d752416411b8e30340" Jul 17 01:53:22.014037 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:22.013948 2583 generic.go:356] "Generic (PLEG): container finished" podID="78194a12-c627-4312-9091-914cf6db8d37" containerID="517069349d2c5b41fb3cb6c3c02565832beed5a8fbd97a8983fbc3a52682bd6f" exitCode=6 Jul 17 01:53:22.014037 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:22.014026 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" event={"ID":"78194a12-c627-4312-9091-914cf6db8d37","Type":"ContainerDied","Data":"517069349d2c5b41fb3cb6c3c02565832beed5a8fbd97a8983fbc3a52682bd6f"} Jul 17 01:53:22.014602 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:22.014081 2583 scope.go:117] "RemoveContainer" containerID="37d81dddb9e573d74a25a5392c3b23d21b6bdfc689fe64d752416411b8e30340" Jul 17 01:53:22.014602 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:22.014303 2583 scope.go:117] "RemoveContainer" containerID="517069349d2c5b41fb3cb6c3c02565832beed5a8fbd97a8983fbc3a52682bd6f" Jul 17 01:53:22.014602 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:53:22.014540 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s_odh-ai-gateway-infra(78194a12-c627-4312-9091-914cf6db8d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" podUID="78194a12-c627-4312-9091-914cf6db8d37" Jul 17 01:53:23.019946 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:23.019921 2583 scope.go:117] "RemoveContainer" containerID="517069349d2c5b41fb3cb6c3c02565832beed5a8fbd97a8983fbc3a52682bd6f" Jul 17 01:53:23.020395 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:53:23.020172 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s_odh-ai-gateway-infra(78194a12-c627-4312-9091-914cf6db8d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" podUID="78194a12-c627-4312-9091-914cf6db8d37" Jul 17 01:53:24.680417 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:24.680382 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:24.746901 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:24.746862 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:25.231654 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:25.231626 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:25.335019 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:25.334986 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:25.477079 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:25.477034 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:26.988122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:26.988068 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:29.810371 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:29.810332 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:32.593895 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:32.593861 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:32.655446 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:32.655419 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:32.747592 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:32.747560 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:32.845975 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:32.845879 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:32.946389 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:32.946356 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:33.372732 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:33.372705 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:53:35.947076 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:35.947043 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s"] Jul 17 01:53:35.947481 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:35.947395 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:35.947481 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:35.947407 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:35.947481 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:35.947419 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc02d150-651e-4e22-8d99-f615d796b3b8" containerName="revoke-keys" Jul 17 01:53:35.947481 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:35.947425 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc02d150-651e-4e22-8d99-f615d796b3b8" containerName="revoke-keys" Jul 17 01:53:35.947481 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:35.947434 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:35.947481 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:35.947439 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:35.947666 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:35.947491 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:35.951745 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:35.951729 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" Jul 17 01:53:35.956664 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:35.956637 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s"] Jul 17 01:53:36.063490 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:36.063447 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/897b8b8e-618c-40d7-b46d-44ca7b40ac83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s\" (UID: \"897b8b8e-618c-40d7-b46d-44ca7b40ac83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" Jul 17 01:53:36.164741 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:36.164705 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/897b8b8e-618c-40d7-b46d-44ca7b40ac83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s\" (UID: \"897b8b8e-618c-40d7-b46d-44ca7b40ac83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" Jul 17 01:53:36.165408 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:36.165384 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/897b8b8e-618c-40d7-b46d-44ca7b40ac83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s\" (UID: \"897b8b8e-618c-40d7-b46d-44ca7b40ac83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" Jul 17 01:53:36.263261 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:36.263169 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" Jul 17 01:53:36.393613 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:36.393560 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s"] Jul 17 01:53:36.395919 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:53:36.395887 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897b8b8e_618c_40d7_b46d_44ca7b40ac83.slice/crio-153c297dba69468b154158bb95bdc8cf48bbdedda18281dab8bd26ea43e831b1 WatchSource:0}: Error finding container 153c297dba69468b154158bb95bdc8cf48bbdedda18281dab8bd26ea43e831b1: Status 404 returned error can't find the container with id 153c297dba69468b154158bb95bdc8cf48bbdedda18281dab8bd26ea43e831b1 Jul 17 01:53:37.084172 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:37.084140 2583 generic.go:356] "Generic (PLEG): container finished" podID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" containerID="27f24aec2217b0da3d02db76f8ed84bc5e2e3fad9c446beb367a685928dc6580" exitCode=6 Jul 17 01:53:37.084551 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:37.084226 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" event={"ID":"897b8b8e-618c-40d7-b46d-44ca7b40ac83","Type":"ContainerDied","Data":"27f24aec2217b0da3d02db76f8ed84bc5e2e3fad9c446beb367a685928dc6580"} Jul 17 01:53:37.084551 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:37.084327 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" event={"ID":"897b8b8e-618c-40d7-b46d-44ca7b40ac83","Type":"ContainerStarted","Data":"153c297dba69468b154158bb95bdc8cf48bbdedda18281dab8bd26ea43e831b1"} Jul 17 01:53:37.084551 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:37.084391 2583 scope.go:117] "RemoveContainer" containerID="27f24aec2217b0da3d02db76f8ed84bc5e2e3fad9c446beb367a685928dc6580" Jul 17 01:53:37.162235 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:37.162206 2583 scope.go:117] "RemoveContainer" containerID="517069349d2c5b41fb3cb6c3c02565832beed5a8fbd97a8983fbc3a52682bd6f" Jul 17 01:53:38.091168 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:38.091134 2583 generic.go:356] "Generic (PLEG): container finished" podID="78194a12-c627-4312-9091-914cf6db8d37" containerID="e92d06c7c3cf3285ccbe041f1b6093029744439f83cb28b02d1743a970cb1c81" exitCode=6 Jul 17 01:53:38.091660 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:38.091209 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" event={"ID":"78194a12-c627-4312-9091-914cf6db8d37","Type":"ContainerDied","Data":"e92d06c7c3cf3285ccbe041f1b6093029744439f83cb28b02d1743a970cb1c81"} Jul 17 01:53:38.091660 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:38.091272 2583 scope.go:117] "RemoveContainer" containerID="517069349d2c5b41fb3cb6c3c02565832beed5a8fbd97a8983fbc3a52682bd6f" Jul 17 01:53:38.091660 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:38.091596 2583 scope.go:117] "RemoveContainer" containerID="e92d06c7c3cf3285ccbe041f1b6093029744439f83cb28b02d1743a970cb1c81" Jul 17 01:53:38.091894 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:53:38.091802 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s_odh-ai-gateway-infra(78194a12-c627-4312-9091-914cf6db8d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" podUID="78194a12-c627-4312-9091-914cf6db8d37" Jul 17 01:53:38.093181 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:38.093158 2583 generic.go:356] "Generic (PLEG): container finished" podID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" containerID="571defbe6584a9bcb9f8d0147ee47e770bf8ab2688ea5911b9f4d1085134d6a7" exitCode=6 Jul 17 01:53:38.093284 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:38.093241 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" event={"ID":"897b8b8e-618c-40d7-b46d-44ca7b40ac83","Type":"ContainerDied","Data":"571defbe6584a9bcb9f8d0147ee47e770bf8ab2688ea5911b9f4d1085134d6a7"} Jul 17 01:53:38.093409 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:38.093395 2583 scope.go:117] "RemoveContainer" containerID="571defbe6584a9bcb9f8d0147ee47e770bf8ab2688ea5911b9f4d1085134d6a7" Jul 17 01:53:38.093576 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:53:38.093558 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s_odh-ai-gateway-infra(897b8b8e-618c-40d7-b46d-44ca7b40ac83)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" podUID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" Jul 17 01:53:38.102767 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:38.102748 2583 scope.go:117] "RemoveContainer" containerID="27f24aec2217b0da3d02db76f8ed84bc5e2e3fad9c446beb367a685928dc6580" Jul 17 01:53:39.101213 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:39.101187 2583 scope.go:117] "RemoveContainer" containerID="571defbe6584a9bcb9f8d0147ee47e770bf8ab2688ea5911b9f4d1085134d6a7" Jul 17 01:53:39.101579 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:53:39.101352 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s_odh-ai-gateway-infra(897b8b8e-618c-40d7-b46d-44ca7b40ac83)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" podUID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" Jul 17 01:53:39.115255 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:39.115223 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s"] Jul 17 01:53:39.246985 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:39.246963 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" Jul 17 01:53:39.391528 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:39.391452 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78194a12-c627-4312-9091-914cf6db8d37-maas-api-service-ca\") pod \"78194a12-c627-4312-9091-914cf6db8d37\" (UID: \"78194a12-c627-4312-9091-914cf6db8d37\") " Jul 17 01:53:39.391789 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:39.391764 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78194a12-c627-4312-9091-914cf6db8d37-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78194a12-c627-4312-9091-914cf6db8d37" (UID: "78194a12-c627-4312-9091-914cf6db8d37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:53:39.492510 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:39.492479 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78194a12-c627-4312-9091-914cf6db8d37-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:53:40.106709 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:40.106675 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" Jul 17 01:53:40.107197 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:40.106673 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s" event={"ID":"78194a12-c627-4312-9091-914cf6db8d37","Type":"ContainerDied","Data":"a1c1b16a854da800b72463d1c3de05cd405b62b0c67a0aa5c89cba362d261723"} Jul 17 01:53:40.107197 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:40.106833 2583 scope.go:117] "RemoveContainer" containerID="e92d06c7c3cf3285ccbe041f1b6093029744439f83cb28b02d1743a970cb1c81" Jul 17 01:53:40.127497 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:40.127472 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s"] Jul 17 01:53:40.133304 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:40.133285 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-nlk5s"] Jul 17 01:53:41.167843 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:41.167812 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78194a12-c627-4312-9091-914cf6db8d37" path="/var/lib/kubelet/pods/78194a12-c627-4312-9091-914cf6db8d37/volumes" Jul 17 01:53:43.368852 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.368822 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj"] Jul 17 01:53:43.369204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.369153 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78194a12-c627-4312-9091-914cf6db8d37" containerName="revoke-keys" Jul 17 01:53:43.369204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.369165 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="78194a12-c627-4312-9091-914cf6db8d37" containerName="revoke-keys" Jul 17 01:53:43.369204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.369180 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:43.369204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.369189 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="244c0c1d-f71c-4bf3-80d5-e655fa015bda" containerName="maas-api" Jul 17 01:53:43.369357 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.369211 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78194a12-c627-4312-9091-914cf6db8d37" containerName="revoke-keys" Jul 17 01:53:43.369357 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.369216 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="78194a12-c627-4312-9091-914cf6db8d37" containerName="revoke-keys" Jul 17 01:53:43.369357 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.369267 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="78194a12-c627-4312-9091-914cf6db8d37" containerName="revoke-keys" Jul 17 01:53:43.369357 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.369277 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="78194a12-c627-4312-9091-914cf6db8d37" containerName="revoke-keys" Jul 17 01:53:43.373651 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.373630 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" Jul 17 01:53:43.378066 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.378041 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj"] Jul 17 01:53:43.530201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.530170 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5dba0-a96e-4f6c-93e9-46d8464f7945-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj\" (UID: \"61b5dba0-a96e-4f6c-93e9-46d8464f7945\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" Jul 17 01:53:43.631151 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.631039 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5dba0-a96e-4f6c-93e9-46d8464f7945-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj\" (UID: \"61b5dba0-a96e-4f6c-93e9-46d8464f7945\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" Jul 17 01:53:43.631648 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.631628 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5dba0-a96e-4f6c-93e9-46d8464f7945-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj\" (UID: \"61b5dba0-a96e-4f6c-93e9-46d8464f7945\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" Jul 17 01:53:43.685825 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.685800 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" Jul 17 01:53:43.811598 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:43.811572 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj"] Jul 17 01:53:43.814313 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:53:43.814289 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b5dba0_a96e_4f6c_93e9_46d8464f7945.slice/crio-0d0fff015d7c4aed04243cbf0690c2aa5a4ed027e1985895fd29cc1ddcac25c5 WatchSource:0}: Error finding container 0d0fff015d7c4aed04243cbf0690c2aa5a4ed027e1985895fd29cc1ddcac25c5: Status 404 returned error can't find the container with id 0d0fff015d7c4aed04243cbf0690c2aa5a4ed027e1985895fd29cc1ddcac25c5 Jul 17 01:53:44.127271 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:44.127233 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" event={"ID":"61b5dba0-a96e-4f6c-93e9-46d8464f7945","Type":"ContainerStarted","Data":"4566b6cb153b94415f9072adc370dcf3fa5a4ff8cf27beab2a46b5b08af4b008"} Jul 17 01:53:44.127413 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:44.127280 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" event={"ID":"61b5dba0-a96e-4f6c-93e9-46d8464f7945","Type":"ContainerStarted","Data":"0d0fff015d7c4aed04243cbf0690c2aa5a4ed027e1985895fd29cc1ddcac25c5"} Jul 17 01:53:44.141823 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:44.141775 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" podStartSLOduration=1.141757613 podStartE2EDuration="1.141757613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:53:44.140674647 +0000 UTC m=+2381.536289508" watchObservedRunningTime="2026-07-17 01:53:44.141757613 +0000 UTC m=+2381.537372466" Jul 17 01:53:45.133509 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:45.133474 2583 generic.go:356] "Generic (PLEG): container finished" podID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" containerID="4566b6cb153b94415f9072adc370dcf3fa5a4ff8cf27beab2a46b5b08af4b008" exitCode=6 Jul 17 01:53:45.133876 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:45.133534 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" event={"ID":"61b5dba0-a96e-4f6c-93e9-46d8464f7945","Type":"ContainerDied","Data":"4566b6cb153b94415f9072adc370dcf3fa5a4ff8cf27beab2a46b5b08af4b008"} Jul 17 01:53:45.133876 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:45.133719 2583 scope.go:117] "RemoveContainer" containerID="4566b6cb153b94415f9072adc370dcf3fa5a4ff8cf27beab2a46b5b08af4b008" Jul 17 01:53:46.140256 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:46.140178 2583 generic.go:356] "Generic (PLEG): container finished" podID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" containerID="0a063d4f80a7186c57d2c833ae08665cc4f7074af5f26cdf690061827108e746" exitCode=6 Jul 17 01:53:46.140256 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:46.140233 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" event={"ID":"61b5dba0-a96e-4f6c-93e9-46d8464f7945","Type":"ContainerDied","Data":"0a063d4f80a7186c57d2c833ae08665cc4f7074af5f26cdf690061827108e746"} Jul 17 01:53:46.140678 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:46.140281 2583 scope.go:117] "RemoveContainer" containerID="4566b6cb153b94415f9072adc370dcf3fa5a4ff8cf27beab2a46b5b08af4b008" Jul 17 01:53:46.140678 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:46.140443 2583 scope.go:117] "RemoveContainer" containerID="0a063d4f80a7186c57d2c833ae08665cc4f7074af5f26cdf690061827108e746" Jul 17 01:53:46.140678 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:53:46.140646 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj_odh-ai-gateway-infra(61b5dba0-a96e-4f6c-93e9-46d8464f7945)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" podUID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" Jul 17 01:53:47.146472 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:47.146440 2583 scope.go:117] "RemoveContainer" containerID="0a063d4f80a7186c57d2c833ae08665cc4f7074af5f26cdf690061827108e746" Jul 17 01:53:47.146887 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:53:47.146621 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj_odh-ai-gateway-infra(61b5dba0-a96e-4f6c-93e9-46d8464f7945)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" podUID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" Jul 17 01:53:53.165769 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:53.165739 2583 scope.go:117] "RemoveContainer" containerID="571defbe6584a9bcb9f8d0147ee47e770bf8ab2688ea5911b9f4d1085134d6a7" Jul 17 01:53:54.180448 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:54.180360 2583 generic.go:356] "Generic (PLEG): container finished" podID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" containerID="32ae055f8b09a7a2be03bc132050043c2a35a04f07a29a34dbc4ed539cc40c79" exitCode=6 Jul 17 01:53:54.180448 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:54.180424 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" event={"ID":"897b8b8e-618c-40d7-b46d-44ca7b40ac83","Type":"ContainerDied","Data":"32ae055f8b09a7a2be03bc132050043c2a35a04f07a29a34dbc4ed539cc40c79"} Jul 17 01:53:54.180926 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:54.180475 2583 scope.go:117] "RemoveContainer" containerID="571defbe6584a9bcb9f8d0147ee47e770bf8ab2688ea5911b9f4d1085134d6a7" Jul 17 01:53:54.180926 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:54.180746 2583 scope.go:117] "RemoveContainer" containerID="32ae055f8b09a7a2be03bc132050043c2a35a04f07a29a34dbc4ed539cc40c79" Jul 17 01:53:54.181025 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:53:54.180968 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s_odh-ai-gateway-infra(897b8b8e-618c-40d7-b46d-44ca7b40ac83)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" podUID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" Jul 17 01:53:55.203602 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:55.203566 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s"] Jul 17 01:53:55.340677 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:55.340654 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" Jul 17 01:53:55.536442 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:55.536419 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/897b8b8e-618c-40d7-b46d-44ca7b40ac83-maas-api-service-ca\") pod \"897b8b8e-618c-40d7-b46d-44ca7b40ac83\" (UID: \"897b8b8e-618c-40d7-b46d-44ca7b40ac83\") " Jul 17 01:53:55.536775 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:55.536748 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897b8b8e-618c-40d7-b46d-44ca7b40ac83-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "897b8b8e-618c-40d7-b46d-44ca7b40ac83" (UID: "897b8b8e-618c-40d7-b46d-44ca7b40ac83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:53:55.637460 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:55.637426 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/897b8b8e-618c-40d7-b46d-44ca7b40ac83-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:53:56.191561 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:56.191534 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" Jul 17 01:53:56.191746 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:56.191571 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s" event={"ID":"897b8b8e-618c-40d7-b46d-44ca7b40ac83","Type":"ContainerDied","Data":"153c297dba69468b154158bb95bdc8cf48bbdedda18281dab8bd26ea43e831b1"} Jul 17 01:53:56.191746 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:56.191634 2583 scope.go:117] "RemoveContainer" containerID="32ae055f8b09a7a2be03bc132050043c2a35a04f07a29a34dbc4ed539cc40c79" Jul 17 01:53:56.213660 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:56.213634 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s"] Jul 17 01:53:56.219398 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:56.219375 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-58f2s"] Jul 17 01:53:57.167359 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:57.167326 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" path="/var/lib/kubelet/pods/897b8b8e-618c-40d7-b46d-44ca7b40ac83/volumes" Jul 17 01:53:59.162850 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:59.162805 2583 scope.go:117] "RemoveContainer" containerID="0a063d4f80a7186c57d2c833ae08665cc4f7074af5f26cdf690061827108e746" Jul 17 01:53:59.333088 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:53:59.333064 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:00.212127 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:00.211764 2583 generic.go:356] "Generic (PLEG): container finished" podID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" containerID="9d0fe298852f34df67f7cd9cc34c3de7af3eff4d61096e369a9f562f9b2950d7" exitCode=6 Jul 17 01:54:00.212641 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:00.211833 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" event={"ID":"61b5dba0-a96e-4f6c-93e9-46d8464f7945","Type":"ContainerDied","Data":"9d0fe298852f34df67f7cd9cc34c3de7af3eff4d61096e369a9f562f9b2950d7"} Jul 17 01:54:00.212836 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:00.212822 2583 scope.go:117] "RemoveContainer" containerID="0a063d4f80a7186c57d2c833ae08665cc4f7074af5f26cdf690061827108e746" Jul 17 01:54:00.213388 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:00.213369 2583 scope.go:117] "RemoveContainer" containerID="9d0fe298852f34df67f7cd9cc34c3de7af3eff4d61096e369a9f562f9b2950d7" Jul 17 01:54:00.213875 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:54:00.213851 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj_odh-ai-gateway-infra(61b5dba0-a96e-4f6c-93e9-46d8464f7945)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" podUID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" Jul 17 01:54:01.239700 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:01.239673 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj"] Jul 17 01:54:01.371799 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:01.371773 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" Jul 17 01:54:01.383348 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:01.383323 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5dba0-a96e-4f6c-93e9-46d8464f7945-maas-api-service-ca\") pod \"61b5dba0-a96e-4f6c-93e9-46d8464f7945\" (UID: \"61b5dba0-a96e-4f6c-93e9-46d8464f7945\") " Jul 17 01:54:01.383667 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:01.383641 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b5dba0-a96e-4f6c-93e9-46d8464f7945-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "61b5dba0-a96e-4f6c-93e9-46d8464f7945" (UID: "61b5dba0-a96e-4f6c-93e9-46d8464f7945"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:54:01.484308 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:01.484273 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b5dba0-a96e-4f6c-93e9-46d8464f7945-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:54:02.153369 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:02.153321 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:02.204613 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:02.204527 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:02.224448 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:02.224419 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" event={"ID":"61b5dba0-a96e-4f6c-93e9-46d8464f7945","Type":"ContainerDied","Data":"0d0fff015d7c4aed04243cbf0690c2aa5a4ed027e1985895fd29cc1ddcac25c5"} Jul 17 01:54:02.224448 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:02.224438 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj" Jul 17 01:54:02.224664 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:02.224472 2583 scope.go:117] "RemoveContainer" containerID="9d0fe298852f34df67f7cd9cc34c3de7af3eff4d61096e369a9f562f9b2950d7" Jul 17 01:54:02.244318 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:02.244289 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj"] Jul 17 01:54:02.248238 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:02.248215 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-jkqqj"] Jul 17 01:54:03.167862 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:03.167832 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" path="/var/lib/kubelet/pods/61b5dba0-a96e-4f6c-93e9-46d8464f7945/volumes" Jul 17 01:54:03.223596 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:03.223572 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:54:03.227332 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:03.227310 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:54:05.015672 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:05.015634 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:05.117630 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:05.117594 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:05.215648 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:05.215605 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:18.009909 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.009872 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k"] Jul 17 01:54:18.010336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010198 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" containerName="revoke-keys" Jul 17 01:54:18.010336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010211 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" containerName="revoke-keys" Jul 17 01:54:18.010336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010223 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" containerName="revoke-keys" Jul 17 01:54:18.010336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010230 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" containerName="revoke-keys" Jul 17 01:54:18.010336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010243 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" containerName="revoke-keys" Jul 17 01:54:18.010336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010248 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" containerName="revoke-keys" Jul 17 01:54:18.010336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010262 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78194a12-c627-4312-9091-914cf6db8d37" containerName="revoke-keys" Jul 17 01:54:18.010336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010270 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="78194a12-c627-4312-9091-914cf6db8d37" containerName="revoke-keys" Jul 17 01:54:18.010336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010280 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" containerName="revoke-keys" Jul 17 01:54:18.010336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010287 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" containerName="revoke-keys" Jul 17 01:54:18.010640 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010348 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" containerName="revoke-keys" Jul 17 01:54:18.010640 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010355 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" containerName="revoke-keys" Jul 17 01:54:18.010640 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010361 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" containerName="revoke-keys" Jul 17 01:54:18.010640 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010368 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="78194a12-c627-4312-9091-914cf6db8d37" containerName="revoke-keys" Jul 17 01:54:18.010640 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.010374 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" containerName="revoke-keys" Jul 17 01:54:18.013147 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.013130 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" Jul 17 01:54:18.015803 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.015775 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:54:18.015917 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.015813 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:54:18.021798 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.021769 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k"] Jul 17 01:54:18.130471 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.130433 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6cb41a7-22a8-4c9f-9f02-4901779f7eff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k\" (UID: \"c6cb41a7-22a8-4c9f-9f02-4901779f7eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" Jul 17 01:54:18.231215 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.231171 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6cb41a7-22a8-4c9f-9f02-4901779f7eff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k\" (UID: \"c6cb41a7-22a8-4c9f-9f02-4901779f7eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" Jul 17 01:54:18.231863 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.231843 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6cb41a7-22a8-4c9f-9f02-4901779f7eff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k\" (UID: \"c6cb41a7-22a8-4c9f-9f02-4901779f7eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" Jul 17 01:54:18.323938 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.323898 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" Jul 17 01:54:18.461052 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:18.460404 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k"] Jul 17 01:54:18.466740 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:54:18.466706 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6cb41a7_22a8_4c9f_9f02_4901779f7eff.slice/crio-bf8e7874f7025047d7eb3c0eb958817bbbf2ecd1312cab98a52d92b792d9dd5d WatchSource:0}: Error finding container bf8e7874f7025047d7eb3c0eb958817bbbf2ecd1312cab98a52d92b792d9dd5d: Status 404 returned error can't find the container with id bf8e7874f7025047d7eb3c0eb958817bbbf2ecd1312cab98a52d92b792d9dd5d Jul 17 01:54:19.302895 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:19.302860 2583 generic.go:356] "Generic (PLEG): container finished" podID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" containerID="fde696fd675350af995beaa57341e018cc7eec1ff465365862df32f9cbef23af" exitCode=6 Jul 17 01:54:19.303357 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:19.302955 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" event={"ID":"c6cb41a7-22a8-4c9f-9f02-4901779f7eff","Type":"ContainerDied","Data":"fde696fd675350af995beaa57341e018cc7eec1ff465365862df32f9cbef23af"} Jul 17 01:54:19.303357 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:19.303021 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" event={"ID":"c6cb41a7-22a8-4c9f-9f02-4901779f7eff","Type":"ContainerStarted","Data":"bf8e7874f7025047d7eb3c0eb958817bbbf2ecd1312cab98a52d92b792d9dd5d"} Jul 17 01:54:19.303357 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:19.303150 2583 scope.go:117] "RemoveContainer" containerID="fde696fd675350af995beaa57341e018cc7eec1ff465365862df32f9cbef23af" Jul 17 01:54:19.896944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:19.896908 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t"] Jul 17 01:54:19.897295 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:19.897282 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" containerName="revoke-keys" Jul 17 01:54:19.897365 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:19.897298 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" containerName="revoke-keys" Jul 17 01:54:19.897365 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:19.897313 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" containerName="revoke-keys" Jul 17 01:54:19.897365 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:19.897318 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" containerName="revoke-keys" Jul 17 01:54:19.897472 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:19.897369 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="61b5dba0-a96e-4f6c-93e9-46d8464f7945" containerName="revoke-keys" Jul 17 01:54:19.897472 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:19.897376 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="897b8b8e-618c-40d7-b46d-44ca7b40ac83" containerName="revoke-keys" Jul 17 01:54:19.900213 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:19.900196 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" Jul 17 01:54:19.908119 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:19.908066 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t"] Jul 17 01:54:19.946931 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:19.946894 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb58b0bf-df27-49de-998d-cba636947f8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t\" (UID: \"eb58b0bf-df27-49de-998d-cba636947f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" Jul 17 01:54:20.048192 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:20.048152 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb58b0bf-df27-49de-998d-cba636947f8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t\" (UID: \"eb58b0bf-df27-49de-998d-cba636947f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" Jul 17 01:54:20.048761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:20.048742 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb58b0bf-df27-49de-998d-cba636947f8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t\" (UID: \"eb58b0bf-df27-49de-998d-cba636947f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" Jul 17 01:54:20.211914 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:20.211825 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" Jul 17 01:54:20.311385 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:20.311348 2583 generic.go:356] "Generic (PLEG): container finished" podID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" containerID="3584531cd9946d56b4ba49a9bb5a6dd0be459d4ebb8c25a3b8108b3865c47c82" exitCode=6 Jul 17 01:54:20.311893 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:20.311426 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" event={"ID":"c6cb41a7-22a8-4c9f-9f02-4901779f7eff","Type":"ContainerDied","Data":"3584531cd9946d56b4ba49a9bb5a6dd0be459d4ebb8c25a3b8108b3865c47c82"} Jul 17 01:54:20.311893 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:20.311489 2583 scope.go:117] "RemoveContainer" containerID="fde696fd675350af995beaa57341e018cc7eec1ff465365862df32f9cbef23af" Jul 17 01:54:20.311893 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:20.311675 2583 scope.go:117] "RemoveContainer" containerID="3584531cd9946d56b4ba49a9bb5a6dd0be459d4ebb8c25a3b8108b3865c47c82" Jul 17 01:54:20.312070 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:54:20.311905 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k_odh-ai-gateway-infra(c6cb41a7-22a8-4c9f-9f02-4901779f7eff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" podUID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" Jul 17 01:54:20.343946 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:20.343924 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t"] Jul 17 01:54:20.346521 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:54:20.346498 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb58b0bf_df27_49de_998d_cba636947f8b.slice/crio-61d21f9891c54b5c129b6eeb6dc7bca498d6a3dff506de20a84cfbf9754ee0f1 WatchSource:0}: Error finding container 61d21f9891c54b5c129b6eeb6dc7bca498d6a3dff506de20a84cfbf9754ee0f1: Status 404 returned error can't find the container with id 61d21f9891c54b5c129b6eeb6dc7bca498d6a3dff506de20a84cfbf9754ee0f1 Jul 17 01:54:21.119196 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:21.119153 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:21.317874 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:21.317836 2583 generic.go:356] "Generic (PLEG): container finished" podID="eb58b0bf-df27-49de-998d-cba636947f8b" containerID="8c696045cba41bc48cb1c9bdc840ca68a972ad12c14c9fbee8701a00f2dfb552" exitCode=6 Jul 17 01:54:21.318336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:21.317927 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" event={"ID":"eb58b0bf-df27-49de-998d-cba636947f8b","Type":"ContainerDied","Data":"8c696045cba41bc48cb1c9bdc840ca68a972ad12c14c9fbee8701a00f2dfb552"} Jul 17 01:54:21.318336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:21.317988 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" event={"ID":"eb58b0bf-df27-49de-998d-cba636947f8b","Type":"ContainerStarted","Data":"61d21f9891c54b5c129b6eeb6dc7bca498d6a3dff506de20a84cfbf9754ee0f1"} Jul 17 01:54:21.318336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:21.318178 2583 scope.go:117] "RemoveContainer" containerID="8c696045cba41bc48cb1c9bdc840ca68a972ad12c14c9fbee8701a00f2dfb552" Jul 17 01:54:21.319659 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:21.319643 2583 scope.go:117] "RemoveContainer" containerID="3584531cd9946d56b4ba49a9bb5a6dd0be459d4ebb8c25a3b8108b3865c47c82" Jul 17 01:54:21.319817 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:54:21.319800 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k_odh-ai-gateway-infra(c6cb41a7-22a8-4c9f-9f02-4901779f7eff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" podUID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" Jul 17 01:54:22.326344 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:22.326312 2583 generic.go:356] "Generic (PLEG): container finished" podID="eb58b0bf-df27-49de-998d-cba636947f8b" containerID="314d317c3444c4d2880210087ad8ec0138b12640ac5c797fc05530f340b017ac" exitCode=6 Jul 17 01:54:22.326813 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:22.326381 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" event={"ID":"eb58b0bf-df27-49de-998d-cba636947f8b","Type":"ContainerDied","Data":"314d317c3444c4d2880210087ad8ec0138b12640ac5c797fc05530f340b017ac"} Jul 17 01:54:22.326813 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:22.326423 2583 scope.go:117] "RemoveContainer" containerID="8c696045cba41bc48cb1c9bdc840ca68a972ad12c14c9fbee8701a00f2dfb552" Jul 17 01:54:22.326813 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:22.326569 2583 scope.go:117] "RemoveContainer" containerID="314d317c3444c4d2880210087ad8ec0138b12640ac5c797fc05530f340b017ac" Jul 17 01:54:22.326813 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:54:22.326800 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t_odh-ai-gateway-infra(eb58b0bf-df27-49de-998d-cba636947f8b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" podUID="eb58b0bf-df27-49de-998d-cba636947f8b" Jul 17 01:54:23.332170 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:23.332141 2583 scope.go:117] "RemoveContainer" containerID="314d317c3444c4d2880210087ad8ec0138b12640ac5c797fc05530f340b017ac" Jul 17 01:54:23.332532 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:54:23.332367 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t_odh-ai-gateway-infra(eb58b0bf-df27-49de-998d-cba636947f8b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" podUID="eb58b0bf-df27-49de-998d-cba636947f8b" Jul 17 01:54:26.668512 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:26.668476 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:26.767579 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:26.767543 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:26.882175 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:26.882140 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:33.407673 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:33.407633 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b"] Jul 17 01:54:33.412518 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:33.412496 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" Jul 17 01:54:33.417851 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:33.417821 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b"] Jul 17 01:54:33.571903 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:33.571861 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41916647-58d8-4e16-a5fb-24825b145130-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b\" (UID: \"41916647-58d8-4e16-a5fb-24825b145130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" Jul 17 01:54:33.673256 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:33.673167 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41916647-58d8-4e16-a5fb-24825b145130-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b\" (UID: \"41916647-58d8-4e16-a5fb-24825b145130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" Jul 17 01:54:33.673817 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:33.673798 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41916647-58d8-4e16-a5fb-24825b145130-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b\" (UID: \"41916647-58d8-4e16-a5fb-24825b145130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" Jul 17 01:54:33.724382 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:33.724342 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" Jul 17 01:54:33.856208 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:33.856141 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b"] Jul 17 01:54:33.858405 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:54:33.858380 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41916647_58d8_4e16_a5fb_24825b145130.slice/crio-1822c2f172f1b56a052e0556608a6fe932c9ea8bf1ba62a0bc199f6e4e7e3155 WatchSource:0}: Error finding container 1822c2f172f1b56a052e0556608a6fe932c9ea8bf1ba62a0bc199f6e4e7e3155: Status 404 returned error can't find the container with id 1822c2f172f1b56a052e0556608a6fe932c9ea8bf1ba62a0bc199f6e4e7e3155 Jul 17 01:54:34.387696 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:34.387608 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" event={"ID":"41916647-58d8-4e16-a5fb-24825b145130","Type":"ContainerStarted","Data":"bd7c1acb82655d2fa2536a4b219f7eba55a94e304fc142e0355651f7a9c3ae01"} Jul 17 01:54:34.387696 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:34.387659 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" event={"ID":"41916647-58d8-4e16-a5fb-24825b145130","Type":"ContainerStarted","Data":"1822c2f172f1b56a052e0556608a6fe932c9ea8bf1ba62a0bc199f6e4e7e3155"} Jul 17 01:54:34.403197 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:34.403140 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" podStartSLOduration=1.403124566 podStartE2EDuration="1.403124566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:54:34.40122294 +0000 UTC m=+2431.796837795" watchObservedRunningTime="2026-07-17 01:54:34.403124566 +0000 UTC m=+2431.798739417" Jul 17 01:54:35.162612 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:35.162587 2583 scope.go:117] "RemoveContainer" containerID="314d317c3444c4d2880210087ad8ec0138b12640ac5c797fc05530f340b017ac" Jul 17 01:54:35.394814 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:35.394766 2583 generic.go:356] "Generic (PLEG): container finished" podID="41916647-58d8-4e16-a5fb-24825b145130" containerID="bd7c1acb82655d2fa2536a4b219f7eba55a94e304fc142e0355651f7a9c3ae01" exitCode=6 Jul 17 01:54:35.394994 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:35.394874 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" event={"ID":"41916647-58d8-4e16-a5fb-24825b145130","Type":"ContainerDied","Data":"bd7c1acb82655d2fa2536a4b219f7eba55a94e304fc142e0355651f7a9c3ae01"} Jul 17 01:54:35.395064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:35.395045 2583 scope.go:117] "RemoveContainer" containerID="bd7c1acb82655d2fa2536a4b219f7eba55a94e304fc142e0355651f7a9c3ae01" Jul 17 01:54:35.396805 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:35.396781 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" event={"ID":"eb58b0bf-df27-49de-998d-cba636947f8b","Type":"ContainerStarted","Data":"831eca11de72fcef40ef65963a034aa427df13fd7bbcfddaaf742ea5f3c7b5a9"} Jul 17 01:54:35.426634 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:35.426543 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" podStartSLOduration=16.426526429 podStartE2EDuration="16.426526429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:54:35.425640833 +0000 UTC m=+2432.821255685" watchObservedRunningTime="2026-07-17 01:54:35.426526429 +0000 UTC m=+2432.822141279" Jul 17 01:54:36.403130 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:36.403082 2583 generic.go:356] "Generic (PLEG): container finished" podID="41916647-58d8-4e16-a5fb-24825b145130" containerID="8279827a4d95384f6001677ea3765cb534c7921f4853221b152ef2631adbaf44" exitCode=6 Jul 17 01:54:36.403575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:36.403158 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" event={"ID":"41916647-58d8-4e16-a5fb-24825b145130","Type":"ContainerDied","Data":"8279827a4d95384f6001677ea3765cb534c7921f4853221b152ef2631adbaf44"} Jul 17 01:54:36.403575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:36.403215 2583 scope.go:117] "RemoveContainer" containerID="bd7c1acb82655d2fa2536a4b219f7eba55a94e304fc142e0355651f7a9c3ae01" Jul 17 01:54:36.403575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:36.403399 2583 scope.go:117] "RemoveContainer" containerID="8279827a4d95384f6001677ea3765cb534c7921f4853221b152ef2631adbaf44" Jul 17 01:54:36.403762 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:54:36.403637 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b_odh-ai-gateway-infra(41916647-58d8-4e16-a5fb-24825b145130)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" podUID="41916647-58d8-4e16-a5fb-24825b145130" Jul 17 01:54:36.405157 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:36.405138 2583 generic.go:356] "Generic (PLEG): container finished" podID="eb58b0bf-df27-49de-998d-cba636947f8b" containerID="831eca11de72fcef40ef65963a034aa427df13fd7bbcfddaaf742ea5f3c7b5a9" exitCode=6 Jul 17 01:54:36.405260 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:36.405219 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" event={"ID":"eb58b0bf-df27-49de-998d-cba636947f8b","Type":"ContainerDied","Data":"831eca11de72fcef40ef65963a034aa427df13fd7bbcfddaaf742ea5f3c7b5a9"} Jul 17 01:54:36.405502 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:36.405488 2583 scope.go:117] "RemoveContainer" containerID="831eca11de72fcef40ef65963a034aa427df13fd7bbcfddaaf742ea5f3c7b5a9" Jul 17 01:54:36.405657 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:54:36.405640 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t_odh-ai-gateway-infra(eb58b0bf-df27-49de-998d-cba636947f8b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" podUID="eb58b0bf-df27-49de-998d-cba636947f8b" Jul 17 01:54:36.413904 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:36.413887 2583 scope.go:117] "RemoveContainer" containerID="314d317c3444c4d2880210087ad8ec0138b12640ac5c797fc05530f340b017ac" Jul 17 01:54:36.434743 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:36.434716 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t"] Jul 17 01:54:37.162296 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:37.162264 2583 scope.go:117] "RemoveContainer" containerID="3584531cd9946d56b4ba49a9bb5a6dd0be459d4ebb8c25a3b8108b3865c47c82" Jul 17 01:54:37.388070 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:37.388037 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:37.412231 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:37.412196 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" event={"ID":"c6cb41a7-22a8-4c9f-9f02-4901779f7eff","Type":"ContainerStarted","Data":"dc481ff3cc31e47d0ab3b15640d181057d312c2f4576adf9683045208ce2fa14"} Jul 17 01:54:37.413868 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:37.413805 2583 scope.go:117] "RemoveContainer" containerID="8279827a4d95384f6001677ea3765cb534c7921f4853221b152ef2631adbaf44" Jul 17 01:54:37.413995 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:54:37.413969 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b_odh-ai-gateway-infra(41916647-58d8-4e16-a5fb-24825b145130)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" podUID="41916647-58d8-4e16-a5fb-24825b145130" Jul 17 01:54:37.429261 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:37.429197 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" podStartSLOduration=20.429176259 podStartE2EDuration="20.429176259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:54:37.426855149 +0000 UTC m=+2434.822470001" watchObservedRunningTime="2026-07-17 01:54:37.429176259 +0000 UTC m=+2434.824791111" Jul 17 01:54:37.633976 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:37.633950 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" Jul 17 01:54:37.811429 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:37.811399 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb58b0bf-df27-49de-998d-cba636947f8b-maas-api-service-ca\") pod \"eb58b0bf-df27-49de-998d-cba636947f8b\" (UID: \"eb58b0bf-df27-49de-998d-cba636947f8b\") " Jul 17 01:54:37.811804 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:37.811777 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb58b0bf-df27-49de-998d-cba636947f8b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb58b0bf-df27-49de-998d-cba636947f8b" (UID: "eb58b0bf-df27-49de-998d-cba636947f8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:54:37.912841 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:37.912805 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb58b0bf-df27-49de-998d-cba636947f8b-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:54:38.224385 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:38.224293 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:38.421758 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:38.421721 2583 generic.go:356] "Generic (PLEG): container finished" podID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" containerID="dc481ff3cc31e47d0ab3b15640d181057d312c2f4576adf9683045208ce2fa14" exitCode=6 Jul 17 01:54:38.422246 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:38.421795 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" event={"ID":"c6cb41a7-22a8-4c9f-9f02-4901779f7eff","Type":"ContainerDied","Data":"dc481ff3cc31e47d0ab3b15640d181057d312c2f4576adf9683045208ce2fa14"} Jul 17 01:54:38.422246 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:38.421852 2583 scope.go:117] "RemoveContainer" containerID="3584531cd9946d56b4ba49a9bb5a6dd0be459d4ebb8c25a3b8108b3865c47c82" Jul 17 01:54:38.422246 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:38.422182 2583 scope.go:117] "RemoveContainer" containerID="dc481ff3cc31e47d0ab3b15640d181057d312c2f4576adf9683045208ce2fa14" Jul 17 01:54:38.422432 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:54:38.422407 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k_odh-ai-gateway-infra(c6cb41a7-22a8-4c9f-9f02-4901779f7eff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" podUID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" Jul 17 01:54:38.423134 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:38.423117 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" Jul 17 01:54:38.423195 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:38.423135 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t" event={"ID":"eb58b0bf-df27-49de-998d-cba636947f8b","Type":"ContainerDied","Data":"61d21f9891c54b5c129b6eeb6dc7bca498d6a3dff506de20a84cfbf9754ee0f1"} Jul 17 01:54:38.432899 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:38.432877 2583 scope.go:117] "RemoveContainer" containerID="831eca11de72fcef40ef65963a034aa427df13fd7bbcfddaaf742ea5f3c7b5a9" Jul 17 01:54:38.435157 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:38.435139 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k"] Jul 17 01:54:38.456997 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:38.456964 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t"] Jul 17 01:54:38.462674 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:38.462643 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-tf75t"] Jul 17 01:54:39.167638 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:39.167601 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb58b0bf-df27-49de-998d-cba636947f8b" path="/var/lib/kubelet/pods/eb58b0bf-df27-49de-998d-cba636947f8b/volumes" Jul 17 01:54:39.565001 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:39.564978 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" Jul 17 01:54:39.730260 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:39.730167 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6cb41a7-22a8-4c9f-9f02-4901779f7eff-maas-api-service-ca\") pod \"c6cb41a7-22a8-4c9f-9f02-4901779f7eff\" (UID: \"c6cb41a7-22a8-4c9f-9f02-4901779f7eff\") " Jul 17 01:54:39.730556 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:39.730534 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6cb41a7-22a8-4c9f-9f02-4901779f7eff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6cb41a7-22a8-4c9f-9f02-4901779f7eff" (UID: "c6cb41a7-22a8-4c9f-9f02-4901779f7eff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:54:39.830942 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:39.830903 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6cb41a7-22a8-4c9f-9f02-4901779f7eff-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:54:40.436258 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:40.436223 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" Jul 17 01:54:40.436258 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:40.436244 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k" event={"ID":"c6cb41a7-22a8-4c9f-9f02-4901779f7eff","Type":"ContainerDied","Data":"bf8e7874f7025047d7eb3c0eb958817bbbf2ecd1312cab98a52d92b792d9dd5d"} Jul 17 01:54:40.436499 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:40.436307 2583 scope.go:117] "RemoveContainer" containerID="dc481ff3cc31e47d0ab3b15640d181057d312c2f4576adf9683045208ce2fa14" Jul 17 01:54:40.460810 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:40.460782 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k"] Jul 17 01:54:40.464214 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:40.464193 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-5qg9k"] Jul 17 01:54:41.167616 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:41.167582 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" path="/var/lib/kubelet/pods/c6cb41a7-22a8-4c9f-9f02-4901779f7eff/volumes" Jul 17 01:54:43.986277 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:43.986241 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:44.053744 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:44.053713 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:44.126015 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:44.125981 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:44.257362 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:44.257260 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:44.338829 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:44.338771 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:45.064952 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:45.064895 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:45.157843 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:45.157793 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:45.256209 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:45.256182 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:46.835026 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:46.834984 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:49.163249 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:49.163214 2583 scope.go:117] "RemoveContainer" containerID="8279827a4d95384f6001677ea3765cb534c7921f4853221b152ef2631adbaf44" Jul 17 01:54:49.481397 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:49.481318 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" event={"ID":"41916647-58d8-4e16-a5fb-24825b145130","Type":"ContainerStarted","Data":"261bbd46d7f371fbe5fa9c8300f18c736eb3b145b96b6ebee8c1e7b330f12259"} Jul 17 01:54:50.487199 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:50.487161 2583 generic.go:356] "Generic (PLEG): container finished" podID="41916647-58d8-4e16-a5fb-24825b145130" containerID="261bbd46d7f371fbe5fa9c8300f18c736eb3b145b96b6ebee8c1e7b330f12259" exitCode=6 Jul 17 01:54:50.487618 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:50.487222 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" event={"ID":"41916647-58d8-4e16-a5fb-24825b145130","Type":"ContainerDied","Data":"261bbd46d7f371fbe5fa9c8300f18c736eb3b145b96b6ebee8c1e7b330f12259"} Jul 17 01:54:50.487618 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:50.487274 2583 scope.go:117] "RemoveContainer" containerID="8279827a4d95384f6001677ea3765cb534c7921f4853221b152ef2631adbaf44" Jul 17 01:54:50.487618 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:50.487585 2583 scope.go:117] "RemoveContainer" containerID="261bbd46d7f371fbe5fa9c8300f18c736eb3b145b96b6ebee8c1e7b330f12259" Jul 17 01:54:50.487800 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:54:50.487781 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b_odh-ai-gateway-infra(41916647-58d8-4e16-a5fb-24825b145130)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" podUID="41916647-58d8-4e16-a5fb-24825b145130" Jul 17 01:54:50.511524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:50.511495 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b"] Jul 17 01:54:51.628246 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:51.628215 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" Jul 17 01:54:51.724327 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:51.724287 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41916647-58d8-4e16-a5fb-24825b145130-maas-api-service-ca\") pod \"41916647-58d8-4e16-a5fb-24825b145130\" (UID: \"41916647-58d8-4e16-a5fb-24825b145130\") " Jul 17 01:54:51.724684 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:51.724663 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41916647-58d8-4e16-a5fb-24825b145130-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41916647-58d8-4e16-a5fb-24825b145130" (UID: "41916647-58d8-4e16-a5fb-24825b145130"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:54:51.825581 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:51.825548 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41916647-58d8-4e16-a5fb-24825b145130-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:54:52.498736 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:52.498700 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" event={"ID":"41916647-58d8-4e16-a5fb-24825b145130","Type":"ContainerDied","Data":"1822c2f172f1b56a052e0556608a6fe932c9ea8bf1ba62a0bc199f6e4e7e3155"} Jul 17 01:54:52.498943 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:52.498738 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b" Jul 17 01:54:52.498943 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:52.498778 2583 scope.go:117] "RemoveContainer" containerID="261bbd46d7f371fbe5fa9c8300f18c736eb3b145b96b6ebee8c1e7b330f12259" Jul 17 01:54:52.519637 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:52.519596 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b"] Jul 17 01:54:52.524995 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:52.524963 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-26q8b"] Jul 17 01:54:52.706122 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:52.706080 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:52.805376 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:52.805345 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:53.167748 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:53.167669 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41916647-58d8-4e16-a5fb-24825b145130" path="/var/lib/kubelet/pods/41916647-58d8-4e16-a5fb-24825b145130/volumes" Jul 17 01:54:53.375564 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:53.375538 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:59.236643 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:59.236592 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:59.294068 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:59.294038 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:59.468721 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:59.468688 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:54:59.788187 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:54:59.788145 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:00.096351 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:00.096266 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:02.564432 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.564397 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:02.664348 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.664315 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:02.773585 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.773556 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb"] Jul 17 01:55:02.773896 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.773884 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41916647-58d8-4e16-a5fb-24825b145130" containerName="revoke-keys" Jul 17 01:55:02.773939 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.773898 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="41916647-58d8-4e16-a5fb-24825b145130" containerName="revoke-keys" Jul 17 01:55:02.773939 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.773906 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" containerName="revoke-keys" Jul 17 01:55:02.773939 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.773912 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" containerName="revoke-keys" Jul 17 01:55:02.773939 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.773919 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb58b0bf-df27-49de-998d-cba636947f8b" containerName="revoke-keys" Jul 17 01:55:02.773939 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.773924 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb58b0bf-df27-49de-998d-cba636947f8b" containerName="revoke-keys" Jul 17 01:55:02.773939 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.773934 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41916647-58d8-4e16-a5fb-24825b145130" containerName="revoke-keys" Jul 17 01:55:02.773939 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.773939 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="41916647-58d8-4e16-a5fb-24825b145130" containerName="revoke-keys" Jul 17 01:55:02.774173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.773951 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb58b0bf-df27-49de-998d-cba636947f8b" containerName="revoke-keys" Jul 17 01:55:02.774173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.773956 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb58b0bf-df27-49de-998d-cba636947f8b" containerName="revoke-keys" Jul 17 01:55:02.774173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.773967 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" containerName="revoke-keys" Jul 17 01:55:02.774173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.773972 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" containerName="revoke-keys" Jul 17 01:55:02.774173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.774032 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb58b0bf-df27-49de-998d-cba636947f8b" containerName="revoke-keys" Jul 17 01:55:02.774173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.774042 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="41916647-58d8-4e16-a5fb-24825b145130" containerName="revoke-keys" Jul 17 01:55:02.774173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.774047 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" containerName="revoke-keys" Jul 17 01:55:02.774173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.774053 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="41916647-58d8-4e16-a5fb-24825b145130" containerName="revoke-keys" Jul 17 01:55:02.774173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.774059 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" containerName="revoke-keys" Jul 17 01:55:02.774173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.774065 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="41916647-58d8-4e16-a5fb-24825b145130" containerName="revoke-keys" Jul 17 01:55:02.774173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.774070 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb58b0bf-df27-49de-998d-cba636947f8b" containerName="revoke-keys" Jul 17 01:55:02.774173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.774075 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" containerName="revoke-keys" Jul 17 01:55:02.778473 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.778456 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:02.780713 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.780691 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5l8p8\"" Jul 17 01:55:02.780877 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.780853 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ac7a9695-serving-cert\"" Jul 17 01:55:02.780951 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.780931 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:55:02.781524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.781507 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 01:55:02.787434 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.787412 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb"] Jul 17 01:55:02.806216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.806188 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:02.822414 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.822355 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-maas-api-tls\") pod \"maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb\" (UID: \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:02.822414 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.822402 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c88gx\" (UniqueName: \"kubernetes.io/projected/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-kube-api-access-c88gx\") pod \"maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb\" (UID: \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:02.822528 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.822463 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-openshift-service-ca\") pod \"maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb\" (UID: \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:02.923823 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.923795 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-openshift-service-ca\") pod \"maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb\" (UID: \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:02.923969 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.923847 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-maas-api-tls\") pod \"maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb\" (UID: \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:02.923969 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.923886 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c88gx\" (UniqueName: \"kubernetes.io/projected/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-kube-api-access-c88gx\") pod \"maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb\" (UID: \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:02.924084 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:55:02.924018 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-serving-cert: secret "maas-api-e2e-mt-ac7a9695-serving-cert" not found Jul 17 01:55:02.924182 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:55:02.924131 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-maas-api-tls podName:f8d8859b-85c7-4a34-b4f6-9c36e9f489bf nodeName:}" failed. No retries permitted until 2026-07-17 01:55:03.424080318 +0000 UTC m=+2460.819695159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-maas-api-tls") pod "maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" (UID: "f8d8859b-85c7-4a34-b4f6-9c36e9f489bf") : secret "maas-api-e2e-mt-ac7a9695-serving-cert" not found Jul 17 01:55:02.924400 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.924383 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-openshift-service-ca\") pod \"maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb\" (UID: \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:02.934213 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:02.934187 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c88gx\" (UniqueName: \"kubernetes.io/projected/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-kube-api-access-c88gx\") pod \"maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb\" (UID: \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:03.223761 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:03.223669 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:03.338754 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:03.338722 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:03.429188 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:03.429153 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-maas-api-tls\") pod \"maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb\" (UID: \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:03.429415 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:55:03.429301 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-serving-cert: secret "maas-api-e2e-mt-ac7a9695-serving-cert" not found Jul 17 01:55:03.429415 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:55:03.429365 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-maas-api-tls podName:f8d8859b-85c7-4a34-b4f6-9c36e9f489bf nodeName:}" failed. No retries permitted until 2026-07-17 01:55:04.429350084 +0000 UTC m=+2461.824964913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-maas-api-tls") pod "maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" (UID: "f8d8859b-85c7-4a34-b4f6-9c36e9f489bf") : secret "maas-api-e2e-mt-ac7a9695-serving-cert" not found Jul 17 01:55:03.493800 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:03.493716 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:04.437634 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:04.437597 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-maas-api-tls\") pod \"maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb\" (UID: \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:04.440128 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:04.440110 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-maas-api-tls\") pod \"maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb\" (UID: \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:04.592492 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:04.592456 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5l8p8\"" Jul 17 01:55:04.600278 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:04.600248 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:04.739450 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:04.739279 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb"] Jul 17 01:55:04.742174 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:55:04.742143 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d8859b_85c7_4a34_b4f6_9c36e9f489bf.slice/crio-e16747a887f2e0d7fc0c38e42811dbbc8ea83b5756932491f4542fe534770142 WatchSource:0}: Error finding container e16747a887f2e0d7fc0c38e42811dbbc8ea83b5756932491f4542fe534770142: Status 404 returned error can't find the container with id e16747a887f2e0d7fc0c38e42811dbbc8ea83b5756932491f4542fe534770142 Jul 17 01:55:04.743933 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:04.743916 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 01:55:05.559819 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:05.559779 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" event={"ID":"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf","Type":"ContainerStarted","Data":"7b2dc238d834a4e200ae7200cf50a7b2b15b45c8cf1e6e28903666f05e8e8712"} Jul 17 01:55:05.560252 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:05.559844 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" event={"ID":"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf","Type":"ContainerStarted","Data":"e16747a887f2e0d7fc0c38e42811dbbc8ea83b5756932491f4542fe534770142"} Jul 17 01:55:05.560252 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:05.559958 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:05.574246 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:05.574197 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" podStartSLOduration=3.065530938 podStartE2EDuration="3.574181915s" totalImagesPullingTime="508.650977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:55:05.573240434 +0000 UTC m=+2462.968855294" watchObservedRunningTime="2026-07-17 01:55:05.574181915 +0000 UTC m=+2462.969796766" Jul 17 01:55:06.987620 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:06.987588 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:07.621768 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:07.621732 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:07.722695 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:07.722661 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:08.037823 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:08.037794 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:11.570151 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:11.570120 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:13.744032 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:13.743998 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:13.805848 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:13.805813 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:14.098513 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.098472 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh"] Jul 17 01:55:14.098872 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.098858 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" containerName="revoke-keys" Jul 17 01:55:14.098919 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.098876 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6cb41a7-22a8-4c9f-9f02-4901779f7eff" containerName="revoke-keys" Jul 17 01:55:14.098919 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.098889 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb58b0bf-df27-49de-998d-cba636947f8b" containerName="revoke-keys" Jul 17 01:55:14.098919 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.098895 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb58b0bf-df27-49de-998d-cba636947f8b" containerName="revoke-keys" Jul 17 01:55:14.098919 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.098905 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41916647-58d8-4e16-a5fb-24825b145130" containerName="revoke-keys" Jul 17 01:55:14.098919 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.098912 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="41916647-58d8-4e16-a5fb-24825b145130" containerName="revoke-keys" Jul 17 01:55:14.099072 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.098978 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb58b0bf-df27-49de-998d-cba636947f8b" containerName="revoke-keys" Jul 17 01:55:14.102012 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.101992 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh" Jul 17 01:55:14.104355 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.104326 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:55:14.107857 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.107822 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh"] Jul 17 01:55:14.232308 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.232263 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh\" (UID: \"d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh" Jul 17 01:55:14.333168 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.333128 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh\" (UID: \"d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh" Jul 17 01:55:14.333786 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.333763 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh\" (UID: \"d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh" Jul 17 01:55:14.413441 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.413356 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh" Jul 17 01:55:14.544336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.544138 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh"] Jul 17 01:55:14.547037 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:55:14.547011 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79d5de9_ccdb_4365_a5b7_9cb8f21f53c8.slice/crio-e5cd17c90fac3810af366def2680ab6adb3d99cd411e25413b73fdd093b5071c WatchSource:0}: Error finding container e5cd17c90fac3810af366def2680ab6adb3d99cd411e25413b73fdd093b5071c: Status 404 returned error can't find the container with id e5cd17c90fac3810af366def2680ab6adb3d99cd411e25413b73fdd093b5071c Jul 17 01:55:14.601797 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:14.601767 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh" event={"ID":"d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8","Type":"ContainerStarted","Data":"e5cd17c90fac3810af366def2680ab6adb3d99cd411e25413b73fdd093b5071c"} Jul 17 01:55:15.211643 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:15.211605 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:15.312652 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:15.312621 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:15.607360 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:15.607329 2583 generic.go:356] "Generic (PLEG): container finished" podID="d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8" containerID="3630e903728be0b2cf8faeb9e860049420c166a4ad72ee56a98dac66aeb496a5" exitCode=0 Jul 17 01:55:15.607529 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:15.607422 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh" event={"ID":"d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8","Type":"ContainerDied","Data":"3630e903728be0b2cf8faeb9e860049420c166a4ad72ee56a98dac66aeb496a5"} Jul 17 01:55:16.737145 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:16.737119 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh" Jul 17 01:55:16.856127 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:16.856079 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8-maas-api-service-ca\") pod \"d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8\" (UID: \"d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8\") " Jul 17 01:55:16.856431 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:16.856406 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8" (UID: "d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:55:16.957149 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:16.957050 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:55:17.617625 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:17.617593 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh" Jul 17 01:55:17.617625 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:17.617603 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh" event={"ID":"d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8","Type":"ContainerDied","Data":"e5cd17c90fac3810af366def2680ab6adb3d99cd411e25413b73fdd093b5071c"} Jul 17 01:55:17.617847 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:17.617646 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5cd17c90fac3810af366def2680ab6adb3d99cd411e25413b73fdd093b5071c" Jul 17 01:55:23.443622 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:23.443589 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk"] Jul 17 01:55:23.444012 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:23.443912 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8" containerName="revoke-keys" Jul 17 01:55:23.444012 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:23.443924 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8" containerName="revoke-keys" Jul 17 01:55:23.444012 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:23.443986 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8" containerName="revoke-keys" Jul 17 01:55:23.446936 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:23.446919 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" Jul 17 01:55:23.449160 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:23.449143 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:55:23.454342 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:23.454318 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk"] Jul 17 01:55:23.611619 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:23.611578 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f08f68b-48ab-4628-b9d0-190e83608d24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk\" (UID: \"4f08f68b-48ab-4628-b9d0-190e83608d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" Jul 17 01:55:23.712197 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:23.712086 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f08f68b-48ab-4628-b9d0-190e83608d24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk\" (UID: \"4f08f68b-48ab-4628-b9d0-190e83608d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" Jul 17 01:55:23.712797 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:23.712770 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f08f68b-48ab-4628-b9d0-190e83608d24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk\" (UID: \"4f08f68b-48ab-4628-b9d0-190e83608d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" Jul 17 01:55:23.758172 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:23.758154 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" Jul 17 01:55:24.093177 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.093150 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk"] Jul 17 01:55:24.096139 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:55:24.096112 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f08f68b_48ab_4628_b9d0_190e83608d24.slice/crio-1ef1e090df665027c4d03bc798d3ee76d04c8604598f341aa533ff05eb5950ae WatchSource:0}: Error finding container 1ef1e090df665027c4d03bc798d3ee76d04c8604598f341aa533ff05eb5950ae: Status 404 returned error can't find the container with id 1ef1e090df665027c4d03bc798d3ee76d04c8604598f341aa533ff05eb5950ae Jul 17 01:55:24.461795 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.461761 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb"] Jul 17 01:55:24.462269 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.461978 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" podUID="f8d8859b-85c7-4a34-b4f6-9c36e9f489bf" containerName="maas-api" containerID="cri-o://7b2dc238d834a4e200ae7200cf50a7b2b15b45c8cf1e6e28903666f05e8e8712" gracePeriod=30 Jul 17 01:55:24.482176 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.482123 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:24.650855 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.650801 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" event={"ID":"4f08f68b-48ab-4628-b9d0-190e83608d24","Type":"ContainerStarted","Data":"aa2296dc11c460b76df2ae7dbf51f0df9998b1ce28d98124115a90bc3400be3b"} Jul 17 01:55:24.651019 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.650877 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" event={"ID":"4f08f68b-48ab-4628-b9d0-190e83608d24","Type":"ContainerStarted","Data":"1ef1e090df665027c4d03bc798d3ee76d04c8604598f341aa533ff05eb5950ae"} Jul 17 01:55:24.652440 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.652414 2583 generic.go:356] "Generic (PLEG): container finished" podID="f8d8859b-85c7-4a34-b4f6-9c36e9f489bf" containerID="7b2dc238d834a4e200ae7200cf50a7b2b15b45c8cf1e6e28903666f05e8e8712" exitCode=0 Jul 17 01:55:24.652544 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.652481 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" event={"ID":"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf","Type":"ContainerDied","Data":"7b2dc238d834a4e200ae7200cf50a7b2b15b45c8cf1e6e28903666f05e8e8712"} Jul 17 01:55:24.666007 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.665952 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" podStartSLOduration=1.665931242 podStartE2EDuration="1.665931242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:55:24.664116149 +0000 UTC m=+2482.059731000" watchObservedRunningTime="2026-07-17 01:55:24.665931242 +0000 UTC m=+2482.061546099" Jul 17 01:55:24.706335 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.706313 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:24.822070 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.822043 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-openshift-service-ca\") pod \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\" (UID: \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\") " Jul 17 01:55:24.822175 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.822153 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-maas-api-tls\") pod \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\" (UID: \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\") " Jul 17 01:55:24.822314 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.822286 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c88gx\" (UniqueName: \"kubernetes.io/projected/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-kube-api-access-c88gx\") pod \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\" (UID: \"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf\") " Jul 17 01:55:24.822539 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.822502 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f8d8859b-85c7-4a34-b4f6-9c36e9f489bf" (UID: "f8d8859b-85c7-4a34-b4f6-9c36e9f489bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:55:24.822735 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.822714 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-openshift-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:55:24.824443 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.824413 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f8d8859b-85c7-4a34-b4f6-9c36e9f489bf" (UID: "f8d8859b-85c7-4a34-b4f6-9c36e9f489bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:55:24.824828 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.824806 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-kube-api-access-c88gx" (OuterVolumeSpecName: "kube-api-access-c88gx") pod "f8d8859b-85c7-4a34-b4f6-9c36e9f489bf" (UID: "f8d8859b-85c7-4a34-b4f6-9c36e9f489bf"). InnerVolumeSpecName "kube-api-access-c88gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:55:24.924254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.924161 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-maas-api-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:55:24.924254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:24.924193 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c88gx\" (UniqueName: \"kubernetes.io/projected/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf-kube-api-access-c88gx\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:55:25.659916 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:25.659881 2583 generic.go:356] "Generic (PLEG): container finished" podID="4f08f68b-48ab-4628-b9d0-190e83608d24" containerID="aa2296dc11c460b76df2ae7dbf51f0df9998b1ce28d98124115a90bc3400be3b" exitCode=6 Jul 17 01:55:25.660437 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:25.659974 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" event={"ID":"4f08f68b-48ab-4628-b9d0-190e83608d24","Type":"ContainerDied","Data":"aa2296dc11c460b76df2ae7dbf51f0df9998b1ce28d98124115a90bc3400be3b"} Jul 17 01:55:25.660437 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:25.660199 2583 scope.go:117] "RemoveContainer" containerID="aa2296dc11c460b76df2ae7dbf51f0df9998b1ce28d98124115a90bc3400be3b" Jul 17 01:55:25.661407 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:25.661389 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" Jul 17 01:55:25.661498 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:25.661386 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb" event={"ID":"f8d8859b-85c7-4a34-b4f6-9c36e9f489bf","Type":"ContainerDied","Data":"e16747a887f2e0d7fc0c38e42811dbbc8ea83b5756932491f4542fe534770142"} Jul 17 01:55:25.661535 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:25.661527 2583 scope.go:117] "RemoveContainer" containerID="7b2dc238d834a4e200ae7200cf50a7b2b15b45c8cf1e6e28903666f05e8e8712" Jul 17 01:55:25.687685 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:25.687648 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb"] Jul 17 01:55:25.689816 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:25.689793 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ac7a9695-6dbbc788b6-7rdzb"] Jul 17 01:55:26.668487 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:26.668452 2583 generic.go:356] "Generic (PLEG): container finished" podID="4f08f68b-48ab-4628-b9d0-190e83608d24" containerID="a1bbdbe30c7d9bbf2564765da5afc702f44a55e3d4345f959eaeedc2fec1aad0" exitCode=6 Jul 17 01:55:26.668936 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:26.668536 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" event={"ID":"4f08f68b-48ab-4628-b9d0-190e83608d24","Type":"ContainerDied","Data":"a1bbdbe30c7d9bbf2564765da5afc702f44a55e3d4345f959eaeedc2fec1aad0"} Jul 17 01:55:26.668936 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:26.668592 2583 scope.go:117] "RemoveContainer" containerID="aa2296dc11c460b76df2ae7dbf51f0df9998b1ce28d98124115a90bc3400be3b" Jul 17 01:55:26.668936 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:26.668728 2583 scope.go:117] "RemoveContainer" containerID="a1bbdbe30c7d9bbf2564765da5afc702f44a55e3d4345f959eaeedc2fec1aad0" Jul 17 01:55:26.669089 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:55:26.668944 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk_odh-ai-gateway-infra(4f08f68b-48ab-4628-b9d0-190e83608d24)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" podUID="4f08f68b-48ab-4628-b9d0-190e83608d24" Jul 17 01:55:27.168669 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:27.168634 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8d8859b-85c7-4a34-b4f6-9c36e9f489bf" path="/var/lib/kubelet/pods/f8d8859b-85c7-4a34-b4f6-9c36e9f489bf/volumes" Jul 17 01:55:27.676420 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:27.676391 2583 scope.go:117] "RemoveContainer" containerID="a1bbdbe30c7d9bbf2564765da5afc702f44a55e3d4345f959eaeedc2fec1aad0" Jul 17 01:55:27.676816 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:55:27.676572 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk_odh-ai-gateway-infra(4f08f68b-48ab-4628-b9d0-190e83608d24)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" podUID="4f08f68b-48ab-4628-b9d0-190e83608d24" Jul 17 01:55:29.176752 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:29.176714 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:34.815541 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:34.815504 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:34.882527 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:34.882492 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:36.036724 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:36.036691 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq"] Jul 17 01:55:36.037217 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:36.037187 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d8859b-85c7-4a34-b4f6-9c36e9f489bf" containerName="maas-api" Jul 17 01:55:36.037217 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:36.037206 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d8859b-85c7-4a34-b4f6-9c36e9f489bf" containerName="maas-api" Jul 17 01:55:36.037336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:36.037313 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d8859b-85c7-4a34-b4f6-9c36e9f489bf" containerName="maas-api" Jul 17 01:55:36.041986 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:36.041964 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" Jul 17 01:55:36.046116 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:36.046069 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq"] Jul 17 01:55:36.124682 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:36.124652 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecbdd44-776c-41f1-8ad4-e42cb60c877e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq\" (UID: \"1ecbdd44-776c-41f1-8ad4-e42cb60c877e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" Jul 17 01:55:36.225609 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:36.225574 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecbdd44-776c-41f1-8ad4-e42cb60c877e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq\" (UID: \"1ecbdd44-776c-41f1-8ad4-e42cb60c877e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" Jul 17 01:55:36.226186 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:36.226167 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecbdd44-776c-41f1-8ad4-e42cb60c877e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq\" (UID: \"1ecbdd44-776c-41f1-8ad4-e42cb60c877e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" Jul 17 01:55:36.352753 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:36.352663 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" Jul 17 01:55:36.480159 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:36.479955 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq"] Jul 17 01:55:36.482907 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:55:36.482878 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecbdd44_776c_41f1_8ad4_e42cb60c877e.slice/crio-3a974592875cdbf4ccc65e234aa4f985d1cff96d55d0af5ce419f3d6c96b5848 WatchSource:0}: Error finding container 3a974592875cdbf4ccc65e234aa4f985d1cff96d55d0af5ce419f3d6c96b5848: Status 404 returned error can't find the container with id 3a974592875cdbf4ccc65e234aa4f985d1cff96d55d0af5ce419f3d6c96b5848 Jul 17 01:55:36.723509 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:36.723470 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" event={"ID":"1ecbdd44-776c-41f1-8ad4-e42cb60c877e","Type":"ContainerStarted","Data":"3a974592875cdbf4ccc65e234aa4f985d1cff96d55d0af5ce419f3d6c96b5848"} Jul 17 01:55:37.729527 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:37.729492 2583 generic.go:356] "Generic (PLEG): container finished" podID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" containerID="80ca92c0241983f0507635cefaad16297f58d7b1baa3adb97d57b58957a72a1b" exitCode=6 Jul 17 01:55:37.729922 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:37.729550 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" event={"ID":"1ecbdd44-776c-41f1-8ad4-e42cb60c877e","Type":"ContainerDied","Data":"80ca92c0241983f0507635cefaad16297f58d7b1baa3adb97d57b58957a72a1b"} Jul 17 01:55:37.729922 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:37.729854 2583 scope.go:117] "RemoveContainer" containerID="80ca92c0241983f0507635cefaad16297f58d7b1baa3adb97d57b58957a72a1b" Jul 17 01:55:38.736935 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:38.736841 2583 generic.go:356] "Generic (PLEG): container finished" podID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" containerID="c2e20a20adb7287f86da62f6b87b4c6dece6fa1be9286283497c7b861bdeaf8a" exitCode=6 Jul 17 01:55:38.736935 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:38.736882 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" event={"ID":"1ecbdd44-776c-41f1-8ad4-e42cb60c877e","Type":"ContainerDied","Data":"c2e20a20adb7287f86da62f6b87b4c6dece6fa1be9286283497c7b861bdeaf8a"} Jul 17 01:55:38.736935 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:38.736928 2583 scope.go:117] "RemoveContainer" containerID="80ca92c0241983f0507635cefaad16297f58d7b1baa3adb97d57b58957a72a1b" Jul 17 01:55:38.737512 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:38.737170 2583 scope.go:117] "RemoveContainer" containerID="c2e20a20adb7287f86da62f6b87b4c6dece6fa1be9286283497c7b861bdeaf8a" Jul 17 01:55:38.737512 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:55:38.737389 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq_odh-ai-gateway-infra(1ecbdd44-776c-41f1-8ad4-e42cb60c877e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" podUID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" Jul 17 01:55:39.743110 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:39.743066 2583 scope.go:117] "RemoveContainer" containerID="c2e20a20adb7287f86da62f6b87b4c6dece6fa1be9286283497c7b861bdeaf8a" Jul 17 01:55:39.743503 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:55:39.743282 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq_odh-ai-gateway-infra(1ecbdd44-776c-41f1-8ad4-e42cb60c877e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" podUID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" Jul 17 01:55:39.858389 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:39.858352 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:40.095774 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:40.095738 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:40.244039 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:40.244008 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:40.332252 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:40.332223 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:40.449433 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:40.449322 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:42.162168 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:42.162140 2583 scope.go:117] "RemoveContainer" containerID="a1bbdbe30c7d9bbf2564765da5afc702f44a55e3d4345f959eaeedc2fec1aad0" Jul 17 01:55:42.758126 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:42.758072 2583 generic.go:356] "Generic (PLEG): container finished" podID="4f08f68b-48ab-4628-b9d0-190e83608d24" containerID="52295f5a507ed8a7b4d30e1676c1743a2a1439300835b0d7398bbe17b0ed87b1" exitCode=6 Jul 17 01:55:42.758285 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:42.758129 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" event={"ID":"4f08f68b-48ab-4628-b9d0-190e83608d24","Type":"ContainerDied","Data":"52295f5a507ed8a7b4d30e1676c1743a2a1439300835b0d7398bbe17b0ed87b1"} Jul 17 01:55:42.758285 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:42.758194 2583 scope.go:117] "RemoveContainer" containerID="a1bbdbe30c7d9bbf2564765da5afc702f44a55e3d4345f959eaeedc2fec1aad0" Jul 17 01:55:42.758518 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:42.758497 2583 scope.go:117] "RemoveContainer" containerID="52295f5a507ed8a7b4d30e1676c1743a2a1439300835b0d7398bbe17b0ed87b1" Jul 17 01:55:42.758734 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:55:42.758714 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk_odh-ai-gateway-infra(4f08f68b-48ab-4628-b9d0-190e83608d24)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" podUID="4f08f68b-48ab-4628-b9d0-190e83608d24" Jul 17 01:55:43.783051 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:43.783016 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk"] Jul 17 01:55:43.912839 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:43.912808 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" Jul 17 01:55:43.989741 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:43.989706 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f08f68b-48ab-4628-b9d0-190e83608d24-maas-api-service-ca\") pod \"4f08f68b-48ab-4628-b9d0-190e83608d24\" (UID: \"4f08f68b-48ab-4628-b9d0-190e83608d24\") " Jul 17 01:55:43.990073 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:43.990050 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f08f68b-48ab-4628-b9d0-190e83608d24-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4f08f68b-48ab-4628-b9d0-190e83608d24" (UID: "4f08f68b-48ab-4628-b9d0-190e83608d24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:55:44.091187 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:44.091079 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f08f68b-48ab-4628-b9d0-190e83608d24-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:55:44.769550 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:44.769523 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" event={"ID":"4f08f68b-48ab-4628-b9d0-190e83608d24","Type":"ContainerDied","Data":"1ef1e090df665027c4d03bc798d3ee76d04c8604598f341aa533ff05eb5950ae"} Jul 17 01:55:44.769733 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:44.769583 2583 scope.go:117] "RemoveContainer" containerID="52295f5a507ed8a7b4d30e1676c1743a2a1439300835b0d7398bbe17b0ed87b1" Jul 17 01:55:44.769733 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:44.769533 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk" Jul 17 01:55:44.789668 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:44.789640 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk"] Jul 17 01:55:44.793067 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:44.793044 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-xctqk"] Jul 17 01:55:45.167951 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:45.167921 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f08f68b-48ab-4628-b9d0-190e83608d24" path="/var/lib/kubelet/pods/4f08f68b-48ab-4628-b9d0-190e83608d24/volumes" Jul 17 01:55:46.153963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:46.153933 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:46.213840 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:46.213810 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:46.294408 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:46.294375 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:46.392611 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:46.392558 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:46.502695 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:46.502584 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:51.163178 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:51.163141 2583 scope.go:117] "RemoveContainer" containerID="c2e20a20adb7287f86da62f6b87b4c6dece6fa1be9286283497c7b861bdeaf8a" Jul 17 01:55:51.802467 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:51.802434 2583 generic.go:356] "Generic (PLEG): container finished" podID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" containerID="fcbcacc5b7d06cb5e201be90434b00e7f915399fba2dc1059765e3aac39189a7" exitCode=6 Jul 17 01:55:51.802672 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:51.802511 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" event={"ID":"1ecbdd44-776c-41f1-8ad4-e42cb60c877e","Type":"ContainerDied","Data":"fcbcacc5b7d06cb5e201be90434b00e7f915399fba2dc1059765e3aac39189a7"} Jul 17 01:55:51.802672 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:51.802589 2583 scope.go:117] "RemoveContainer" containerID="c2e20a20adb7287f86da62f6b87b4c6dece6fa1be9286283497c7b861bdeaf8a" Jul 17 01:55:51.802923 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:51.802910 2583 scope.go:117] "RemoveContainer" containerID="fcbcacc5b7d06cb5e201be90434b00e7f915399fba2dc1059765e3aac39189a7" Jul 17 01:55:51.803197 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:55:51.803173 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq_odh-ai-gateway-infra(1ecbdd44-776c-41f1-8ad4-e42cb60c877e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" podUID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" Jul 17 01:55:52.109764 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:52.109701 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:52.164347 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:52.164287 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:52.827970 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:52.827931 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq"] Jul 17 01:55:52.957790 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:52.957765 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" Jul 17 01:55:53.073835 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:53.073806 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecbdd44-776c-41f1-8ad4-e42cb60c877e-maas-api-service-ca\") pod \"1ecbdd44-776c-41f1-8ad4-e42cb60c877e\" (UID: \"1ecbdd44-776c-41f1-8ad4-e42cb60c877e\") " Jul 17 01:55:53.074281 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:53.074242 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecbdd44-776c-41f1-8ad4-e42cb60c877e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ecbdd44-776c-41f1-8ad4-e42cb60c877e" (UID: "1ecbdd44-776c-41f1-8ad4-e42cb60c877e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:55:53.174336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:53.174308 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecbdd44-776c-41f1-8ad4-e42cb60c877e-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:55:53.814242 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:53.814210 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" event={"ID":"1ecbdd44-776c-41f1-8ad4-e42cb60c877e","Type":"ContainerDied","Data":"3a974592875cdbf4ccc65e234aa4f985d1cff96d55d0af5ce419f3d6c96b5848"} Jul 17 01:55:53.814424 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:53.814273 2583 scope.go:117] "RemoveContainer" containerID="fcbcacc5b7d06cb5e201be90434b00e7f915399fba2dc1059765e3aac39189a7" Jul 17 01:55:53.814424 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:53.814218 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq" Jul 17 01:55:53.832129 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:53.832088 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq"] Jul 17 01:55:53.837758 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:53.837734 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9whgq"] Jul 17 01:55:55.168163 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:55.168120 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" path="/var/lib/kubelet/pods/1ecbdd44-776c-41f1-8ad4-e42cb60c877e/volumes" Jul 17 01:55:58.877661 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:58.877629 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:59.188425 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.188323 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:59.365076 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.365042 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:55:59.987813 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.987776 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv"] Jul 17 01:55:59.988216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.988076 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" containerName="revoke-keys" Jul 17 01:55:59.988216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.988087 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" containerName="revoke-keys" Jul 17 01:55:59.988216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.988112 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" containerName="revoke-keys" Jul 17 01:55:59.988216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.988118 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" containerName="revoke-keys" Jul 17 01:55:59.988216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.988126 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f08f68b-48ab-4628-b9d0-190e83608d24" containerName="revoke-keys" Jul 17 01:55:59.988216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.988131 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f08f68b-48ab-4628-b9d0-190e83608d24" containerName="revoke-keys" Jul 17 01:55:59.988216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.988146 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f08f68b-48ab-4628-b9d0-190e83608d24" containerName="revoke-keys" Jul 17 01:55:59.988216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.988151 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f08f68b-48ab-4628-b9d0-190e83608d24" containerName="revoke-keys" Jul 17 01:55:59.988216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.988206 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f08f68b-48ab-4628-b9d0-190e83608d24" containerName="revoke-keys" Jul 17 01:55:59.988216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.988214 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f08f68b-48ab-4628-b9d0-190e83608d24" containerName="revoke-keys" Jul 17 01:55:59.988216 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.988219 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" containerName="revoke-keys" Jul 17 01:55:59.988586 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.988225 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" containerName="revoke-keys" Jul 17 01:55:59.992561 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.992544 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" Jul 17 01:55:59.995403 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.995379 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:55:59.995521 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.995466 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:55:59.999797 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:55:59.999775 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv"] Jul 17 01:56:00.140989 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:00.140949 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcff2e9b-f323-4c34-a2d5-097e432dffc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv\" (UID: \"fcff2e9b-f323-4c34-a2d5-097e432dffc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" Jul 17 01:56:00.241918 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:00.241822 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcff2e9b-f323-4c34-a2d5-097e432dffc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv\" (UID: \"fcff2e9b-f323-4c34-a2d5-097e432dffc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" Jul 17 01:56:00.242522 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:00.242504 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcff2e9b-f323-4c34-a2d5-097e432dffc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv\" (UID: \"fcff2e9b-f323-4c34-a2d5-097e432dffc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" Jul 17 01:56:00.304926 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:00.304875 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" Jul 17 01:56:00.454172 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:00.454134 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv"] Jul 17 01:56:00.456119 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:56:00.456078 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcff2e9b_f323_4c34_a2d5_097e432dffc5.slice/crio-61bc7efa42dca9b351dca3be6fbd93808549ae08e2edfaefc0bd01570a27f8d1 WatchSource:0}: Error finding container 61bc7efa42dca9b351dca3be6fbd93808549ae08e2edfaefc0bd01570a27f8d1: Status 404 returned error can't find the container with id 61bc7efa42dca9b351dca3be6fbd93808549ae08e2edfaefc0bd01570a27f8d1 Jul 17 01:56:00.851009 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:00.850972 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" event={"ID":"fcff2e9b-f323-4c34-a2d5-097e432dffc5","Type":"ContainerStarted","Data":"8dbe5e9f113f765daa53c09e1b593f3a852f9cdf4dba87faab71a5244bb74997"} Jul 17 01:56:00.851183 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:00.851026 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" event={"ID":"fcff2e9b-f323-4c34-a2d5-097e432dffc5","Type":"ContainerStarted","Data":"61bc7efa42dca9b351dca3be6fbd93808549ae08e2edfaefc0bd01570a27f8d1"} Jul 17 01:56:00.865633 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:00.865574 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" podStartSLOduration=1.865556255 podStartE2EDuration="1.865556255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:56:00.864312552 +0000 UTC m=+2518.259927406" watchObservedRunningTime="2026-07-17 01:56:00.865556255 +0000 UTC m=+2518.261171105" Jul 17 01:56:01.737330 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.737285 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:56:01.822853 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.822814 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:56:01.823681 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.823652 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg"] Jul 17 01:56:01.824173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.824155 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f08f68b-48ab-4628-b9d0-190e83608d24" containerName="revoke-keys" Jul 17 01:56:01.824229 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.824177 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f08f68b-48ab-4628-b9d0-190e83608d24" containerName="revoke-keys" Jul 17 01:56:01.824229 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.824210 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" containerName="revoke-keys" Jul 17 01:56:01.824229 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.824220 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" containerName="revoke-keys" Jul 17 01:56:01.824384 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.824302 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f08f68b-48ab-4628-b9d0-190e83608d24" containerName="revoke-keys" Jul 17 01:56:01.824384 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.824316 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ecbdd44-776c-41f1-8ad4-e42cb60c877e" containerName="revoke-keys" Jul 17 01:56:01.827563 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.827547 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:01.829625 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.829602 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8c5b7a14-serving-cert\"" Jul 17 01:56:01.829726 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.829687 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5l8p8\"" Jul 17 01:56:01.829771 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.829696 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 01:56:01.837138 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.837116 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg"] Jul 17 01:56:01.857509 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.857480 2583 generic.go:356] "Generic (PLEG): container finished" podID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" containerID="8dbe5e9f113f765daa53c09e1b593f3a852f9cdf4dba87faab71a5244bb74997" exitCode=6 Jul 17 01:56:01.857642 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.857583 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" event={"ID":"fcff2e9b-f323-4c34-a2d5-097e432dffc5","Type":"ContainerDied","Data":"8dbe5e9f113f765daa53c09e1b593f3a852f9cdf4dba87faab71a5244bb74997"} Jul 17 01:56:01.857724 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.857713 2583 scope.go:117] "RemoveContainer" containerID="8dbe5e9f113f765daa53c09e1b593f3a852f9cdf4dba87faab71a5244bb74997" Jul 17 01:56:01.920308 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.920278 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:56:01.958182 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.958157 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5zb9\" (UniqueName: \"kubernetes.io/projected/12a90c1a-667d-4e62-8de3-c65c12668dad-kube-api-access-h5zb9\") pod \"maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg\" (UID: \"12a90c1a-667d-4e62-8de3-c65c12668dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:01.958342 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.958210 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a90c1a-667d-4e62-8de3-c65c12668dad-maas-api-tls\") pod \"maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg\" (UID: \"12a90c1a-667d-4e62-8de3-c65c12668dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:01.958342 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:01.958235 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a90c1a-667d-4e62-8de3-c65c12668dad-openshift-service-ca\") pod \"maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg\" (UID: \"12a90c1a-667d-4e62-8de3-c65c12668dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:02.059063 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.059029 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a90c1a-667d-4e62-8de3-c65c12668dad-maas-api-tls\") pod \"maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg\" (UID: \"12a90c1a-667d-4e62-8de3-c65c12668dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:02.059225 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.059066 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a90c1a-667d-4e62-8de3-c65c12668dad-openshift-service-ca\") pod \"maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg\" (UID: \"12a90c1a-667d-4e62-8de3-c65c12668dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:02.059225 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.059154 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zb9\" (UniqueName: \"kubernetes.io/projected/12a90c1a-667d-4e62-8de3-c65c12668dad-kube-api-access-h5zb9\") pod \"maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg\" (UID: \"12a90c1a-667d-4e62-8de3-c65c12668dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:02.059225 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:56:02.059198 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-serving-cert: secret "maas-api-e2e-mt-8c5b7a14-serving-cert" not found Jul 17 01:56:02.059353 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:56:02.059255 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a90c1a-667d-4e62-8de3-c65c12668dad-maas-api-tls podName:12a90c1a-667d-4e62-8de3-c65c12668dad nodeName:}" failed. No retries permitted until 2026-07-17 01:56:02.559239326 +0000 UTC m=+2519.954854155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12a90c1a-667d-4e62-8de3-c65c12668dad-maas-api-tls") pod "maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" (UID: "12a90c1a-667d-4e62-8de3-c65c12668dad") : secret "maas-api-e2e-mt-8c5b7a14-serving-cert" not found Jul 17 01:56:02.059759 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.059741 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a90c1a-667d-4e62-8de3-c65c12668dad-openshift-service-ca\") pod \"maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg\" (UID: \"12a90c1a-667d-4e62-8de3-c65c12668dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:02.067570 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.067549 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zb9\" (UniqueName: \"kubernetes.io/projected/12a90c1a-667d-4e62-8de3-c65c12668dad-kube-api-access-h5zb9\") pod \"maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg\" (UID: \"12a90c1a-667d-4e62-8de3-c65c12668dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:02.279352 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.279315 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:56:02.493649 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.493564 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:56:02.564817 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.564786 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a90c1a-667d-4e62-8de3-c65c12668dad-maas-api-tls\") pod \"maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg\" (UID: \"12a90c1a-667d-4e62-8de3-c65c12668dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:02.567349 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.567315 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a90c1a-667d-4e62-8de3-c65c12668dad-maas-api-tls\") pod \"maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg\" (UID: \"12a90c1a-667d-4e62-8de3-c65c12668dad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:02.594676 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.594649 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:56:02.739305 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.739274 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:02.863901 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.863864 2583 generic.go:356] "Generic (PLEG): container finished" podID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" containerID="8a9051a3ff8790d5a7c4ce4ea6d28f6497b5543cdc6b1b9c2f9491d143b81354" exitCode=6 Jul 17 01:56:02.864077 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.863949 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" event={"ID":"fcff2e9b-f323-4c34-a2d5-097e432dffc5","Type":"ContainerDied","Data":"8a9051a3ff8790d5a7c4ce4ea6d28f6497b5543cdc6b1b9c2f9491d143b81354"} Jul 17 01:56:02.864077 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.864015 2583 scope.go:117] "RemoveContainer" containerID="8dbe5e9f113f765daa53c09e1b593f3a852f9cdf4dba87faab71a5244bb74997" Jul 17 01:56:02.864260 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.864237 2583 scope.go:117] "RemoveContainer" containerID="8a9051a3ff8790d5a7c4ce4ea6d28f6497b5543cdc6b1b9c2f9491d143b81354" Jul 17 01:56:02.864535 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:56:02.864509 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv_odh-ai-gateway-infra(fcff2e9b-f323-4c34-a2d5-097e432dffc5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" podUID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" Jul 17 01:56:02.871397 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:02.871372 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg"] Jul 17 01:56:02.875701 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:56:02.875648 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a90c1a_667d_4e62_8de3_c65c12668dad.slice/crio-a13fb467b7565bb0c197dd8d1504670d40a18e99b3058997b7cf0029d3a1388e WatchSource:0}: Error finding container a13fb467b7565bb0c197dd8d1504670d40a18e99b3058997b7cf0029d3a1388e: Status 404 returned error can't find the container with id a13fb467b7565bb0c197dd8d1504670d40a18e99b3058997b7cf0029d3a1388e Jul 17 01:56:03.876142 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:03.876076 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" event={"ID":"12a90c1a-667d-4e62-8de3-c65c12668dad","Type":"ContainerStarted","Data":"9a8c95e7aa1c2c024d3d4f81e7aea6278710bc341b4d28e724a7f4f5055d7551"} Jul 17 01:56:03.876634 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:03.876201 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:03.876634 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:03.876229 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" event={"ID":"12a90c1a-667d-4e62-8de3-c65c12668dad","Type":"ContainerStarted","Data":"a13fb467b7565bb0c197dd8d1504670d40a18e99b3058997b7cf0029d3a1388e"} Jul 17 01:56:03.877829 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:03.877812 2583 scope.go:117] "RemoveContainer" containerID="8a9051a3ff8790d5a7c4ce4ea6d28f6497b5543cdc6b1b9c2f9491d143b81354" Jul 17 01:56:03.877982 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:56:03.877963 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv_odh-ai-gateway-infra(fcff2e9b-f323-4c34-a2d5-097e432dffc5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" podUID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" Jul 17 01:56:03.893570 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:03.893511 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" podStartSLOduration=2.599356471 podStartE2EDuration="2.893497205s" totalImagesPullingTime="294.140734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:56:03.891194253 +0000 UTC m=+2521.286809104" watchObservedRunningTime="2026-07-17 01:56:03.893497205 +0000 UTC m=+2521.289112056" Jul 17 01:56:06.241947 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:06.241911 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:56:09.888412 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:09.888385 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:18.163148 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:18.163094 2583 scope.go:117] "RemoveContainer" containerID="8a9051a3ff8790d5a7c4ce4ea6d28f6497b5543cdc6b1b9c2f9491d143b81354" Jul 17 01:56:18.210248 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:18.210213 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:56:18.946310 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:18.946275 2583 generic.go:356] "Generic (PLEG): container finished" podID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" containerID="1a80b71e0ba98512d5a7cef981f0d1702fd299b1368f08d3381a396e80eed2c7" exitCode=6 Jul 17 01:56:18.946310 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:18.946315 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" event={"ID":"fcff2e9b-f323-4c34-a2d5-097e432dffc5","Type":"ContainerDied","Data":"1a80b71e0ba98512d5a7cef981f0d1702fd299b1368f08d3381a396e80eed2c7"} Jul 17 01:56:18.946527 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:18.946361 2583 scope.go:117] "RemoveContainer" containerID="8a9051a3ff8790d5a7c4ce4ea6d28f6497b5543cdc6b1b9c2f9491d143b81354" Jul 17 01:56:18.946652 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:18.946627 2583 scope.go:117] "RemoveContainer" containerID="1a80b71e0ba98512d5a7cef981f0d1702fd299b1368f08d3381a396e80eed2c7" Jul 17 01:56:18.946865 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:56:18.946845 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv_odh-ai-gateway-infra(fcff2e9b-f323-4c34-a2d5-097e432dffc5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" podUID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" Jul 17 01:56:19.970186 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:19.970153 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv"] Jul 17 01:56:20.102449 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:20.102426 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" Jul 17 01:56:20.119854 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:20.119829 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcff2e9b-f323-4c34-a2d5-097e432dffc5-maas-api-service-ca\") pod \"fcff2e9b-f323-4c34-a2d5-097e432dffc5\" (UID: \"fcff2e9b-f323-4c34-a2d5-097e432dffc5\") " Jul 17 01:56:20.120270 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:20.120241 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcff2e9b-f323-4c34-a2d5-097e432dffc5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fcff2e9b-f323-4c34-a2d5-097e432dffc5" (UID: "fcff2e9b-f323-4c34-a2d5-097e432dffc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:56:20.221339 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:20.221253 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcff2e9b-f323-4c34-a2d5-097e432dffc5-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:56:20.961317 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:20.961286 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" Jul 17 01:56:20.961507 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:20.961292 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv" event={"ID":"fcff2e9b-f323-4c34-a2d5-097e432dffc5","Type":"ContainerDied","Data":"61bc7efa42dca9b351dca3be6fbd93808549ae08e2edfaefc0bd01570a27f8d1"} Jul 17 01:56:20.961507 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:20.961417 2583 scope.go:117] "RemoveContainer" containerID="1a80b71e0ba98512d5a7cef981f0d1702fd299b1368f08d3381a396e80eed2c7" Jul 17 01:56:20.986416 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:20.986381 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv"] Jul 17 01:56:20.989702 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:20.989679 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-flhmv"] Jul 17 01:56:21.167887 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:21.167843 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" path="/var/lib/kubelet/pods/fcff2e9b-f323-4c34-a2d5-097e432dffc5/volumes" Jul 17 01:56:23.482606 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.482573 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf"] Jul 17 01:56:23.482991 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.482903 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" containerName="revoke-keys" Jul 17 01:56:23.482991 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.482914 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" containerName="revoke-keys" Jul 17 01:56:23.482991 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.482924 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" containerName="revoke-keys" Jul 17 01:56:23.482991 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.482930 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" containerName="revoke-keys" Jul 17 01:56:23.482991 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.482989 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" containerName="revoke-keys" Jul 17 01:56:23.483189 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.482996 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" containerName="revoke-keys" Jul 17 01:56:23.483189 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.483001 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" containerName="revoke-keys" Jul 17 01:56:23.487360 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.487340 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" Jul 17 01:56:23.489481 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.489460 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:56:23.492503 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.492481 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf"] Jul 17 01:56:23.549514 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.549479 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003639d8-db6c-4be3-8689-eb788969874e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf\" (UID: \"003639d8-db6c-4be3-8689-eb788969874e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" Jul 17 01:56:23.650194 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.650157 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003639d8-db6c-4be3-8689-eb788969874e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf\" (UID: \"003639d8-db6c-4be3-8689-eb788969874e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" Jul 17 01:56:23.650878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.650860 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003639d8-db6c-4be3-8689-eb788969874e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf\" (UID: \"003639d8-db6c-4be3-8689-eb788969874e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" Jul 17 01:56:23.799594 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.799561 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" Jul 17 01:56:23.930972 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.930934 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf"] Jul 17 01:56:23.934881 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:56:23.934845 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003639d8_db6c_4be3_8689_eb788969874e.slice/crio-0a11040159355a6274148f837cfceb40b9c7cce5df295fcec38353e49ce14556 WatchSource:0}: Error finding container 0a11040159355a6274148f837cfceb40b9c7cce5df295fcec38353e49ce14556: Status 404 returned error can't find the container with id 0a11040159355a6274148f837cfceb40b9c7cce5df295fcec38353e49ce14556 Jul 17 01:56:23.977127 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:23.977080 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" event={"ID":"003639d8-db6c-4be3-8689-eb788969874e","Type":"ContainerStarted","Data":"0a11040159355a6274148f837cfceb40b9c7cce5df295fcec38353e49ce14556"} Jul 17 01:56:24.888854 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:24.888810 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:56:24.983476 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:24.983443 2583 generic.go:356] "Generic (PLEG): container finished" podID="003639d8-db6c-4be3-8689-eb788969874e" containerID="aaab0dda1438194be7f0a66663397d1e54c95bf1566ceb1f8c3f71bd5712ced3" exitCode=6 Jul 17 01:56:24.983648 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:24.983524 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" event={"ID":"003639d8-db6c-4be3-8689-eb788969874e","Type":"ContainerDied","Data":"aaab0dda1438194be7f0a66663397d1e54c95bf1566ceb1f8c3f71bd5712ced3"} Jul 17 01:56:24.983705 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:24.983690 2583 scope.go:117] "RemoveContainer" containerID="aaab0dda1438194be7f0a66663397d1e54c95bf1566ceb1f8c3f71bd5712ced3" Jul 17 01:56:25.990208 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:25.990126 2583 generic.go:356] "Generic (PLEG): container finished" podID="003639d8-db6c-4be3-8689-eb788969874e" containerID="8bd7fccbc1c85d6f2cf5f25116f1c10600283c571994cbc4e3be801c8c72579d" exitCode=6 Jul 17 01:56:25.990208 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:25.990179 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" event={"ID":"003639d8-db6c-4be3-8689-eb788969874e","Type":"ContainerDied","Data":"8bd7fccbc1c85d6f2cf5f25116f1c10600283c571994cbc4e3be801c8c72579d"} Jul 17 01:56:25.990654 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:25.990240 2583 scope.go:117] "RemoveContainer" containerID="aaab0dda1438194be7f0a66663397d1e54c95bf1566ceb1f8c3f71bd5712ced3" Jul 17 01:56:25.990654 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:25.990424 2583 scope.go:117] "RemoveContainer" containerID="8bd7fccbc1c85d6f2cf5f25116f1c10600283c571994cbc4e3be801c8c72579d" Jul 17 01:56:25.990654 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:56:25.990636 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf_odh-ai-gateway-infra(003639d8-db6c-4be3-8689-eb788969874e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" podUID="003639d8-db6c-4be3-8689-eb788969874e" Jul 17 01:56:26.071073 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:26.071040 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv"] Jul 17 01:56:26.071498 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:26.071479 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" containerName="revoke-keys" Jul 17 01:56:26.071583 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:26.071501 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcff2e9b-f323-4c34-a2d5-097e432dffc5" containerName="revoke-keys" Jul 17 01:56:26.075622 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:26.075600 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" Jul 17 01:56:26.080173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:26.080148 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv"] Jul 17 01:56:26.174934 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:26.174884 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b12c959c-8ee0-473e-99b6-648e497470c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv\" (UID: \"b12c959c-8ee0-473e-99b6-648e497470c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" Jul 17 01:56:26.276278 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:26.276169 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b12c959c-8ee0-473e-99b6-648e497470c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv\" (UID: \"b12c959c-8ee0-473e-99b6-648e497470c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" Jul 17 01:56:26.276835 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:26.276809 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b12c959c-8ee0-473e-99b6-648e497470c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv\" (UID: \"b12c959c-8ee0-473e-99b6-648e497470c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" Jul 17 01:56:26.387168 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:26.387124 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" Jul 17 01:56:26.519502 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:26.519478 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv"] Jul 17 01:56:26.524011 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:56:26.523923 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12c959c_8ee0_473e_99b6_648e497470c0.slice/crio-1bbb17644ff1d884acb73bf2634f9cfaa3561e10449c525cf078824ff626fc75 WatchSource:0}: Error finding container 1bbb17644ff1d884acb73bf2634f9cfaa3561e10449c525cf078824ff626fc75: Status 404 returned error can't find the container with id 1bbb17644ff1d884acb73bf2634f9cfaa3561e10449c525cf078824ff626fc75 Jul 17 01:56:26.996614 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:26.996583 2583 scope.go:117] "RemoveContainer" containerID="8bd7fccbc1c85d6f2cf5f25116f1c10600283c571994cbc4e3be801c8c72579d" Jul 17 01:56:26.997070 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:56:26.996808 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf_odh-ai-gateway-infra(003639d8-db6c-4be3-8689-eb788969874e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" podUID="003639d8-db6c-4be3-8689-eb788969874e" Jul 17 01:56:26.997872 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:26.997846 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" event={"ID":"b12c959c-8ee0-473e-99b6-648e497470c0","Type":"ContainerStarted","Data":"fe6eff269eec14fbe4be3820c995e02a6119a543cdaeaa3aa89a4f8bfeba0bac"} Jul 17 01:56:26.997963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:26.997924 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" event={"ID":"b12c959c-8ee0-473e-99b6-648e497470c0","Type":"ContainerStarted","Data":"1bbb17644ff1d884acb73bf2634f9cfaa3561e10449c525cf078824ff626fc75"} Jul 17 01:56:27.025905 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:27.025855 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" podStartSLOduration=1.025839119 podStartE2EDuration="1.025839119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:56:27.025157252 +0000 UTC m=+2544.420772102" watchObservedRunningTime="2026-07-17 01:56:27.025839119 +0000 UTC m=+2544.421453981" Jul 17 01:56:28.003763 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:28.003724 2583 generic.go:356] "Generic (PLEG): container finished" podID="b12c959c-8ee0-473e-99b6-648e497470c0" containerID="fe6eff269eec14fbe4be3820c995e02a6119a543cdaeaa3aa89a4f8bfeba0bac" exitCode=6 Jul 17 01:56:28.004160 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:28.003810 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" event={"ID":"b12c959c-8ee0-473e-99b6-648e497470c0","Type":"ContainerDied","Data":"fe6eff269eec14fbe4be3820c995e02a6119a543cdaeaa3aa89a4f8bfeba0bac"} Jul 17 01:56:28.004160 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:28.003983 2583 scope.go:117] "RemoveContainer" containerID="fe6eff269eec14fbe4be3820c995e02a6119a543cdaeaa3aa89a4f8bfeba0bac" Jul 17 01:56:29.010166 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:29.010072 2583 generic.go:356] "Generic (PLEG): container finished" podID="b12c959c-8ee0-473e-99b6-648e497470c0" containerID="10cd640e77f291c74935ff377042f64f33ef2e13e5f9f0813d37fca2a1332f54" exitCode=6 Jul 17 01:56:29.010166 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:29.010141 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" event={"ID":"b12c959c-8ee0-473e-99b6-648e497470c0","Type":"ContainerDied","Data":"10cd640e77f291c74935ff377042f64f33ef2e13e5f9f0813d37fca2a1332f54"} Jul 17 01:56:29.010651 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:29.010212 2583 scope.go:117] "RemoveContainer" containerID="fe6eff269eec14fbe4be3820c995e02a6119a543cdaeaa3aa89a4f8bfeba0bac" Jul 17 01:56:29.010651 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:29.010358 2583 scope.go:117] "RemoveContainer" containerID="10cd640e77f291c74935ff377042f64f33ef2e13e5f9f0813d37fca2a1332f54" Jul 17 01:56:29.010651 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:56:29.010597 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv_odh-ai-gateway-infra(b12c959c-8ee0-473e-99b6-648e497470c0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" podUID="b12c959c-8ee0-473e-99b6-648e497470c0" Jul 17 01:56:30.016417 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:30.016388 2583 scope.go:117] "RemoveContainer" containerID="10cd640e77f291c74935ff377042f64f33ef2e13e5f9f0813d37fca2a1332f54" Jul 17 01:56:30.016774 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:56:30.016602 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv_odh-ai-gateway-infra(b12c959c-8ee0-473e-99b6-648e497470c0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" podUID="b12c959c-8ee0-473e-99b6-648e497470c0" Jul 17 01:56:42.163010 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:42.162971 2583 scope.go:117] "RemoveContainer" containerID="10cd640e77f291c74935ff377042f64f33ef2e13e5f9f0813d37fca2a1332f54" Jul 17 01:56:42.163601 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:42.163226 2583 scope.go:117] "RemoveContainer" containerID="8bd7fccbc1c85d6f2cf5f25116f1c10600283c571994cbc4e3be801c8c72579d" Jul 17 01:56:43.082323 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:43.082294 2583 generic.go:356] "Generic (PLEG): container finished" podID="003639d8-db6c-4be3-8689-eb788969874e" containerID="8934004371dbd31a3d3ffaa8dbfc130903a089db3dd9bb192ec789ed4dad6613" exitCode=6 Jul 17 01:56:43.082506 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:43.082370 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" event={"ID":"003639d8-db6c-4be3-8689-eb788969874e","Type":"ContainerDied","Data":"8934004371dbd31a3d3ffaa8dbfc130903a089db3dd9bb192ec789ed4dad6613"} Jul 17 01:56:43.082506 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:43.082428 2583 scope.go:117] "RemoveContainer" containerID="8bd7fccbc1c85d6f2cf5f25116f1c10600283c571994cbc4e3be801c8c72579d" Jul 17 01:56:43.082840 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:43.082795 2583 scope.go:117] "RemoveContainer" containerID="8934004371dbd31a3d3ffaa8dbfc130903a089db3dd9bb192ec789ed4dad6613" Jul 17 01:56:43.083063 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:56:43.083037 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf_odh-ai-gateway-infra(003639d8-db6c-4be3-8689-eb788969874e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" podUID="003639d8-db6c-4be3-8689-eb788969874e" Jul 17 01:56:43.084113 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:43.084080 2583 generic.go:356] "Generic (PLEG): container finished" podID="b12c959c-8ee0-473e-99b6-648e497470c0" containerID="02bc0d8329d72a66c6002adfcea6b04bead693d1aabfc2f8b1270e9469aeddd1" exitCode=6 Jul 17 01:56:43.084209 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:43.084127 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" event={"ID":"b12c959c-8ee0-473e-99b6-648e497470c0","Type":"ContainerDied","Data":"02bc0d8329d72a66c6002adfcea6b04bead693d1aabfc2f8b1270e9469aeddd1"} Jul 17 01:56:43.084440 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:43.084424 2583 scope.go:117] "RemoveContainer" containerID="02bc0d8329d72a66c6002adfcea6b04bead693d1aabfc2f8b1270e9469aeddd1" Jul 17 01:56:43.084618 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:56:43.084599 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv_odh-ai-gateway-infra(b12c959c-8ee0-473e-99b6-648e497470c0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" podUID="b12c959c-8ee0-473e-99b6-648e497470c0" Jul 17 01:56:43.093732 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:43.093713 2583 scope.go:117] "RemoveContainer" containerID="10cd640e77f291c74935ff377042f64f33ef2e13e5f9f0813d37fca2a1332f54" Jul 17 01:56:44.106886 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.106857 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf"] Jul 17 01:56:44.119710 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.119674 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv"] Jul 17 01:56:44.255984 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.255955 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" Jul 17 01:56:44.280481 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.280458 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" Jul 17 01:56:44.338441 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.338411 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003639d8-db6c-4be3-8689-eb788969874e-maas-api-service-ca\") pod \"003639d8-db6c-4be3-8689-eb788969874e\" (UID: \"003639d8-db6c-4be3-8689-eb788969874e\") " Jul 17 01:56:44.338754 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.338732 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003639d8-db6c-4be3-8689-eb788969874e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "003639d8-db6c-4be3-8689-eb788969874e" (UID: "003639d8-db6c-4be3-8689-eb788969874e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:56:44.439874 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.439786 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b12c959c-8ee0-473e-99b6-648e497470c0-maas-api-service-ca\") pod \"b12c959c-8ee0-473e-99b6-648e497470c0\" (UID: \"b12c959c-8ee0-473e-99b6-648e497470c0\") " Jul 17 01:56:44.440016 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.439915 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003639d8-db6c-4be3-8689-eb788969874e-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:56:44.440182 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.440158 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12c959c-8ee0-473e-99b6-648e497470c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b12c959c-8ee0-473e-99b6-648e497470c0" (UID: "b12c959c-8ee0-473e-99b6-648e497470c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:56:44.525650 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.525610 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng"] Jul 17 01:56:44.525945 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.525933 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12c959c-8ee0-473e-99b6-648e497470c0" containerName="revoke-keys" Jul 17 01:56:44.526001 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.525946 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12c959c-8ee0-473e-99b6-648e497470c0" containerName="revoke-keys" Jul 17 01:56:44.526001 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.525967 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003639d8-db6c-4be3-8689-eb788969874e" containerName="revoke-keys" Jul 17 01:56:44.526001 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.525973 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="003639d8-db6c-4be3-8689-eb788969874e" containerName="revoke-keys" Jul 17 01:56:44.526001 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.525986 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12c959c-8ee0-473e-99b6-648e497470c0" containerName="revoke-keys" Jul 17 01:56:44.526001 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.525991 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12c959c-8ee0-473e-99b6-648e497470c0" containerName="revoke-keys" Jul 17 01:56:44.526001 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.525996 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12c959c-8ee0-473e-99b6-648e497470c0" containerName="revoke-keys" Jul 17 01:56:44.526001 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.526001 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12c959c-8ee0-473e-99b6-648e497470c0" containerName="revoke-keys" Jul 17 01:56:44.526254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.526008 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003639d8-db6c-4be3-8689-eb788969874e" containerName="revoke-keys" Jul 17 01:56:44.526254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.526013 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="003639d8-db6c-4be3-8689-eb788969874e" containerName="revoke-keys" Jul 17 01:56:44.526254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.526020 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003639d8-db6c-4be3-8689-eb788969874e" containerName="revoke-keys" Jul 17 01:56:44.526254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.526025 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="003639d8-db6c-4be3-8689-eb788969874e" containerName="revoke-keys" Jul 17 01:56:44.526254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.526073 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b12c959c-8ee0-473e-99b6-648e497470c0" containerName="revoke-keys" Jul 17 01:56:44.526254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.526081 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="003639d8-db6c-4be3-8689-eb788969874e" containerName="revoke-keys" Jul 17 01:56:44.526254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.526087 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="003639d8-db6c-4be3-8689-eb788969874e" containerName="revoke-keys" Jul 17 01:56:44.526254 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.526108 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b12c959c-8ee0-473e-99b6-648e497470c0" containerName="revoke-keys" Jul 17 01:56:44.529013 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.528997 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng" Jul 17 01:56:44.535324 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.535299 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng"] Jul 17 01:56:44.540999 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.540972 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b12c959c-8ee0-473e-99b6-648e497470c0-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:56:44.642048 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.642006 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1581115b-9eb5-4a33-8448-851186f870b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng\" (UID: \"1581115b-9eb5-4a33-8448-851186f870b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng" Jul 17 01:56:44.743381 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.743287 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1581115b-9eb5-4a33-8448-851186f870b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng\" (UID: \"1581115b-9eb5-4a33-8448-851186f870b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng" Jul 17 01:56:44.743981 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.743955 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1581115b-9eb5-4a33-8448-851186f870b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng\" (UID: \"1581115b-9eb5-4a33-8448-851186f870b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng" Jul 17 01:56:44.840215 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.840182 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng" Jul 17 01:56:44.971038 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:44.971014 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng"] Jul 17 01:56:44.972743 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:56:44.972721 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1581115b_9eb5_4a33_8448_851186f870b5.slice/crio-f146b7014b536624c1d8bd383918b075a8d1df7e91d704c82467a4a5b198c747 WatchSource:0}: Error finding container f146b7014b536624c1d8bd383918b075a8d1df7e91d704c82467a4a5b198c747: Status 404 returned error can't find the container with id f146b7014b536624c1d8bd383918b075a8d1df7e91d704c82467a4a5b198c747 Jul 17 01:56:45.098284 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:45.098252 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" Jul 17 01:56:45.098443 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:45.098252 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf" event={"ID":"003639d8-db6c-4be3-8689-eb788969874e","Type":"ContainerDied","Data":"0a11040159355a6274148f837cfceb40b9c7cce5df295fcec38353e49ce14556"} Jul 17 01:56:45.098443 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:45.098390 2583 scope.go:117] "RemoveContainer" containerID="8934004371dbd31a3d3ffaa8dbfc130903a089db3dd9bb192ec789ed4dad6613" Jul 17 01:56:45.099624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:45.099591 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" Jul 17 01:56:45.099722 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:45.099631 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv" event={"ID":"b12c959c-8ee0-473e-99b6-648e497470c0","Type":"ContainerDied","Data":"1bbb17644ff1d884acb73bf2634f9cfaa3561e10449c525cf078824ff626fc75"} Jul 17 01:56:45.101277 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:45.101251 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng" event={"ID":"1581115b-9eb5-4a33-8448-851186f870b5","Type":"ContainerStarted","Data":"f146b7014b536624c1d8bd383918b075a8d1df7e91d704c82467a4a5b198c747"} Jul 17 01:56:45.114283 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:45.113955 2583 scope.go:117] "RemoveContainer" containerID="02bc0d8329d72a66c6002adfcea6b04bead693d1aabfc2f8b1270e9469aeddd1" Jul 17 01:56:45.129527 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:45.129499 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv"] Jul 17 01:56:45.135996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:45.135973 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-f4rpv"] Jul 17 01:56:45.144992 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:45.144970 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf"] Jul 17 01:56:45.148855 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:45.148832 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-bhvgf"] Jul 17 01:56:45.167949 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:45.167918 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="003639d8-db6c-4be3-8689-eb788969874e" path="/var/lib/kubelet/pods/003639d8-db6c-4be3-8689-eb788969874e/volumes" Jul 17 01:56:45.168449 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:45.168431 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b12c959c-8ee0-473e-99b6-648e497470c0" path="/var/lib/kubelet/pods/b12c959c-8ee0-473e-99b6-648e497470c0/volumes" Jul 17 01:56:46.108980 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:46.108944 2583 generic.go:356] "Generic (PLEG): container finished" podID="1581115b-9eb5-4a33-8448-851186f870b5" containerID="3e3e84d44097cc5956a39b0a8335e8ee252ea14a3ac15656ad5dcab4c1817807" exitCode=0 Jul 17 01:56:46.109224 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:46.109034 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng" event={"ID":"1581115b-9eb5-4a33-8448-851186f870b5","Type":"ContainerDied","Data":"3e3e84d44097cc5956a39b0a8335e8ee252ea14a3ac15656ad5dcab4c1817807"} Jul 17 01:56:47.247497 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:47.247470 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng" Jul 17 01:56:47.367397 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:47.367311 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1581115b-9eb5-4a33-8448-851186f870b5-maas-api-service-ca\") pod \"1581115b-9eb5-4a33-8448-851186f870b5\" (UID: \"1581115b-9eb5-4a33-8448-851186f870b5\") " Jul 17 01:56:47.367663 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:47.367638 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1581115b-9eb5-4a33-8448-851186f870b5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1581115b-9eb5-4a33-8448-851186f870b5" (UID: "1581115b-9eb5-4a33-8448-851186f870b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:56:47.468315 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:47.468280 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1581115b-9eb5-4a33-8448-851186f870b5-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:56:48.120452 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:48.120416 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng" Jul 17 01:56:48.120621 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:48.120423 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng" event={"ID":"1581115b-9eb5-4a33-8448-851186f870b5","Type":"ContainerDied","Data":"f146b7014b536624c1d8bd383918b075a8d1df7e91d704c82467a4a5b198c747"} Jul 17 01:56:48.120621 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:48.120555 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f146b7014b536624c1d8bd383918b075a8d1df7e91d704c82467a4a5b198c747" Jul 17 01:56:54.924997 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:54.924955 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg"] Jul 17 01:56:54.925479 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:54.925221 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" podUID="12a90c1a-667d-4e62-8de3-c65c12668dad" containerName="maas-api" containerID="cri-o://9a8c95e7aa1c2c024d3d4f81e7aea6278710bc341b4d28e724a7f4f5055d7551" gracePeriod=30 Jul 17 01:56:54.952166 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:54.952093 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:56:55.154070 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:55.154040 2583 generic.go:356] "Generic (PLEG): container finished" podID="12a90c1a-667d-4e62-8de3-c65c12668dad" containerID="9a8c95e7aa1c2c024d3d4f81e7aea6278710bc341b4d28e724a7f4f5055d7551" exitCode=0 Jul 17 01:56:55.154238 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:55.154141 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" event={"ID":"12a90c1a-667d-4e62-8de3-c65c12668dad","Type":"ContainerDied","Data":"9a8c95e7aa1c2c024d3d4f81e7aea6278710bc341b4d28e724a7f4f5055d7551"} Jul 17 01:56:55.168978 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:55.168955 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:55.339278 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:55.339252 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a90c1a-667d-4e62-8de3-c65c12668dad-maas-api-tls\") pod \"12a90c1a-667d-4e62-8de3-c65c12668dad\" (UID: \"12a90c1a-667d-4e62-8de3-c65c12668dad\") " Jul 17 01:56:55.339446 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:55.339293 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a90c1a-667d-4e62-8de3-c65c12668dad-openshift-service-ca\") pod \"12a90c1a-667d-4e62-8de3-c65c12668dad\" (UID: \"12a90c1a-667d-4e62-8de3-c65c12668dad\") " Jul 17 01:56:55.339446 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:55.339364 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5zb9\" (UniqueName: \"kubernetes.io/projected/12a90c1a-667d-4e62-8de3-c65c12668dad-kube-api-access-h5zb9\") pod \"12a90c1a-667d-4e62-8de3-c65c12668dad\" (UID: \"12a90c1a-667d-4e62-8de3-c65c12668dad\") " Jul 17 01:56:55.339690 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:55.339666 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a90c1a-667d-4e62-8de3-c65c12668dad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12a90c1a-667d-4e62-8de3-c65c12668dad" (UID: "12a90c1a-667d-4e62-8de3-c65c12668dad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:56:55.341504 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:55.341481 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a90c1a-667d-4e62-8de3-c65c12668dad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12a90c1a-667d-4e62-8de3-c65c12668dad" (UID: "12a90c1a-667d-4e62-8de3-c65c12668dad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:56:55.341593 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:55.341583 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a90c1a-667d-4e62-8de3-c65c12668dad-kube-api-access-h5zb9" (OuterVolumeSpecName: "kube-api-access-h5zb9") pod "12a90c1a-667d-4e62-8de3-c65c12668dad" (UID: "12a90c1a-667d-4e62-8de3-c65c12668dad"). InnerVolumeSpecName "kube-api-access-h5zb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:56:55.440381 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:55.440341 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a90c1a-667d-4e62-8de3-c65c12668dad-openshift-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:56:55.440381 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:55.440377 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5zb9\" (UniqueName: \"kubernetes.io/projected/12a90c1a-667d-4e62-8de3-c65c12668dad-kube-api-access-h5zb9\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:56:55.440381 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:55.440387 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a90c1a-667d-4e62-8de3-c65c12668dad-maas-api-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:56:56.160346 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:56.160315 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" event={"ID":"12a90c1a-667d-4e62-8de3-c65c12668dad","Type":"ContainerDied","Data":"a13fb467b7565bb0c197dd8d1504670d40a18e99b3058997b7cf0029d3a1388e"} Jul 17 01:56:56.160784 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:56.160349 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg" Jul 17 01:56:56.160784 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:56.160374 2583 scope.go:117] "RemoveContainer" containerID="9a8c95e7aa1c2c024d3d4f81e7aea6278710bc341b4d28e724a7f4f5055d7551" Jul 17 01:56:56.180011 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:56.179982 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg"] Jul 17 01:56:56.183064 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:56.183041 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c5b7a14-6c7cc86bfb-srhfg"] Jul 17 01:56:57.168680 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:57.168648 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12a90c1a-667d-4e62-8de3-c65c12668dad" path="/var/lib/kubelet/pods/12a90c1a-667d-4e62-8de3-c65c12668dad/volumes" Jul 17 01:56:59.600299 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:56:59.600268 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:57:00.037641 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.037604 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9"] Jul 17 01:57:00.037937 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.037925 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a90c1a-667d-4e62-8de3-c65c12668dad" containerName="maas-api" Jul 17 01:57:00.037978 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.037938 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a90c1a-667d-4e62-8de3-c65c12668dad" containerName="maas-api" Jul 17 01:57:00.037978 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.037956 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1581115b-9eb5-4a33-8448-851186f870b5" containerName="revoke-keys" Jul 17 01:57:00.037978 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.037961 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1581115b-9eb5-4a33-8448-851186f870b5" containerName="revoke-keys" Jul 17 01:57:00.038065 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.038007 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b12c959c-8ee0-473e-99b6-648e497470c0" containerName="revoke-keys" Jul 17 01:57:00.038065 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.038015 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1581115b-9eb5-4a33-8448-851186f870b5" containerName="revoke-keys" Jul 17 01:57:00.038065 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.038023 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="003639d8-db6c-4be3-8689-eb788969874e" containerName="revoke-keys" Jul 17 01:57:00.038065 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.038029 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a90c1a-667d-4e62-8de3-c65c12668dad" containerName="maas-api" Jul 17 01:57:00.042067 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.042049 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" Jul 17 01:57:00.044384 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.044366 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:57:00.044479 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.044388 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:57:00.048074 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.048043 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9"] Jul 17 01:57:00.075921 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.075886 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02b99c-eda5-4a2b-8968-96b647e6b188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9\" (UID: \"ad02b99c-eda5-4a2b-8968-96b647e6b188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" Jul 17 01:57:00.176866 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.176834 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02b99c-eda5-4a2b-8968-96b647e6b188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9\" (UID: \"ad02b99c-eda5-4a2b-8968-96b647e6b188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" Jul 17 01:57:00.177588 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.177567 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02b99c-eda5-4a2b-8968-96b647e6b188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9\" (UID: \"ad02b99c-eda5-4a2b-8968-96b647e6b188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" Jul 17 01:57:00.354586 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.354497 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" Jul 17 01:57:00.484722 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:00.484689 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9"] Jul 17 01:57:00.488169 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:57:00.488137 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad02b99c_eda5_4a2b_8968_96b647e6b188.slice/crio-2228f275c476d339ed50ea71d95b2c69233ce75f439720d1ac6acc1cbb63c73f WatchSource:0}: Error finding container 2228f275c476d339ed50ea71d95b2c69233ce75f439720d1ac6acc1cbb63c73f: Status 404 returned error can't find the container with id 2228f275c476d339ed50ea71d95b2c69233ce75f439720d1ac6acc1cbb63c73f Jul 17 01:57:01.185175 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:01.185141 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" event={"ID":"ad02b99c-eda5-4a2b-8968-96b647e6b188","Type":"ContainerStarted","Data":"512a205539d15702f2e630e7e5c30aa94495501ff6e55c56442400a7ae595400"} Jul 17 01:57:01.185602 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:01.185198 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" event={"ID":"ad02b99c-eda5-4a2b-8968-96b647e6b188","Type":"ContainerStarted","Data":"2228f275c476d339ed50ea71d95b2c69233ce75f439720d1ac6acc1cbb63c73f"} Jul 17 01:57:01.201339 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:01.201281 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" podStartSLOduration=1.201261264 podStartE2EDuration="1.201261264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:57:01.198723867 +0000 UTC m=+2578.594338719" watchObservedRunningTime="2026-07-17 01:57:01.201261264 +0000 UTC m=+2578.596876115" Jul 17 01:57:02.191475 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:02.191439 2583 generic.go:356] "Generic (PLEG): container finished" podID="ad02b99c-eda5-4a2b-8968-96b647e6b188" containerID="512a205539d15702f2e630e7e5c30aa94495501ff6e55c56442400a7ae595400" exitCode=6 Jul 17 01:57:02.191963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:02.191526 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" event={"ID":"ad02b99c-eda5-4a2b-8968-96b647e6b188","Type":"ContainerDied","Data":"512a205539d15702f2e630e7e5c30aa94495501ff6e55c56442400a7ae595400"} Jul 17 01:57:02.191963 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:02.191777 2583 scope.go:117] "RemoveContainer" containerID="512a205539d15702f2e630e7e5c30aa94495501ff6e55c56442400a7ae595400" Jul 17 01:57:03.197785 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:03.197686 2583 generic.go:356] "Generic (PLEG): container finished" podID="ad02b99c-eda5-4a2b-8968-96b647e6b188" containerID="ab4e101eeff2e59cb36cae128417d34e509f3a513fa4e8f97f3e386183381392" exitCode=6 Jul 17 01:57:03.197785 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:03.197769 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" event={"ID":"ad02b99c-eda5-4a2b-8968-96b647e6b188","Type":"ContainerDied","Data":"ab4e101eeff2e59cb36cae128417d34e509f3a513fa4e8f97f3e386183381392"} Jul 17 01:57:03.198274 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:03.197835 2583 scope.go:117] "RemoveContainer" containerID="512a205539d15702f2e630e7e5c30aa94495501ff6e55c56442400a7ae595400" Jul 17 01:57:03.198274 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:03.198075 2583 scope.go:117] "RemoveContainer" containerID="ab4e101eeff2e59cb36cae128417d34e509f3a513fa4e8f97f3e386183381392" Jul 17 01:57:03.198353 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:57:03.198310 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9_odh-ai-gateway-infra(ad02b99c-eda5-4a2b-8968-96b647e6b188)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" podUID="ad02b99c-eda5-4a2b-8968-96b647e6b188" Jul 17 01:57:04.204213 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:04.204185 2583 scope.go:117] "RemoveContainer" containerID="ab4e101eeff2e59cb36cae128417d34e509f3a513fa4e8f97f3e386183381392" Jul 17 01:57:04.204596 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:57:04.204380 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9_odh-ai-gateway-infra(ad02b99c-eda5-4a2b-8968-96b647e6b188)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" podUID="ad02b99c-eda5-4a2b-8968-96b647e6b188" Jul 17 01:57:04.658040 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:04.658007 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:57:08.149654 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:08.149616 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw"] Jul 17 01:57:08.152874 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:08.152853 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" Jul 17 01:57:08.161955 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:08.161922 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw"] Jul 17 01:57:08.243470 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:08.243431 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5a010e-8ecc-496c-8e3d-cc7c987e03a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw\" (UID: \"fa5a010e-8ecc-496c-8e3d-cc7c987e03a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" Jul 17 01:57:08.344732 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:08.344696 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5a010e-8ecc-496c-8e3d-cc7c987e03a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw\" (UID: \"fa5a010e-8ecc-496c-8e3d-cc7c987e03a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" Jul 17 01:57:08.345353 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:08.345333 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5a010e-8ecc-496c-8e3d-cc7c987e03a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw\" (UID: \"fa5a010e-8ecc-496c-8e3d-cc7c987e03a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" Jul 17 01:57:08.464708 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:08.464623 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" Jul 17 01:57:08.808550 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:08.808505 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw"] Jul 17 01:57:08.811809 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:57:08.811775 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5a010e_8ecc_496c_8e3d_cc7c987e03a1.slice/crio-d1c9aa89291a32c88a2fcaee694cd128f139df83cb145266c99641e9d6a496d3 WatchSource:0}: Error finding container d1c9aa89291a32c88a2fcaee694cd128f139df83cb145266c99641e9d6a496d3: Status 404 returned error can't find the container with id d1c9aa89291a32c88a2fcaee694cd128f139df83cb145266c99641e9d6a496d3 Jul 17 01:57:09.230080 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:09.229975 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" event={"ID":"fa5a010e-8ecc-496c-8e3d-cc7c987e03a1","Type":"ContainerStarted","Data":"bae72997f3551938d6cbcdb56e8254577338e9886b172dcc3f986b3617b376e7"} Jul 17 01:57:09.230080 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:09.230044 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" event={"ID":"fa5a010e-8ecc-496c-8e3d-cc7c987e03a1","Type":"ContainerStarted","Data":"d1c9aa89291a32c88a2fcaee694cd128f139df83cb145266c99641e9d6a496d3"} Jul 17 01:57:09.248009 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:09.247951 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" podStartSLOduration=1.247934995 podStartE2EDuration="1.247934995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:57:09.24644572 +0000 UTC m=+2586.642060573" watchObservedRunningTime="2026-07-17 01:57:09.247934995 +0000 UTC m=+2586.643549846" Jul 17 01:57:10.236205 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:10.236173 2583 generic.go:356] "Generic (PLEG): container finished" podID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" containerID="bae72997f3551938d6cbcdb56e8254577338e9886b172dcc3f986b3617b376e7" exitCode=6 Jul 17 01:57:10.236612 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:10.236268 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" event={"ID":"fa5a010e-8ecc-496c-8e3d-cc7c987e03a1","Type":"ContainerDied","Data":"bae72997f3551938d6cbcdb56e8254577338e9886b172dcc3f986b3617b376e7"} Jul 17 01:57:10.236612 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:10.236443 2583 scope.go:117] "RemoveContainer" containerID="bae72997f3551938d6cbcdb56e8254577338e9886b172dcc3f986b3617b376e7" Jul 17 01:57:10.347599 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:10.347567 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:57:10.444264 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:10.444241 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:57:11.243229 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:11.243145 2583 generic.go:356] "Generic (PLEG): container finished" podID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" containerID="449da788f4a3ba40707c5e2bfea11c44672fb2d956b6156d827477030206b6d0" exitCode=6 Jul 17 01:57:11.243624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:11.243233 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" event={"ID":"fa5a010e-8ecc-496c-8e3d-cc7c987e03a1","Type":"ContainerDied","Data":"449da788f4a3ba40707c5e2bfea11c44672fb2d956b6156d827477030206b6d0"} Jul 17 01:57:11.243624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:11.243293 2583 scope.go:117] "RemoveContainer" containerID="bae72997f3551938d6cbcdb56e8254577338e9886b172dcc3f986b3617b376e7" Jul 17 01:57:11.243624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:11.243558 2583 scope.go:117] "RemoveContainer" containerID="449da788f4a3ba40707c5e2bfea11c44672fb2d956b6156d827477030206b6d0" Jul 17 01:57:11.243792 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:57:11.243756 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw_odh-ai-gateway-infra(fa5a010e-8ecc-496c-8e3d-cc7c987e03a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" podUID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" Jul 17 01:57:12.249924 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:12.249894 2583 scope.go:117] "RemoveContainer" containerID="449da788f4a3ba40707c5e2bfea11c44672fb2d956b6156d827477030206b6d0" Jul 17 01:57:12.250371 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:57:12.250076 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw_odh-ai-gateway-infra(fa5a010e-8ecc-496c-8e3d-cc7c987e03a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" podUID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" Jul 17 01:57:15.169222 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:15.169195 2583 scope.go:117] "RemoveContainer" containerID="ab4e101eeff2e59cb36cae128417d34e509f3a513fa4e8f97f3e386183381392" Jul 17 01:57:16.111548 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:16.111509 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9"] Jul 17 01:57:16.114785 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:16.114768 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" Jul 17 01:57:16.122711 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:16.122684 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9"] Jul 17 01:57:16.219284 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:16.219247 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9b4dde-25ab-47b9-a350-440343a0c6b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9\" (UID: \"aa9b4dde-25ab-47b9-a350-440343a0c6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" Jul 17 01:57:16.269741 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:16.269710 2583 generic.go:356] "Generic (PLEG): container finished" podID="ad02b99c-eda5-4a2b-8968-96b647e6b188" containerID="5f1c4a488adccd524eb1bfd8738acbe05b10959bb78d2fd196f46c5da6ce7415" exitCode=6 Jul 17 01:57:16.269919 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:16.269769 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" event={"ID":"ad02b99c-eda5-4a2b-8968-96b647e6b188","Type":"ContainerDied","Data":"5f1c4a488adccd524eb1bfd8738acbe05b10959bb78d2fd196f46c5da6ce7415"} Jul 17 01:57:16.269919 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:16.269819 2583 scope.go:117] "RemoveContainer" containerID="ab4e101eeff2e59cb36cae128417d34e509f3a513fa4e8f97f3e386183381392" Jul 17 01:57:16.270160 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:16.270135 2583 scope.go:117] "RemoveContainer" containerID="5f1c4a488adccd524eb1bfd8738acbe05b10959bb78d2fd196f46c5da6ce7415" Jul 17 01:57:16.270368 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:57:16.270342 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9_odh-ai-gateway-infra(ad02b99c-eda5-4a2b-8968-96b647e6b188)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" podUID="ad02b99c-eda5-4a2b-8968-96b647e6b188" Jul 17 01:57:16.320331 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:16.320296 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9b4dde-25ab-47b9-a350-440343a0c6b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9\" (UID: \"aa9b4dde-25ab-47b9-a350-440343a0c6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" Jul 17 01:57:16.320904 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:16.320884 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9b4dde-25ab-47b9-a350-440343a0c6b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9\" (UID: \"aa9b4dde-25ab-47b9-a350-440343a0c6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" Jul 17 01:57:16.427093 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:16.426999 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" Jul 17 01:57:16.567991 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:16.567962 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9"] Jul 17 01:57:16.569921 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:57:16.569890 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9b4dde_25ab_47b9_a350_440343a0c6b1.slice/crio-ea1824b54f8bb56c921d9074d3e643361810b58c098ebe25bcbf8a4323a10496 WatchSource:0}: Error finding container ea1824b54f8bb56c921d9074d3e643361810b58c098ebe25bcbf8a4323a10496: Status 404 returned error can't find the container with id ea1824b54f8bb56c921d9074d3e643361810b58c098ebe25bcbf8a4323a10496 Jul 17 01:57:17.278260 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:17.278220 2583 generic.go:356] "Generic (PLEG): container finished" podID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" containerID="e5d579e0baa5dc1468ecdc3c43b36b5b98a79b60abbb98577e19a574342ed510" exitCode=6 Jul 17 01:57:17.278681 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:17.278307 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" event={"ID":"aa9b4dde-25ab-47b9-a350-440343a0c6b1","Type":"ContainerDied","Data":"e5d579e0baa5dc1468ecdc3c43b36b5b98a79b60abbb98577e19a574342ed510"} Jul 17 01:57:17.278681 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:17.278356 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" event={"ID":"aa9b4dde-25ab-47b9-a350-440343a0c6b1","Type":"ContainerStarted","Data":"ea1824b54f8bb56c921d9074d3e643361810b58c098ebe25bcbf8a4323a10496"} Jul 17 01:57:17.278681 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:17.278496 2583 scope.go:117] "RemoveContainer" containerID="e5d579e0baa5dc1468ecdc3c43b36b5b98a79b60abbb98577e19a574342ed510" Jul 17 01:57:17.306110 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:17.306062 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9"] Jul 17 01:57:17.436245 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:17.436222 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" Jul 17 01:57:17.529980 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:17.529957 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02b99c-eda5-4a2b-8968-96b647e6b188-maas-api-service-ca\") pod \"ad02b99c-eda5-4a2b-8968-96b647e6b188\" (UID: \"ad02b99c-eda5-4a2b-8968-96b647e6b188\") " Jul 17 01:57:17.530320 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:17.530300 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad02b99c-eda5-4a2b-8968-96b647e6b188-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad02b99c-eda5-4a2b-8968-96b647e6b188" (UID: "ad02b99c-eda5-4a2b-8968-96b647e6b188"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:57:17.630636 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:17.630594 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02b99c-eda5-4a2b-8968-96b647e6b188-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:57:18.284754 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:18.284721 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" Jul 17 01:57:18.284754 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:18.284740 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9" event={"ID":"ad02b99c-eda5-4a2b-8968-96b647e6b188","Type":"ContainerDied","Data":"2228f275c476d339ed50ea71d95b2c69233ce75f439720d1ac6acc1cbb63c73f"} Jul 17 01:57:18.285342 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:18.284804 2583 scope.go:117] "RemoveContainer" containerID="5f1c4a488adccd524eb1bfd8738acbe05b10959bb78d2fd196f46c5da6ce7415" Jul 17 01:57:18.286910 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:18.286882 2583 generic.go:356] "Generic (PLEG): container finished" podID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" containerID="6aa266b85fda6b5e571f797a9122f1b1663b6812a6e4aac973641c6b341b1d0c" exitCode=6 Jul 17 01:57:18.287038 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:18.286917 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" event={"ID":"aa9b4dde-25ab-47b9-a350-440343a0c6b1","Type":"ContainerDied","Data":"6aa266b85fda6b5e571f797a9122f1b1663b6812a6e4aac973641c6b341b1d0c"} Jul 17 01:57:18.287246 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:18.287230 2583 scope.go:117] "RemoveContainer" containerID="6aa266b85fda6b5e571f797a9122f1b1663b6812a6e4aac973641c6b341b1d0c" Jul 17 01:57:18.287481 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:57:18.287459 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9_odh-ai-gateway-infra(aa9b4dde-25ab-47b9-a350-440343a0c6b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" podUID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" Jul 17 01:57:18.294625 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:18.294609 2583 scope.go:117] "RemoveContainer" containerID="e5d579e0baa5dc1468ecdc3c43b36b5b98a79b60abbb98577e19a574342ed510" Jul 17 01:57:18.321861 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:18.321828 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9"] Jul 17 01:57:18.325692 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:18.325665 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-64zt9"] Jul 17 01:57:19.168019 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:19.167986 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad02b99c-eda5-4a2b-8968-96b647e6b188" path="/var/lib/kubelet/pods/ad02b99c-eda5-4a2b-8968-96b647e6b188/volumes" Jul 17 01:57:19.293718 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:19.293694 2583 scope.go:117] "RemoveContainer" containerID="6aa266b85fda6b5e571f797a9122f1b1663b6812a6e4aac973641c6b341b1d0c" Jul 17 01:57:19.294069 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:57:19.293892 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9_odh-ai-gateway-infra(aa9b4dde-25ab-47b9-a350-440343a0c6b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" podUID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" Jul 17 01:57:23.526947 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.526909 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj"] Jul 17 01:57:23.527353 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.527267 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad02b99c-eda5-4a2b-8968-96b647e6b188" containerName="revoke-keys" Jul 17 01:57:23.527353 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.527280 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad02b99c-eda5-4a2b-8968-96b647e6b188" containerName="revoke-keys" Jul 17 01:57:23.527353 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.527291 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad02b99c-eda5-4a2b-8968-96b647e6b188" containerName="revoke-keys" Jul 17 01:57:23.527353 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.527298 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad02b99c-eda5-4a2b-8968-96b647e6b188" containerName="revoke-keys" Jul 17 01:57:23.527353 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.527312 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad02b99c-eda5-4a2b-8968-96b647e6b188" containerName="revoke-keys" Jul 17 01:57:23.527353 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.527317 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad02b99c-eda5-4a2b-8968-96b647e6b188" containerName="revoke-keys" Jul 17 01:57:23.527543 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.527371 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad02b99c-eda5-4a2b-8968-96b647e6b188" containerName="revoke-keys" Jul 17 01:57:23.527543 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.527378 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad02b99c-eda5-4a2b-8968-96b647e6b188" containerName="revoke-keys" Jul 17 01:57:23.531752 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.531730 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" Jul 17 01:57:23.538152 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.538122 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj"] Jul 17 01:57:23.581566 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.581519 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7eb407-b314-4040-95ab-b2a36312c19d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj\" (UID: \"6c7eb407-b314-4040-95ab-b2a36312c19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" Jul 17 01:57:23.682136 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.682084 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7eb407-b314-4040-95ab-b2a36312c19d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj\" (UID: \"6c7eb407-b314-4040-95ab-b2a36312c19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" Jul 17 01:57:23.682779 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.682758 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7eb407-b314-4040-95ab-b2a36312c19d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj\" (UID: \"6c7eb407-b314-4040-95ab-b2a36312c19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" Jul 17 01:57:23.844417 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.844375 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" Jul 17 01:57:23.978125 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:23.978081 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj"] Jul 17 01:57:23.980590 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:57:23.980563 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7eb407_b314_4040_95ab_b2a36312c19d.slice/crio-63f9acc603848f4da98913962f520c4456436c42f4ce091b90db26de73d79e05 WatchSource:0}: Error finding container 63f9acc603848f4da98913962f520c4456436c42f4ce091b90db26de73d79e05: Status 404 returned error can't find the container with id 63f9acc603848f4da98913962f520c4456436c42f4ce091b90db26de73d79e05 Jul 17 01:57:24.319428 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:24.319393 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" event={"ID":"6c7eb407-b314-4040-95ab-b2a36312c19d","Type":"ContainerStarted","Data":"a2747b8abf96094355b0ca563e643a7f402f78f85a531676d046b395ddd0a1c1"} Jul 17 01:57:24.319599 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:24.319454 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" event={"ID":"6c7eb407-b314-4040-95ab-b2a36312c19d","Type":"ContainerStarted","Data":"63f9acc603848f4da98913962f520c4456436c42f4ce091b90db26de73d79e05"} Jul 17 01:57:24.336922 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:24.336862 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" podStartSLOduration=1.336840552 podStartE2EDuration="1.336840552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:57:24.334371885 +0000 UTC m=+2601.729986736" watchObservedRunningTime="2026-07-17 01:57:24.336840552 +0000 UTC m=+2601.732455404" Jul 17 01:57:25.325074 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:25.325038 2583 generic.go:356] "Generic (PLEG): container finished" podID="6c7eb407-b314-4040-95ab-b2a36312c19d" containerID="a2747b8abf96094355b0ca563e643a7f402f78f85a531676d046b395ddd0a1c1" exitCode=6 Jul 17 01:57:25.325484 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:25.325128 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" event={"ID":"6c7eb407-b314-4040-95ab-b2a36312c19d","Type":"ContainerDied","Data":"a2747b8abf96094355b0ca563e643a7f402f78f85a531676d046b395ddd0a1c1"} Jul 17 01:57:25.325484 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:25.325316 2583 scope.go:117] "RemoveContainer" containerID="a2747b8abf96094355b0ca563e643a7f402f78f85a531676d046b395ddd0a1c1" Jul 17 01:57:26.163172 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:26.163137 2583 scope.go:117] "RemoveContainer" containerID="449da788f4a3ba40707c5e2bfea11c44672fb2d956b6156d827477030206b6d0" Jul 17 01:57:26.331755 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:26.331728 2583 generic.go:356] "Generic (PLEG): container finished" podID="6c7eb407-b314-4040-95ab-b2a36312c19d" containerID="18449f548b4940dcd8b636500c1b2e3b190964b21ea6658702f63be9e459cc37" exitCode=6 Jul 17 01:57:26.332132 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:26.331817 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" event={"ID":"6c7eb407-b314-4040-95ab-b2a36312c19d","Type":"ContainerDied","Data":"18449f548b4940dcd8b636500c1b2e3b190964b21ea6658702f63be9e459cc37"} Jul 17 01:57:26.332132 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:26.331868 2583 scope.go:117] "RemoveContainer" containerID="a2747b8abf96094355b0ca563e643a7f402f78f85a531676d046b395ddd0a1c1" Jul 17 01:57:26.332132 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:26.332041 2583 scope.go:117] "RemoveContainer" containerID="18449f548b4940dcd8b636500c1b2e3b190964b21ea6658702f63be9e459cc37" Jul 17 01:57:26.332302 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:57:26.332280 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj_odh-ai-gateway-infra(6c7eb407-b314-4040-95ab-b2a36312c19d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" podUID="6c7eb407-b314-4040-95ab-b2a36312c19d" Jul 17 01:57:27.338509 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:27.338481 2583 generic.go:356] "Generic (PLEG): container finished" podID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" containerID="e9bff9bec472ce6f285ecd87ebf8ed8655835b999cdee1a8cb81f8c99f734a8d" exitCode=6 Jul 17 01:57:27.338984 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:27.338559 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" event={"ID":"fa5a010e-8ecc-496c-8e3d-cc7c987e03a1","Type":"ContainerDied","Data":"e9bff9bec472ce6f285ecd87ebf8ed8655835b999cdee1a8cb81f8c99f734a8d"} Jul 17 01:57:27.338984 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:27.338615 2583 scope.go:117] "RemoveContainer" containerID="449da788f4a3ba40707c5e2bfea11c44672fb2d956b6156d827477030206b6d0" Jul 17 01:57:27.338984 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:27.338927 2583 scope.go:117] "RemoveContainer" containerID="e9bff9bec472ce6f285ecd87ebf8ed8655835b999cdee1a8cb81f8c99f734a8d" Jul 17 01:57:27.339224 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:57:27.339203 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw_odh-ai-gateway-infra(fa5a010e-8ecc-496c-8e3d-cc7c987e03a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" podUID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" Jul 17 01:57:27.340429 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:27.340416 2583 scope.go:117] "RemoveContainer" containerID="18449f548b4940dcd8b636500c1b2e3b190964b21ea6658702f63be9e459cc37" Jul 17 01:57:27.340585 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:57:27.340569 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj_odh-ai-gateway-infra(6c7eb407-b314-4040-95ab-b2a36312c19d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" podUID="6c7eb407-b314-4040-95ab-b2a36312c19d" Jul 17 01:57:28.370948 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:28.370913 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw"] Jul 17 01:57:28.506489 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:28.506464 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" Jul 17 01:57:28.629404 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:28.629317 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5a010e-8ecc-496c-8e3d-cc7c987e03a1-maas-api-service-ca\") pod \"fa5a010e-8ecc-496c-8e3d-cc7c987e03a1\" (UID: \"fa5a010e-8ecc-496c-8e3d-cc7c987e03a1\") " Jul 17 01:57:28.629682 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:28.629661 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5a010e-8ecc-496c-8e3d-cc7c987e03a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" (UID: "fa5a010e-8ecc-496c-8e3d-cc7c987e03a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:57:28.729924 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:28.729882 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa5a010e-8ecc-496c-8e3d-cc7c987e03a1-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:57:29.352505 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:29.352476 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" Jul 17 01:57:29.352505 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:29.352496 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw" event={"ID":"fa5a010e-8ecc-496c-8e3d-cc7c987e03a1","Type":"ContainerDied","Data":"d1c9aa89291a32c88a2fcaee694cd128f139df83cb145266c99641e9d6a496d3"} Jul 17 01:57:29.352727 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:29.352560 2583 scope.go:117] "RemoveContainer" containerID="e9bff9bec472ce6f285ecd87ebf8ed8655835b999cdee1a8cb81f8c99f734a8d" Jul 17 01:57:29.372087 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:29.372052 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw"] Jul 17 01:57:29.379048 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:29.379024 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-b4cfw"] Jul 17 01:57:31.167642 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:31.167610 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" path="/var/lib/kubelet/pods/fa5a010e-8ecc-496c-8e3d-cc7c987e03a1/volumes" Jul 17 01:57:33.165205 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:33.165173 2583 scope.go:117] "RemoveContainer" containerID="6aa266b85fda6b5e571f797a9122f1b1663b6812a6e4aac973641c6b341b1d0c" Jul 17 01:57:34.376707 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:34.376677 2583 generic.go:356] "Generic (PLEG): container finished" podID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" containerID="68f825fe43a7d45ef0180fe35c40b9838441d6e0c9773bf4babeaa2d3ed2a09c" exitCode=6 Jul 17 01:57:34.377173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:34.376732 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" event={"ID":"aa9b4dde-25ab-47b9-a350-440343a0c6b1","Type":"ContainerDied","Data":"68f825fe43a7d45ef0180fe35c40b9838441d6e0c9773bf4babeaa2d3ed2a09c"} Jul 17 01:57:34.377173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:34.376793 2583 scope.go:117] "RemoveContainer" containerID="6aa266b85fda6b5e571f797a9122f1b1663b6812a6e4aac973641c6b341b1d0c" Jul 17 01:57:34.377173 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:34.377090 2583 scope.go:117] "RemoveContainer" containerID="68f825fe43a7d45ef0180fe35c40b9838441d6e0c9773bf4babeaa2d3ed2a09c" Jul 17 01:57:34.377367 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:57:34.377342 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9_odh-ai-gateway-infra(aa9b4dde-25ab-47b9-a350-440343a0c6b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" podUID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" Jul 17 01:57:35.411527 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:35.411493 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9"] Jul 17 01:57:35.535063 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:35.535037 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" Jul 17 01:57:35.694121 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:35.694003 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9b4dde-25ab-47b9-a350-440343a0c6b1-maas-api-service-ca\") pod \"aa9b4dde-25ab-47b9-a350-440343a0c6b1\" (UID: \"aa9b4dde-25ab-47b9-a350-440343a0c6b1\") " Jul 17 01:57:35.694454 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:35.694422 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9b4dde-25ab-47b9-a350-440343a0c6b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa9b4dde-25ab-47b9-a350-440343a0c6b1" (UID: "aa9b4dde-25ab-47b9-a350-440343a0c6b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:57:35.795620 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:35.795585 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9b4dde-25ab-47b9-a350-440343a0c6b1-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:57:36.388582 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:36.388550 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" event={"ID":"aa9b4dde-25ab-47b9-a350-440343a0c6b1","Type":"ContainerDied","Data":"ea1824b54f8bb56c921d9074d3e643361810b58c098ebe25bcbf8a4323a10496"} Jul 17 01:57:36.388582 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:36.388576 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9" Jul 17 01:57:36.388815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:36.388610 2583 scope.go:117] "RemoveContainer" containerID="68f825fe43a7d45ef0180fe35c40b9838441d6e0c9773bf4babeaa2d3ed2a09c" Jul 17 01:57:36.409841 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:36.409802 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9"] Jul 17 01:57:36.415620 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:36.415593 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-h9ht9"] Jul 17 01:57:37.168145 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:37.168087 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" path="/var/lib/kubelet/pods/aa9b4dde-25ab-47b9-a350-440343a0c6b1/volumes" Jul 17 01:57:39.162272 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:39.162240 2583 scope.go:117] "RemoveContainer" containerID="18449f548b4940dcd8b636500c1b2e3b190964b21ea6658702f63be9e459cc37" Jul 17 01:57:39.404493 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:39.404456 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" event={"ID":"6c7eb407-b314-4040-95ab-b2a36312c19d","Type":"ContainerStarted","Data":"c04848687b86f1f5d591ff92fa2ebc02049f1f840fbabb23b0b723938dcef840"} Jul 17 01:57:40.412517 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:40.412485 2583 generic.go:356] "Generic (PLEG): container finished" podID="6c7eb407-b314-4040-95ab-b2a36312c19d" containerID="c04848687b86f1f5d591ff92fa2ebc02049f1f840fbabb23b0b723938dcef840" exitCode=6 Jul 17 01:57:40.412954 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:40.412554 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" event={"ID":"6c7eb407-b314-4040-95ab-b2a36312c19d","Type":"ContainerDied","Data":"c04848687b86f1f5d591ff92fa2ebc02049f1f840fbabb23b0b723938dcef840"} Jul 17 01:57:40.412954 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:40.412621 2583 scope.go:117] "RemoveContainer" containerID="18449f548b4940dcd8b636500c1b2e3b190964b21ea6658702f63be9e459cc37" Jul 17 01:57:40.413063 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:40.412957 2583 scope.go:117] "RemoveContainer" containerID="c04848687b86f1f5d591ff92fa2ebc02049f1f840fbabb23b0b723938dcef840" Jul 17 01:57:40.413195 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:57:40.413172 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj_odh-ai-gateway-infra(6c7eb407-b314-4040-95ab-b2a36312c19d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" podUID="6c7eb407-b314-4040-95ab-b2a36312c19d" Jul 17 01:57:40.428991 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:40.428964 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj"] Jul 17 01:57:40.746630 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:40.746538 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:57:41.553194 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:41.553166 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" Jul 17 01:57:41.649925 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:41.649888 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7eb407-b314-4040-95ab-b2a36312c19d-maas-api-service-ca\") pod \"6c7eb407-b314-4040-95ab-b2a36312c19d\" (UID: \"6c7eb407-b314-4040-95ab-b2a36312c19d\") " Jul 17 01:57:41.650304 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:41.650281 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c7eb407-b314-4040-95ab-b2a36312c19d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c7eb407-b314-4040-95ab-b2a36312c19d" (UID: "6c7eb407-b314-4040-95ab-b2a36312c19d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:57:41.751544 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:41.751456 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c7eb407-b314-4040-95ab-b2a36312c19d-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:57:42.007465 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:42.007383 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:57:42.425109 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:42.425066 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" Jul 17 01:57:42.425109 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:42.425072 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj" event={"ID":"6c7eb407-b314-4040-95ab-b2a36312c19d","Type":"ContainerDied","Data":"63f9acc603848f4da98913962f520c4456436c42f4ce091b90db26de73d79e05"} Jul 17 01:57:42.425370 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:42.425162 2583 scope.go:117] "RemoveContainer" containerID="c04848687b86f1f5d591ff92fa2ebc02049f1f840fbabb23b0b723938dcef840" Jul 17 01:57:42.450688 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:42.450649 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj"] Jul 17 01:57:42.454835 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:42.454794 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-wxvbj"] Jul 17 01:57:43.168214 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:43.168178 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c7eb407-b314-4040-95ab-b2a36312c19d" path="/var/lib/kubelet/pods/6c7eb407-b314-4040-95ab-b2a36312c19d/volumes" Jul 17 01:57:47.745309 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:47.745269 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:57:47.876402 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:47.876368 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:57:50.115203 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115168 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4"] Jul 17 01:57:50.115624 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115607 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c7eb407-b314-4040-95ab-b2a36312c19d" containerName="revoke-keys" Jul 17 01:57:50.115668 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115627 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c7eb407-b314-4040-95ab-b2a36312c19d" containerName="revoke-keys" Jul 17 01:57:50.115668 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115636 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" containerName="revoke-keys" Jul 17 01:57:50.115668 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115642 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" containerName="revoke-keys" Jul 17 01:57:50.115668 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115649 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" containerName="revoke-keys" Jul 17 01:57:50.115668 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115655 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" containerName="revoke-keys" Jul 17 01:57:50.115668 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115664 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c7eb407-b314-4040-95ab-b2a36312c19d" containerName="revoke-keys" Jul 17 01:57:50.115668 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115669 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c7eb407-b314-4040-95ab-b2a36312c19d" containerName="revoke-keys" Jul 17 01:57:50.115878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115679 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" containerName="revoke-keys" Jul 17 01:57:50.115878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115684 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" containerName="revoke-keys" Jul 17 01:57:50.115878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115696 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" containerName="revoke-keys" Jul 17 01:57:50.115878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115701 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" containerName="revoke-keys" Jul 17 01:57:50.115878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115710 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" containerName="revoke-keys" Jul 17 01:57:50.115878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115715 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" containerName="revoke-keys" Jul 17 01:57:50.115878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115764 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" containerName="revoke-keys" Jul 17 01:57:50.115878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115771 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c7eb407-b314-4040-95ab-b2a36312c19d" containerName="revoke-keys" Jul 17 01:57:50.115878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115776 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" containerName="revoke-keys" Jul 17 01:57:50.115878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115784 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c7eb407-b314-4040-95ab-b2a36312c19d" containerName="revoke-keys" Jul 17 01:57:50.115878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115789 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad02b99c-eda5-4a2b-8968-96b647e6b188" containerName="revoke-keys" Jul 17 01:57:50.115878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115795 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" containerName="revoke-keys" Jul 17 01:57:50.115878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115799 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c7eb407-b314-4040-95ab-b2a36312c19d" containerName="revoke-keys" Jul 17 01:57:50.115878 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.115804 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" containerName="revoke-keys" Jul 17 01:57:50.119723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.119703 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" Jul 17 01:57:50.125472 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.125444 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:57:50.126061 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.125785 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:57:50.148075 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.148046 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4"] Jul 17 01:57:50.228911 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.228869 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/258b32c6-562f-43e0-a99b-9d87cedc690a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4\" (UID: \"258b32c6-562f-43e0-a99b-9d87cedc690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" Jul 17 01:57:50.329723 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.329690 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/258b32c6-562f-43e0-a99b-9d87cedc690a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4\" (UID: \"258b32c6-562f-43e0-a99b-9d87cedc690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" Jul 17 01:57:50.330334 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.330310 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/258b32c6-562f-43e0-a99b-9d87cedc690a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4\" (UID: \"258b32c6-562f-43e0-a99b-9d87cedc690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" Jul 17 01:57:50.429395 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.429305 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" Jul 17 01:57:50.578759 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:50.578726 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4"] Jul 17 01:57:50.582382 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:57:50.582353 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258b32c6_562f_43e0_a99b_9d87cedc690a.slice/crio-f11a21328dccaa7b03ba174d2e98b877b9dc1ae1acdc0e4203fad889644e3e86 WatchSource:0}: Error finding container f11a21328dccaa7b03ba174d2e98b877b9dc1ae1acdc0e4203fad889644e3e86: Status 404 returned error can't find the container with id f11a21328dccaa7b03ba174d2e98b877b9dc1ae1acdc0e4203fad889644e3e86 Jul 17 01:57:51.470112 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:51.470066 2583 generic.go:356] "Generic (PLEG): container finished" podID="258b32c6-562f-43e0-a99b-9d87cedc690a" containerID="3710af945e8352991f55117e669e259b6bb33041baadf5dc63d4ddfbcd0723e2" exitCode=6 Jul 17 01:57:51.470530 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:51.470125 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" event={"ID":"258b32c6-562f-43e0-a99b-9d87cedc690a","Type":"ContainerDied","Data":"3710af945e8352991f55117e669e259b6bb33041baadf5dc63d4ddfbcd0723e2"} Jul 17 01:57:51.470530 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:51.470186 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" event={"ID":"258b32c6-562f-43e0-a99b-9d87cedc690a","Type":"ContainerStarted","Data":"f11a21328dccaa7b03ba174d2e98b877b9dc1ae1acdc0e4203fad889644e3e86"} Jul 17 01:57:51.470530 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:51.470376 2583 scope.go:117] "RemoveContainer" containerID="3710af945e8352991f55117e669e259b6bb33041baadf5dc63d4ddfbcd0723e2" Jul 17 01:57:52.085444 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:52.085416 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:57:52.265649 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:52.265615 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:57:52.477822 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:52.477736 2583 generic.go:356] "Generic (PLEG): container finished" podID="258b32c6-562f-43e0-a99b-9d87cedc690a" containerID="bc926bc52e0219169ad63e613bd0449b41427366e9ae9f6e2cdd1af98e987676" exitCode=6 Jul 17 01:57:52.477822 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:52.477780 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" event={"ID":"258b32c6-562f-43e0-a99b-9d87cedc690a","Type":"ContainerDied","Data":"bc926bc52e0219169ad63e613bd0449b41427366e9ae9f6e2cdd1af98e987676"} Jul 17 01:57:52.478280 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:52.477829 2583 scope.go:117] "RemoveContainer" containerID="3710af945e8352991f55117e669e259b6bb33041baadf5dc63d4ddfbcd0723e2" Jul 17 01:57:52.478280 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:52.478081 2583 scope.go:117] "RemoveContainer" containerID="bc926bc52e0219169ad63e613bd0449b41427366e9ae9f6e2cdd1af98e987676" Jul 17 01:57:52.478361 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:57:52.478314 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4_odh-ai-gateway-infra(258b32c6-562f-43e0-a99b-9d87cedc690a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" podUID="258b32c6-562f-43e0-a99b-9d87cedc690a" Jul 17 01:57:53.435227 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:53.435195 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:57:53.484445 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:53.484416 2583 scope.go:117] "RemoveContainer" containerID="bc926bc52e0219169ad63e613bd0449b41427366e9ae9f6e2cdd1af98e987676" Jul 17 01:57:53.484938 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:57:53.484593 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4_odh-ai-gateway-infra(258b32c6-562f-43e0-a99b-9d87cedc690a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" podUID="258b32c6-562f-43e0-a99b-9d87cedc690a" Jul 17 01:57:53.577677 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:53.577647 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:57:54.466570 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:54.466536 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:57:54.706852 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:54.706810 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:57:56.208692 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:57:56.208654 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:01.810629 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:01.810595 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:01.907823 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:01.907786 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:02.476395 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:02.476367 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:04.163007 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:04.162976 2583 scope.go:117] "RemoveContainer" containerID="bc926bc52e0219169ad63e613bd0449b41427366e9ae9f6e2cdd1af98e987676" Jul 17 01:58:04.538289 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:04.538254 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" event={"ID":"258b32c6-562f-43e0-a99b-9d87cedc690a","Type":"ContainerStarted","Data":"84b621604a01e6b84eb1fb55a405a16d356b24a347baa4b87f5a18fe8c5c9b69"} Jul 17 01:58:04.556680 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:04.556629 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" podStartSLOduration=14.556612746 podStartE2EDuration="14.556612746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:58:04.554905272 +0000 UTC m=+2641.950520123" watchObservedRunningTime="2026-07-17 01:58:04.556612746 +0000 UTC m=+2641.952227598" Jul 17 01:58:05.545255 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:05.545225 2583 generic.go:356] "Generic (PLEG): container finished" podID="258b32c6-562f-43e0-a99b-9d87cedc690a" containerID="84b621604a01e6b84eb1fb55a405a16d356b24a347baa4b87f5a18fe8c5c9b69" exitCode=6 Jul 17 01:58:05.545653 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:05.545304 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" event={"ID":"258b32c6-562f-43e0-a99b-9d87cedc690a","Type":"ContainerDied","Data":"84b621604a01e6b84eb1fb55a405a16d356b24a347baa4b87f5a18fe8c5c9b69"} Jul 17 01:58:05.545653 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:05.545370 2583 scope.go:117] "RemoveContainer" containerID="bc926bc52e0219169ad63e613bd0449b41427366e9ae9f6e2cdd1af98e987676" Jul 17 01:58:05.545777 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:05.545682 2583 scope.go:117] "RemoveContainer" containerID="84b621604a01e6b84eb1fb55a405a16d356b24a347baa4b87f5a18fe8c5c9b69" Jul 17 01:58:05.545926 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:58:05.545899 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4_odh-ai-gateway-infra(258b32c6-562f-43e0-a99b-9d87cedc690a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" podUID="258b32c6-562f-43e0-a99b-9d87cedc690a" Jul 17 01:58:05.573704 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:05.573657 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4"] Jul 17 01:58:06.681815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:06.681787 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" Jul 17 01:58:06.773322 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:06.773290 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/258b32c6-562f-43e0-a99b-9d87cedc690a-maas-api-service-ca\") pod \"258b32c6-562f-43e0-a99b-9d87cedc690a\" (UID: \"258b32c6-562f-43e0-a99b-9d87cedc690a\") " Jul 17 01:58:06.773695 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:06.773671 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258b32c6-562f-43e0-a99b-9d87cedc690a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "258b32c6-562f-43e0-a99b-9d87cedc690a" (UID: "258b32c6-562f-43e0-a99b-9d87cedc690a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:58:06.874773 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:06.874690 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/258b32c6-562f-43e0-a99b-9d87cedc690a-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:58:07.558092 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:07.558062 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" Jul 17 01:58:07.558092 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:07.558072 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4" event={"ID":"258b32c6-562f-43e0-a99b-9d87cedc690a","Type":"ContainerDied","Data":"f11a21328dccaa7b03ba174d2e98b877b9dc1ae1acdc0e4203fad889644e3e86"} Jul 17 01:58:07.558346 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:07.558153 2583 scope.go:117] "RemoveContainer" containerID="84b621604a01e6b84eb1fb55a405a16d356b24a347baa4b87f5a18fe8c5c9b69" Jul 17 01:58:07.577925 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:07.577895 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4"] Jul 17 01:58:07.580261 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:07.580236 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-rjjc4"] Jul 17 01:58:08.195575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.195541 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v"] Jul 17 01:58:08.195954 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.195850 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c7eb407-b314-4040-95ab-b2a36312c19d" containerName="revoke-keys" Jul 17 01:58:08.195954 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.195860 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c7eb407-b314-4040-95ab-b2a36312c19d" containerName="revoke-keys" Jul 17 01:58:08.195954 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.195871 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="258b32c6-562f-43e0-a99b-9d87cedc690a" containerName="revoke-keys" Jul 17 01:58:08.195954 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.195876 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="258b32c6-562f-43e0-a99b-9d87cedc690a" containerName="revoke-keys" Jul 17 01:58:08.195954 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.195887 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" containerName="revoke-keys" Jul 17 01:58:08.195954 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.195893 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" containerName="revoke-keys" Jul 17 01:58:08.195954 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.195900 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="258b32c6-562f-43e0-a99b-9d87cedc690a" containerName="revoke-keys" Jul 17 01:58:08.195954 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.195904 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="258b32c6-562f-43e0-a99b-9d87cedc690a" containerName="revoke-keys" Jul 17 01:58:08.195954 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.195958 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa5a010e-8ecc-496c-8e3d-cc7c987e03a1" containerName="revoke-keys" Jul 17 01:58:08.196269 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.195967 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="258b32c6-562f-43e0-a99b-9d87cedc690a" containerName="revoke-keys" Jul 17 01:58:08.196269 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.195973 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="258b32c6-562f-43e0-a99b-9d87cedc690a" containerName="revoke-keys" Jul 17 01:58:08.196269 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.195980 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9b4dde-25ab-47b9-a350-440343a0c6b1" containerName="revoke-keys" Jul 17 01:58:08.200081 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.200061 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" Jul 17 01:58:08.202376 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.202357 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:58:08.202476 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.202439 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:58:08.205702 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.205666 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v"] Jul 17 01:58:08.286658 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.286617 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93efbae0-f362-4a46-8921-0e6b67d7d4cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v\" (UID: \"93efbae0-f362-4a46-8921-0e6b67d7d4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" Jul 17 01:58:08.387666 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.387632 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93efbae0-f362-4a46-8921-0e6b67d7d4cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v\" (UID: \"93efbae0-f362-4a46-8921-0e6b67d7d4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" Jul 17 01:58:08.388277 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.388258 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93efbae0-f362-4a46-8921-0e6b67d7d4cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v\" (UID: \"93efbae0-f362-4a46-8921-0e6b67d7d4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" Jul 17 01:58:08.421416 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.421380 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:08.511864 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.511785 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" Jul 17 01:58:08.531353 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.531324 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:08.628057 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.628019 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:08.653385 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:08.653363 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v"] Jul 17 01:58:08.657233 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:58:08.657204 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93efbae0_f362_4a46_8921_0e6b67d7d4cb.slice/crio-b19681930380538b9ac247b18cf706cad8073458676087113f43dd8a423b0867 WatchSource:0}: Error finding container b19681930380538b9ac247b18cf706cad8073458676087113f43dd8a423b0867: Status 404 returned error can't find the container with id b19681930380538b9ac247b18cf706cad8073458676087113f43dd8a423b0867 Jul 17 01:58:09.167741 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:09.167648 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="258b32c6-562f-43e0-a99b-9d87cedc690a" path="/var/lib/kubelet/pods/258b32c6-562f-43e0-a99b-9d87cedc690a/volumes" Jul 17 01:58:09.570280 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:09.570241 2583 generic.go:356] "Generic (PLEG): container finished" podID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" containerID="053972dba7e44bf724677943dd62eab9832aa9d10eab443ebd189f648b0d87f4" exitCode=6 Jul 17 01:58:09.570743 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:09.570330 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" event={"ID":"93efbae0-f362-4a46-8921-0e6b67d7d4cb","Type":"ContainerDied","Data":"053972dba7e44bf724677943dd62eab9832aa9d10eab443ebd189f648b0d87f4"} Jul 17 01:58:09.570743 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:09.570408 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" event={"ID":"93efbae0-f362-4a46-8921-0e6b67d7d4cb","Type":"ContainerStarted","Data":"b19681930380538b9ac247b18cf706cad8073458676087113f43dd8a423b0867"} Jul 17 01:58:09.570743 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:09.570515 2583 scope.go:117] "RemoveContainer" containerID="053972dba7e44bf724677943dd62eab9832aa9d10eab443ebd189f648b0d87f4" Jul 17 01:58:10.576522 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:10.576487 2583 generic.go:356] "Generic (PLEG): container finished" podID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" containerID="f9005a900374c8e3cff0225fb3b264830ba559fbfc52aa57ed83c4617cce17e2" exitCode=6 Jul 17 01:58:10.576902 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:10.576577 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" event={"ID":"93efbae0-f362-4a46-8921-0e6b67d7d4cb","Type":"ContainerDied","Data":"f9005a900374c8e3cff0225fb3b264830ba559fbfc52aa57ed83c4617cce17e2"} Jul 17 01:58:10.576902 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:10.576635 2583 scope.go:117] "RemoveContainer" containerID="053972dba7e44bf724677943dd62eab9832aa9d10eab443ebd189f648b0d87f4" Jul 17 01:58:10.576902 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:10.576746 2583 scope.go:117] "RemoveContainer" containerID="f9005a900374c8e3cff0225fb3b264830ba559fbfc52aa57ed83c4617cce17e2" Jul 17 01:58:10.577049 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:58:10.576956 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v_odh-ai-gateway-infra(93efbae0-f362-4a46-8921-0e6b67d7d4cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" podUID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" Jul 17 01:58:11.582988 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:11.582962 2583 scope.go:117] "RemoveContainer" containerID="f9005a900374c8e3cff0225fb3b264830ba559fbfc52aa57ed83c4617cce17e2" Jul 17 01:58:11.583394 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:58:11.583177 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v_odh-ai-gateway-infra(93efbae0-f362-4a46-8921-0e6b67d7d4cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" podUID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" Jul 17 01:58:16.164778 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:16.164745 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg"] Jul 17 01:58:16.165193 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:16.165060 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="258b32c6-562f-43e0-a99b-9d87cedc690a" containerName="revoke-keys" Jul 17 01:58:16.165193 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:16.165070 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="258b32c6-562f-43e0-a99b-9d87cedc690a" containerName="revoke-keys" Jul 17 01:58:16.165193 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:16.165163 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="258b32c6-562f-43e0-a99b-9d87cedc690a" containerName="revoke-keys" Jul 17 01:58:16.169398 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:16.169378 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" Jul 17 01:58:16.177033 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:16.176899 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg"] Jul 17 01:58:16.260263 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:16.260225 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/266bf732-3bfb-425d-8da9-75ac8032b874-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg\" (UID: \"266bf732-3bfb-425d-8da9-75ac8032b874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" Jul 17 01:58:16.361480 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:16.361437 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/266bf732-3bfb-425d-8da9-75ac8032b874-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg\" (UID: \"266bf732-3bfb-425d-8da9-75ac8032b874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" Jul 17 01:58:16.362066 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:16.362042 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/266bf732-3bfb-425d-8da9-75ac8032b874-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg\" (UID: \"266bf732-3bfb-425d-8da9-75ac8032b874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" Jul 17 01:58:16.481084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:16.480994 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" Jul 17 01:58:16.617828 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:16.617800 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg"] Jul 17 01:58:16.620127 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:58:16.620082 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266bf732_3bfb_425d_8da9_75ac8032b874.slice/crio-d72630d43ee0689c5c92e9926de0881272b9ffb015b0bcbf88cf0b8145eeeedf WatchSource:0}: Error finding container d72630d43ee0689c5c92e9926de0881272b9ffb015b0bcbf88cf0b8145eeeedf: Status 404 returned error can't find the container with id d72630d43ee0689c5c92e9926de0881272b9ffb015b0bcbf88cf0b8145eeeedf Jul 17 01:58:17.613844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:17.613739 2583 generic.go:356] "Generic (PLEG): container finished" podID="266bf732-3bfb-425d-8da9-75ac8032b874" containerID="4d7fa3985ec86d1160db34510f1a0733e5466bcb14b33208382be0c9d287aa48" exitCode=6 Jul 17 01:58:17.613844 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:17.613830 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" event={"ID":"266bf732-3bfb-425d-8da9-75ac8032b874","Type":"ContainerDied","Data":"4d7fa3985ec86d1160db34510f1a0733e5466bcb14b33208382be0c9d287aa48"} Jul 17 01:58:17.614334 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:17.613887 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" event={"ID":"266bf732-3bfb-425d-8da9-75ac8032b874","Type":"ContainerStarted","Data":"d72630d43ee0689c5c92e9926de0881272b9ffb015b0bcbf88cf0b8145eeeedf"} Jul 17 01:58:17.614334 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:17.614013 2583 scope.go:117] "RemoveContainer" containerID="4d7fa3985ec86d1160db34510f1a0733e5466bcb14b33208382be0c9d287aa48" Jul 17 01:58:18.620598 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:18.620564 2583 generic.go:356] "Generic (PLEG): container finished" podID="266bf732-3bfb-425d-8da9-75ac8032b874" containerID="52846a648acb0cdfe5aeb2bac54e4fb9f60330572dbf0801c5795955d0f200e0" exitCode=6 Jul 17 01:58:18.621072 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:18.620621 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" event={"ID":"266bf732-3bfb-425d-8da9-75ac8032b874","Type":"ContainerDied","Data":"52846a648acb0cdfe5aeb2bac54e4fb9f60330572dbf0801c5795955d0f200e0"} Jul 17 01:58:18.621072 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:18.620670 2583 scope.go:117] "RemoveContainer" containerID="4d7fa3985ec86d1160db34510f1a0733e5466bcb14b33208382be0c9d287aa48" Jul 17 01:58:18.621072 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:18.620883 2583 scope.go:117] "RemoveContainer" containerID="52846a648acb0cdfe5aeb2bac54e4fb9f60330572dbf0801c5795955d0f200e0" Jul 17 01:58:18.621395 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:58:18.621120 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg_odh-ai-gateway-infra(266bf732-3bfb-425d-8da9-75ac8032b874)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" podUID="266bf732-3bfb-425d-8da9-75ac8032b874" Jul 17 01:58:19.110478 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:19.110447 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:19.459606 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:19.459522 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:19.632336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:19.632307 2583 scope.go:117] "RemoveContainer" containerID="52846a648acb0cdfe5aeb2bac54e4fb9f60330572dbf0801c5795955d0f200e0" Jul 17 01:58:19.632771 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:58:19.632474 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg_odh-ai-gateway-infra(266bf732-3bfb-425d-8da9-75ac8032b874)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" podUID="266bf732-3bfb-425d-8da9-75ac8032b874" Jul 17 01:58:27.162249 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:27.162216 2583 scope.go:117] "RemoveContainer" containerID="f9005a900374c8e3cff0225fb3b264830ba559fbfc52aa57ed83c4617cce17e2" Jul 17 01:58:27.670976 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:27.670943 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" event={"ID":"93efbae0-f362-4a46-8921-0e6b67d7d4cb","Type":"ContainerStarted","Data":"b74e7edbcc456b10cb1e673e5ad76814eb15bc95b314bee675e9357cd5af5c50"} Jul 17 01:58:27.686685 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:27.686572 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" podStartSLOduration=19.686556084 podStartE2EDuration="19.686556084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:58:27.68566838 +0000 UTC m=+2665.081283226" watchObservedRunningTime="2026-07-17 01:58:27.686556084 +0000 UTC m=+2665.082171004" Jul 17 01:58:28.677187 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:28.677149 2583 generic.go:356] "Generic (PLEG): container finished" podID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" containerID="b74e7edbcc456b10cb1e673e5ad76814eb15bc95b314bee675e9357cd5af5c50" exitCode=6 Jul 17 01:58:28.677631 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:28.677222 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" event={"ID":"93efbae0-f362-4a46-8921-0e6b67d7d4cb","Type":"ContainerDied","Data":"b74e7edbcc456b10cb1e673e5ad76814eb15bc95b314bee675e9357cd5af5c50"} Jul 17 01:58:28.677631 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:28.677284 2583 scope.go:117] "RemoveContainer" containerID="f9005a900374c8e3cff0225fb3b264830ba559fbfc52aa57ed83c4617cce17e2" Jul 17 01:58:28.677631 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:28.677533 2583 scope.go:117] "RemoveContainer" containerID="b74e7edbcc456b10cb1e673e5ad76814eb15bc95b314bee675e9357cd5af5c50" Jul 17 01:58:28.677757 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:58:28.677731 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v_odh-ai-gateway-infra(93efbae0-f362-4a46-8921-0e6b67d7d4cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" podUID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" Jul 17 01:58:28.698464 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:28.698436 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v"] Jul 17 01:58:29.815980 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:29.815956 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" Jul 17 01:58:29.884372 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:29.884324 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93efbae0-f362-4a46-8921-0e6b67d7d4cb-maas-api-service-ca\") pod \"93efbae0-f362-4a46-8921-0e6b67d7d4cb\" (UID: \"93efbae0-f362-4a46-8921-0e6b67d7d4cb\") " Jul 17 01:58:29.884681 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:29.884659 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93efbae0-f362-4a46-8921-0e6b67d7d4cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93efbae0-f362-4a46-8921-0e6b67d7d4cb" (UID: "93efbae0-f362-4a46-8921-0e6b67d7d4cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:58:29.985830 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:29.985747 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93efbae0-f362-4a46-8921-0e6b67d7d4cb-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:58:30.689631 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:30.689598 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" Jul 17 01:58:30.689835 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:30.689634 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v" event={"ID":"93efbae0-f362-4a46-8921-0e6b67d7d4cb","Type":"ContainerDied","Data":"b19681930380538b9ac247b18cf706cad8073458676087113f43dd8a423b0867"} Jul 17 01:58:30.689835 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:30.689705 2583 scope.go:117] "RemoveContainer" containerID="b74e7edbcc456b10cb1e673e5ad76814eb15bc95b314bee675e9357cd5af5c50" Jul 17 01:58:30.711996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:30.711967 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v"] Jul 17 01:58:30.719483 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:30.719458 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-7tb2v"] Jul 17 01:58:31.168237 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:31.168204 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" path="/var/lib/kubelet/pods/93efbae0-f362-4a46-8921-0e6b67d7d4cb/volumes" Jul 17 01:58:33.168087 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:33.168057 2583 scope.go:117] "RemoveContainer" containerID="52846a648acb0cdfe5aeb2bac54e4fb9f60330572dbf0801c5795955d0f200e0" Jul 17 01:58:33.705336 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:33.705304 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" event={"ID":"266bf732-3bfb-425d-8da9-75ac8032b874","Type":"ContainerStarted","Data":"666329b9190c759ed3e89f6c6bad9a9d25037c9f712e66c19fb2d8c84a975e9a"} Jul 17 01:58:33.722317 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:33.722268 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" podStartSLOduration=17.722251775 podStartE2EDuration="17.722251775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:58:33.719789396 +0000 UTC m=+2671.115404247" watchObservedRunningTime="2026-07-17 01:58:33.722251775 +0000 UTC m=+2671.117866631" Jul 17 01:58:34.711244 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:34.711214 2583 generic.go:356] "Generic (PLEG): container finished" podID="266bf732-3bfb-425d-8da9-75ac8032b874" containerID="666329b9190c759ed3e89f6c6bad9a9d25037c9f712e66c19fb2d8c84a975e9a" exitCode=6 Jul 17 01:58:34.711712 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:34.711285 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" event={"ID":"266bf732-3bfb-425d-8da9-75ac8032b874","Type":"ContainerDied","Data":"666329b9190c759ed3e89f6c6bad9a9d25037c9f712e66c19fb2d8c84a975e9a"} Jul 17 01:58:34.711712 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:34.711352 2583 scope.go:117] "RemoveContainer" containerID="52846a648acb0cdfe5aeb2bac54e4fb9f60330572dbf0801c5795955d0f200e0" Jul 17 01:58:34.711712 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:34.711664 2583 scope.go:117] "RemoveContainer" containerID="666329b9190c759ed3e89f6c6bad9a9d25037c9f712e66c19fb2d8c84a975e9a" Jul 17 01:58:34.711906 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:58:34.711884 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg_odh-ai-gateway-infra(266bf732-3bfb-425d-8da9-75ac8032b874)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" podUID="266bf732-3bfb-425d-8da9-75ac8032b874" Jul 17 01:58:34.729627 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:34.729602 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg"] Jul 17 01:58:35.858842 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:35.858816 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" Jul 17 01:58:35.938068 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:35.938026 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/266bf732-3bfb-425d-8da9-75ac8032b874-maas-api-service-ca\") pod \"266bf732-3bfb-425d-8da9-75ac8032b874\" (UID: \"266bf732-3bfb-425d-8da9-75ac8032b874\") " Jul 17 01:58:35.938456 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:35.938432 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266bf732-3bfb-425d-8da9-75ac8032b874-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "266bf732-3bfb-425d-8da9-75ac8032b874" (UID: "266bf732-3bfb-425d-8da9-75ac8032b874"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:58:36.039508 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:36.039473 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/266bf732-3bfb-425d-8da9-75ac8032b874-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:58:36.723239 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:36.723203 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" Jul 17 01:58:36.723239 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:36.723212 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg" event={"ID":"266bf732-3bfb-425d-8da9-75ac8032b874","Type":"ContainerDied","Data":"d72630d43ee0689c5c92e9926de0881272b9ffb015b0bcbf88cf0b8145eeeedf"} Jul 17 01:58:36.723507 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:36.723316 2583 scope.go:117] "RemoveContainer" containerID="666329b9190c759ed3e89f6c6bad9a9d25037c9f712e66c19fb2d8c84a975e9a" Jul 17 01:58:36.746777 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:36.746747 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg"] Jul 17 01:58:36.753928 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:36.753900 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xffgg"] Jul 17 01:58:37.167439 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:37.167409 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="266bf732-3bfb-425d-8da9-75ac8032b874" path="/var/lib/kubelet/pods/266bf732-3bfb-425d-8da9-75ac8032b874/volumes" Jul 17 01:58:40.440697 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:40.440665 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:45.998370 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:45.998338 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:46.101144 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:46.101082 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:46.367937 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:46.367890 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:46.733959 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:46.733871 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:47.098009 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:47.097978 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:49.707198 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.707158 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:49.786361 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.786141 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:49.786361 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.786353 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79"] Jul 17 01:58:49.786833 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.786801 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="266bf732-3bfb-425d-8da9-75ac8032b874" containerName="revoke-keys" Jul 17 01:58:49.786833 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.786822 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="266bf732-3bfb-425d-8da9-75ac8032b874" containerName="revoke-keys" Jul 17 01:58:49.787021 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.786842 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" containerName="revoke-keys" Jul 17 01:58:49.787021 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.786851 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" containerName="revoke-keys" Jul 17 01:58:49.787021 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.786874 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" containerName="revoke-keys" Jul 17 01:58:49.787021 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.786883 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" containerName="revoke-keys" Jul 17 01:58:49.787021 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.786900 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="266bf732-3bfb-425d-8da9-75ac8032b874" containerName="revoke-keys" Jul 17 01:58:49.787021 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.786908 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="266bf732-3bfb-425d-8da9-75ac8032b874" containerName="revoke-keys" Jul 17 01:58:49.787021 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.786988 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" containerName="revoke-keys" Jul 17 01:58:49.787021 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.787001 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" containerName="revoke-keys" Jul 17 01:58:49.787021 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.787014 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" containerName="revoke-keys" Jul 17 01:58:49.787021 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.787022 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="266bf732-3bfb-425d-8da9-75ac8032b874" containerName="revoke-keys" Jul 17 01:58:49.787527 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.787033 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="266bf732-3bfb-425d-8da9-75ac8032b874" containerName="revoke-keys" Jul 17 01:58:49.787527 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.787042 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="266bf732-3bfb-425d-8da9-75ac8032b874" containerName="revoke-keys" Jul 17 01:58:49.797486 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.797458 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:58:49.803505 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.803474 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5l8p8\"" Jul 17 01:58:49.803821 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.803801 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-089e15-serving-cert\"" Jul 17 01:58:49.804047 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.804030 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 01:58:49.804543 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.804364 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 01:58:49.817321 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.817290 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79"] Jul 17 01:58:49.857816 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.857780 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/256a2841-8135-4ecf-9c61-04fb8b0c9172-maas-api-tls\") pod \"maas-api-e2e-inf-a-089e15-65b8d89485-5nl79\" (UID: \"256a2841-8135-4ecf-9c61-04fb8b0c9172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:58:49.857816 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.857829 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/256a2841-8135-4ecf-9c61-04fb8b0c9172-openshift-service-ca\") pod \"maas-api-e2e-inf-a-089e15-65b8d89485-5nl79\" (UID: \"256a2841-8135-4ecf-9c61-04fb8b0c9172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:58:49.858136 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.857891 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5k9x\" (UniqueName: \"kubernetes.io/projected/256a2841-8135-4ecf-9c61-04fb8b0c9172-kube-api-access-c5k9x\") pod \"maas-api-e2e-inf-a-089e15-65b8d89485-5nl79\" (UID: \"256a2841-8135-4ecf-9c61-04fb8b0c9172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:58:49.858579 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.858559 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:49.958612 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.958520 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/256a2841-8135-4ecf-9c61-04fb8b0c9172-maas-api-tls\") pod \"maas-api-e2e-inf-a-089e15-65b8d89485-5nl79\" (UID: \"256a2841-8135-4ecf-9c61-04fb8b0c9172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:58:49.958612 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.958568 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/256a2841-8135-4ecf-9c61-04fb8b0c9172-openshift-service-ca\") pod \"maas-api-e2e-inf-a-089e15-65b8d89485-5nl79\" (UID: \"256a2841-8135-4ecf-9c61-04fb8b0c9172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:58:49.958612 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.958600 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5k9x\" (UniqueName: \"kubernetes.io/projected/256a2841-8135-4ecf-9c61-04fb8b0c9172-kube-api-access-c5k9x\") pod \"maas-api-e2e-inf-a-089e15-65b8d89485-5nl79\" (UID: \"256a2841-8135-4ecf-9c61-04fb8b0c9172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:58:49.958941 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:58:49.958688 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-serving-cert: secret "maas-api-e2e-inf-a-089e15-serving-cert" not found Jul 17 01:58:49.958941 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:58:49.958794 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256a2841-8135-4ecf-9c61-04fb8b0c9172-maas-api-tls podName:256a2841-8135-4ecf-9c61-04fb8b0c9172 nodeName:}" failed. No retries permitted until 2026-07-17 01:58:50.458764425 +0000 UTC m=+2687.854379257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/256a2841-8135-4ecf-9c61-04fb8b0c9172-maas-api-tls") pod "maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" (UID: "256a2841-8135-4ecf-9c61-04fb8b0c9172") : secret "maas-api-e2e-inf-a-089e15-serving-cert" not found Jul 17 01:58:49.959346 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.959329 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/256a2841-8135-4ecf-9c61-04fb8b0c9172-openshift-service-ca\") pod \"maas-api-e2e-inf-a-089e15-65b8d89485-5nl79\" (UID: \"256a2841-8135-4ecf-9c61-04fb8b0c9172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:58:49.968307 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:49.968280 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5k9x\" (UniqueName: \"kubernetes.io/projected/256a2841-8135-4ecf-9c61-04fb8b0c9172-kube-api-access-c5k9x\") pod \"maas-api-e2e-inf-a-089e15-65b8d89485-5nl79\" (UID: \"256a2841-8135-4ecf-9c61-04fb8b0c9172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:58:50.037018 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:50.036984 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:50.383082 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:50.383046 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:50.463039 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:50.462992 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/256a2841-8135-4ecf-9c61-04fb8b0c9172-maas-api-tls\") pod \"maas-api-e2e-inf-a-089e15-65b8d89485-5nl79\" (UID: \"256a2841-8135-4ecf-9c61-04fb8b0c9172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:58:50.465615 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:50.465595 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/256a2841-8135-4ecf-9c61-04fb8b0c9172-maas-api-tls\") pod \"maas-api-e2e-inf-a-089e15-65b8d89485-5nl79\" (UID: \"256a2841-8135-4ecf-9c61-04fb8b0c9172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:58:50.484493 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:50.484453 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:50.716754 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:50.716653 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:58:50.856369 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:50.856303 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79"] Jul 17 01:58:50.858828 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:58:50.858799 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256a2841_8135_4ecf_9c61_04fb8b0c9172.slice/crio-89ec9601a46e735a110e494d1e52aa8054055dd4336e1ec7fe7a591413287cb9 WatchSource:0}: Error finding container 89ec9601a46e735a110e494d1e52aa8054055dd4336e1ec7fe7a591413287cb9: Status 404 returned error can't find the container with id 89ec9601a46e735a110e494d1e52aa8054055dd4336e1ec7fe7a591413287cb9 Jul 17 01:58:51.798812 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:51.798769 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" event={"ID":"256a2841-8135-4ecf-9c61-04fb8b0c9172","Type":"ContainerStarted","Data":"183e6127e4e29f58437244e345f276b264420733cbd02f6c2679c28c4bae49fa"} Jul 17 01:58:51.799278 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:51.798827 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" event={"ID":"256a2841-8135-4ecf-9c61-04fb8b0c9172","Type":"ContainerStarted","Data":"89ec9601a46e735a110e494d1e52aa8054055dd4336e1ec7fe7a591413287cb9"} Jul 17 01:58:51.799278 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:51.798906 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:58:51.816725 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:51.816680 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" podStartSLOduration=2.431300987 podStartE2EDuration="2.816665088s" totalImagesPullingTime="385.364101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:58:51.815796461 +0000 UTC m=+2689.211411335" watchObservedRunningTime="2026-07-17 01:58:51.816665088 +0000 UTC m=+2689.212279953" Jul 17 01:58:51.833791 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:51.833757 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:54.215719 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:54.215643 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:54.553405 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:54.553370 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:54.856712 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:54.856624 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:55.147353 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:55.147273 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:57.809191 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:57.809156 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:58:58.125283 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.125200 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:58.232468 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.232439 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:58.265846 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.265809 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7"] Jul 17 01:58:58.266347 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.266325 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="266bf732-3bfb-425d-8da9-75ac8032b874" containerName="revoke-keys" Jul 17 01:58:58.266347 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.266348 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="266bf732-3bfb-425d-8da9-75ac8032b874" containerName="revoke-keys" Jul 17 01:58:58.266514 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.266367 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" containerName="revoke-keys" Jul 17 01:58:58.266514 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.266376 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="93efbae0-f362-4a46-8921-0e6b67d7d4cb" containerName="revoke-keys" Jul 17 01:58:58.270072 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.270049 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 01:58:58.272447 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.272432 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ab7f72-serving-cert\"" Jul 17 01:58:58.276930 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.276905 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7"] Jul 17 01:58:58.325304 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.325266 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:58.329452 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.329428 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b579fca9-f19f-4090-973d-f8bea3e360df-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7\" (UID: \"b579fca9-f19f-4090-973d-f8bea3e360df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 01:58:58.329558 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.329460 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b579fca9-f19f-4090-973d-f8bea3e360df-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7\" (UID: \"b579fca9-f19f-4090-973d-f8bea3e360df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 01:58:58.329558 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.329492 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppktv\" (UniqueName: \"kubernetes.io/projected/b579fca9-f19f-4090-973d-f8bea3e360df-kube-api-access-ppktv\") pod \"maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7\" (UID: \"b579fca9-f19f-4090-973d-f8bea3e360df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 01:58:58.430715 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.430620 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppktv\" (UniqueName: \"kubernetes.io/projected/b579fca9-f19f-4090-973d-f8bea3e360df-kube-api-access-ppktv\") pod \"maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7\" (UID: \"b579fca9-f19f-4090-973d-f8bea3e360df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 01:58:58.430715 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.430697 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b579fca9-f19f-4090-973d-f8bea3e360df-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7\" (UID: \"b579fca9-f19f-4090-973d-f8bea3e360df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 01:58:58.430715 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.430717 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b579fca9-f19f-4090-973d-f8bea3e360df-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7\" (UID: \"b579fca9-f19f-4090-973d-f8bea3e360df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 01:58:58.431028 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:58:58.430802 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-serving-cert: secret "maas-api-e2e-inf-b-ab7f72-serving-cert" not found Jul 17 01:58:58.431028 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:58:58.430858 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b579fca9-f19f-4090-973d-f8bea3e360df-maas-api-tls podName:b579fca9-f19f-4090-973d-f8bea3e360df nodeName:}" failed. No retries permitted until 2026-07-17 01:58:58.930842361 +0000 UTC m=+2696.326457190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b579fca9-f19f-4090-973d-f8bea3e360df-maas-api-tls") pod "maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" (UID: "b579fca9-f19f-4090-973d-f8bea3e360df") : secret "maas-api-e2e-inf-b-ab7f72-serving-cert" not found Jul 17 01:58:58.431449 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.431426 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b579fca9-f19f-4090-973d-f8bea3e360df-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7\" (UID: \"b579fca9-f19f-4090-973d-f8bea3e360df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 01:58:58.439740 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.439714 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppktv\" (UniqueName: \"kubernetes.io/projected/b579fca9-f19f-4090-973d-f8bea3e360df-kube-api-access-ppktv\") pod \"maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7\" (UID: \"b579fca9-f19f-4090-973d-f8bea3e360df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 01:58:58.803921 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.803887 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:58.904823 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.904794 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:58:58.934971 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.934930 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b579fca9-f19f-4090-973d-f8bea3e360df-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7\" (UID: \"b579fca9-f19f-4090-973d-f8bea3e360df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 01:58:58.937469 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:58.937446 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b579fca9-f19f-4090-973d-f8bea3e360df-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7\" (UID: \"b579fca9-f19f-4090-973d-f8bea3e360df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 01:58:59.183886 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:59.183797 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 01:58:59.316497 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:59.316436 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7"] Jul 17 01:58:59.318950 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:58:59.318917 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb579fca9_f19f_4090_973d_f8bea3e360df.slice/crio-d0e82c5bbbff306b7ba7f73704a01b95b5678eefb8cd15fb88cec3ea364f5e1b WatchSource:0}: Error finding container d0e82c5bbbff306b7ba7f73704a01b95b5678eefb8cd15fb88cec3ea364f5e1b: Status 404 returned error can't find the container with id d0e82c5bbbff306b7ba7f73704a01b95b5678eefb8cd15fb88cec3ea364f5e1b Jul 17 01:58:59.837587 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:59.837555 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" event={"ID":"b579fca9-f19f-4090-973d-f8bea3e360df","Type":"ContainerStarted","Data":"b0c4cf132cff540574ddb33456dd8db42fa739e748f05865206c7cbb10cbb1d8"} Jul 17 01:58:59.837770 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:59.837606 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" event={"ID":"b579fca9-f19f-4090-973d-f8bea3e360df","Type":"ContainerStarted","Data":"d0e82c5bbbff306b7ba7f73704a01b95b5678eefb8cd15fb88cec3ea364f5e1b"} Jul 17 01:58:59.837770 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:59.837669 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 01:58:59.860085 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:58:59.860031 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" podStartSLOduration=1.545318709 podStartE2EDuration="1.860014356s" totalImagesPullingTime="314.695647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:58:59.858481443 +0000 UTC m=+2697.254096295" watchObservedRunningTime="2026-07-17 01:58:59.860014356 +0000 UTC m=+2697.255629244" Jul 17 01:59:02.642339 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:02.642303 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:59:03.251267 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:03.251239 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:59:03.260971 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:03.260941 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 01:59:05.848423 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:05.848392 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 01:59:08.241938 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:08.241896 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff"] Jul 17 01:59:08.246775 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:08.246755 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" Jul 17 01:59:08.249034 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:08.249017 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 01:59:08.252204 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:08.252179 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff"] Jul 17 01:59:08.322062 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:08.322022 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9aa9e8f-541d-43a9-abba-203cc9c08779-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff\" (UID: \"c9aa9e8f-541d-43a9-abba-203cc9c08779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" Jul 17 01:59:08.423430 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:08.423390 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9aa9e8f-541d-43a9-abba-203cc9c08779-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff\" (UID: \"c9aa9e8f-541d-43a9-abba-203cc9c08779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" Jul 17 01:59:08.424018 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:08.424000 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9aa9e8f-541d-43a9-abba-203cc9c08779-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff\" (UID: \"c9aa9e8f-541d-43a9-abba-203cc9c08779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" Jul 17 01:59:08.559170 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:08.559126 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" Jul 17 01:59:08.694159 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:08.694134 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff"] Jul 17 01:59:08.696279 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:59:08.696251 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9aa9e8f_541d_43a9_abba_203cc9c08779.slice/crio-94f784aabae79252f4d7c0879cad10f2278ecc8c56dd841ca88f192f1a396038 WatchSource:0}: Error finding container 94f784aabae79252f4d7c0879cad10f2278ecc8c56dd841ca88f192f1a396038: Status 404 returned error can't find the container with id 94f784aabae79252f4d7c0879cad10f2278ecc8c56dd841ca88f192f1a396038 Jul 17 01:59:08.881379 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:08.881289 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" event={"ID":"c9aa9e8f-541d-43a9-abba-203cc9c08779","Type":"ContainerStarted","Data":"94f784aabae79252f4d7c0879cad10f2278ecc8c56dd841ca88f192f1a396038"} Jul 17 01:59:09.888158 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:09.888126 2583 generic.go:356] "Generic (PLEG): container finished" podID="c9aa9e8f-541d-43a9-abba-203cc9c08779" containerID="0b0b48e0e5c896532c9298580350d18ee1ead8c73351ce5abafee8d7dae14a70" exitCode=6 Jul 17 01:59:09.888548 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:09.888220 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" event={"ID":"c9aa9e8f-541d-43a9-abba-203cc9c08779","Type":"ContainerDied","Data":"0b0b48e0e5c896532c9298580350d18ee1ead8c73351ce5abafee8d7dae14a70"} Jul 17 01:59:09.888548 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:09.888476 2583 scope.go:117] "RemoveContainer" containerID="0b0b48e0e5c896532c9298580350d18ee1ead8c73351ce5abafee8d7dae14a70" Jul 17 01:59:10.895701 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:10.895605 2583 generic.go:356] "Generic (PLEG): container finished" podID="c9aa9e8f-541d-43a9-abba-203cc9c08779" containerID="2ba970bfffa14539847a01063576a7bddb045d38a94a621aa291e5257896cdfe" exitCode=6 Jul 17 01:59:10.895701 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:10.895681 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" event={"ID":"c9aa9e8f-541d-43a9-abba-203cc9c08779","Type":"ContainerDied","Data":"2ba970bfffa14539847a01063576a7bddb045d38a94a621aa291e5257896cdfe"} Jul 17 01:59:10.896229 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:10.895767 2583 scope.go:117] "RemoveContainer" containerID="0b0b48e0e5c896532c9298580350d18ee1ead8c73351ce5abafee8d7dae14a70" Jul 17 01:59:10.896229 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:10.895991 2583 scope.go:117] "RemoveContainer" containerID="2ba970bfffa14539847a01063576a7bddb045d38a94a621aa291e5257896cdfe" Jul 17 01:59:10.896336 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:59:10.896247 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff_odh-ai-gateway-infra(c9aa9e8f-541d-43a9-abba-203cc9c08779)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" podUID="c9aa9e8f-541d-43a9-abba-203cc9c08779" Jul 17 01:59:11.101002 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:11.100965 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:59:11.200508 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:11.200430 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:59:11.266251 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:11.266221 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:59:11.902370 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:11.902340 2583 scope.go:117] "RemoveContainer" containerID="2ba970bfffa14539847a01063576a7bddb045d38a94a621aa291e5257896cdfe" Jul 17 01:59:11.902759 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:59:11.902565 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff_odh-ai-gateway-infra(c9aa9e8f-541d-43a9-abba-203cc9c08779)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" podUID="c9aa9e8f-541d-43a9-abba-203cc9c08779" Jul 17 01:59:13.621542 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:13.621504 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt"] Jul 17 01:59:13.626065 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:13.626045 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" Jul 17 01:59:13.633768 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:13.633736 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt"] Jul 17 01:59:13.774226 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:13.774191 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e3d99d-84d2-4e28-9c92-aa8755145e65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt\" (UID: \"70e3d99d-84d2-4e28-9c92-aa8755145e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" Jul 17 01:59:13.875230 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:13.875140 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e3d99d-84d2-4e28-9c92-aa8755145e65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt\" (UID: \"70e3d99d-84d2-4e28-9c92-aa8755145e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" Jul 17 01:59:13.875781 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:13.875759 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e3d99d-84d2-4e28-9c92-aa8755145e65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt\" (UID: \"70e3d99d-84d2-4e28-9c92-aa8755145e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" Jul 17 01:59:13.937835 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:13.937785 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" Jul 17 01:59:14.073242 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:14.073118 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt"] Jul 17 01:59:14.075949 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:59:14.075918 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e3d99d_84d2_4e28_9c92_aa8755145e65.slice/crio-45bf1feb88cad367d226b727833b4423a62d5ccf14fd14f06169c8a49181f892 WatchSource:0}: Error finding container 45bf1feb88cad367d226b727833b4423a62d5ccf14fd14f06169c8a49181f892: Status 404 returned error can't find the container with id 45bf1feb88cad367d226b727833b4423a62d5ccf14fd14f06169c8a49181f892 Jul 17 01:59:14.918502 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:14.918461 2583 generic.go:356] "Generic (PLEG): container finished" podID="70e3d99d-84d2-4e28-9c92-aa8755145e65" containerID="2b4964f428aa2f6ae3395d25947265a7246da180d83915e4bd0314a3d5a0bddd" exitCode=6 Jul 17 01:59:14.918502 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:14.918496 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" event={"ID":"70e3d99d-84d2-4e28-9c92-aa8755145e65","Type":"ContainerDied","Data":"2b4964f428aa2f6ae3395d25947265a7246da180d83915e4bd0314a3d5a0bddd"} Jul 17 01:59:14.918996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:14.918552 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" event={"ID":"70e3d99d-84d2-4e28-9c92-aa8755145e65","Type":"ContainerStarted","Data":"45bf1feb88cad367d226b727833b4423a62d5ccf14fd14f06169c8a49181f892"} Jul 17 01:59:14.918996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:14.918760 2583 scope.go:117] "RemoveContainer" containerID="2b4964f428aa2f6ae3395d25947265a7246da180d83915e4bd0314a3d5a0bddd" Jul 17 01:59:15.925428 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:15.925394 2583 generic.go:356] "Generic (PLEG): container finished" podID="70e3d99d-84d2-4e28-9c92-aa8755145e65" containerID="c2485b847655a3b058d5dcf78e107e9e6a77577b8bd6cccb6045a9185138a607" exitCode=6 Jul 17 01:59:15.925891 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:15.925463 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" event={"ID":"70e3d99d-84d2-4e28-9c92-aa8755145e65","Type":"ContainerDied","Data":"c2485b847655a3b058d5dcf78e107e9e6a77577b8bd6cccb6045a9185138a607"} Jul 17 01:59:15.925891 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:15.925505 2583 scope.go:117] "RemoveContainer" containerID="2b4964f428aa2f6ae3395d25947265a7246da180d83915e4bd0314a3d5a0bddd" Jul 17 01:59:15.925891 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:15.925693 2583 scope.go:117] "RemoveContainer" containerID="c2485b847655a3b058d5dcf78e107e9e6a77577b8bd6cccb6045a9185138a607" Jul 17 01:59:15.926017 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:59:15.925897 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt_odh-ai-gateway-infra(70e3d99d-84d2-4e28-9c92-aa8755145e65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" podUID="70e3d99d-84d2-4e28-9c92-aa8755145e65" Jul 17 01:59:16.206551 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:16.206465 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n"] Jul 17 01:59:16.211320 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:16.211299 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" Jul 17 01:59:16.216154 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:16.216121 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n"] Jul 17 01:59:16.399069 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:16.399033 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8e6a61-7a1b-4f29-852a-7dc6183deba2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n\" (UID: \"7e8e6a61-7a1b-4f29-852a-7dc6183deba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" Jul 17 01:59:16.499933 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:16.499838 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8e6a61-7a1b-4f29-852a-7dc6183deba2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n\" (UID: \"7e8e6a61-7a1b-4f29-852a-7dc6183deba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" Jul 17 01:59:16.500526 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:16.500507 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8e6a61-7a1b-4f29-852a-7dc6183deba2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n\" (UID: \"7e8e6a61-7a1b-4f29-852a-7dc6183deba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" Jul 17 01:59:16.523334 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:16.523297 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" Jul 17 01:59:16.659515 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:16.659491 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n"] Jul 17 01:59:16.662293 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:59:16.662248 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8e6a61_7a1b_4f29_852a_7dc6183deba2.slice/crio-185cb5327c1c0ee3a0fc6d01cca52e0308b05263ba4e070679696dace4b8c9bc WatchSource:0}: Error finding container 185cb5327c1c0ee3a0fc6d01cca52e0308b05263ba4e070679696dace4b8c9bc: Status 404 returned error can't find the container with id 185cb5327c1c0ee3a0fc6d01cca52e0308b05263ba4e070679696dace4b8c9bc Jul 17 01:59:16.931463 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:16.931427 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" event={"ID":"7e8e6a61-7a1b-4f29-852a-7dc6183deba2","Type":"ContainerStarted","Data":"aa6db574ca02498f4655f174c39bb38a0250ec9220dfee452782c875cbbb4d9c"} Jul 17 01:59:16.931892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:16.931494 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" event={"ID":"7e8e6a61-7a1b-4f29-852a-7dc6183deba2","Type":"ContainerStarted","Data":"185cb5327c1c0ee3a0fc6d01cca52e0308b05263ba4e070679696dace4b8c9bc"} Jul 17 01:59:16.933030 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:16.933012 2583 scope.go:117] "RemoveContainer" containerID="c2485b847655a3b058d5dcf78e107e9e6a77577b8bd6cccb6045a9185138a607" Jul 17 01:59:16.933197 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:59:16.933177 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt_odh-ai-gateway-infra(70e3d99d-84d2-4e28-9c92-aa8755145e65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" podUID="70e3d99d-84d2-4e28-9c92-aa8755145e65" Jul 17 01:59:16.950656 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:16.950605 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" podStartSLOduration=0.950589481 podStartE2EDuration="950.589481ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:59:16.94866362 +0000 UTC m=+2714.344278471" watchObservedRunningTime="2026-07-17 01:59:16.950589481 +0000 UTC m=+2714.346204332" Jul 17 01:59:17.940976 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:17.940941 2583 generic.go:356] "Generic (PLEG): container finished" podID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" containerID="aa6db574ca02498f4655f174c39bb38a0250ec9220dfee452782c875cbbb4d9c" exitCode=6 Jul 17 01:59:17.941420 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:17.941026 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" event={"ID":"7e8e6a61-7a1b-4f29-852a-7dc6183deba2","Type":"ContainerDied","Data":"aa6db574ca02498f4655f174c39bb38a0250ec9220dfee452782c875cbbb4d9c"} Jul 17 01:59:17.941420 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:17.941201 2583 scope.go:117] "RemoveContainer" containerID="aa6db574ca02498f4655f174c39bb38a0250ec9220dfee452782c875cbbb4d9c" Jul 17 01:59:18.947385 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:18.947353 2583 generic.go:356] "Generic (PLEG): container finished" podID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" containerID="6734dc374889b035a4921dc11495b902f0f8ef9abd2f9c9a41a0f114b68e34a2" exitCode=6 Jul 17 01:59:18.947789 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:18.947416 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" event={"ID":"7e8e6a61-7a1b-4f29-852a-7dc6183deba2","Type":"ContainerDied","Data":"6734dc374889b035a4921dc11495b902f0f8ef9abd2f9c9a41a0f114b68e34a2"} Jul 17 01:59:18.947789 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:18.947462 2583 scope.go:117] "RemoveContainer" containerID="aa6db574ca02498f4655f174c39bb38a0250ec9220dfee452782c875cbbb4d9c" Jul 17 01:59:18.947789 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:18.947625 2583 scope.go:117] "RemoveContainer" containerID="6734dc374889b035a4921dc11495b902f0f8ef9abd2f9c9a41a0f114b68e34a2" Jul 17 01:59:18.947926 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:59:18.947877 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n_odh-ai-gateway-infra(7e8e6a61-7a1b-4f29-852a-7dc6183deba2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" podUID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" Jul 17 01:59:19.620053 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:19.620011 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:59:19.720587 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:19.720558 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:59:19.954331 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:19.954258 2583 scope.go:117] "RemoveContainer" containerID="6734dc374889b035a4921dc11495b902f0f8ef9abd2f9c9a41a0f114b68e34a2" Jul 17 01:59:19.954674 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:59:19.954453 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n_odh-ai-gateway-infra(7e8e6a61-7a1b-4f29-852a-7dc6183deba2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" podUID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" Jul 17 01:59:26.162453 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:26.162412 2583 scope.go:117] "RemoveContainer" containerID="2ba970bfffa14539847a01063576a7bddb045d38a94a621aa291e5257896cdfe" Jul 17 01:59:26.988160 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:26.988050 2583 generic.go:356] "Generic (PLEG): container finished" podID="c9aa9e8f-541d-43a9-abba-203cc9c08779" containerID="33affe7cdc2b31b6e896218a9118c88da9d59d75e2364c19143d8e6f062a9c57" exitCode=6 Jul 17 01:59:26.988160 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:26.988133 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" event={"ID":"c9aa9e8f-541d-43a9-abba-203cc9c08779","Type":"ContainerDied","Data":"33affe7cdc2b31b6e896218a9118c88da9d59d75e2364c19143d8e6f062a9c57"} Jul 17 01:59:26.988394 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:26.988210 2583 scope.go:117] "RemoveContainer" containerID="2ba970bfffa14539847a01063576a7bddb045d38a94a621aa291e5257896cdfe" Jul 17 01:59:26.988575 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:26.988552 2583 scope.go:117] "RemoveContainer" containerID="33affe7cdc2b31b6e896218a9118c88da9d59d75e2364c19143d8e6f062a9c57" Jul 17 01:59:26.988801 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:59:26.988774 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff_odh-ai-gateway-infra(c9aa9e8f-541d-43a9-abba-203cc9c08779)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" podUID="c9aa9e8f-541d-43a9-abba-203cc9c08779" Jul 17 01:59:27.892144 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:27.892110 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:59:28.013497 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:28.013464 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff"] Jul 17 01:59:28.146076 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:28.146002 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" Jul 17 01:59:28.162381 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:28.162352 2583 scope.go:117] "RemoveContainer" containerID="c2485b847655a3b058d5dcf78e107e9e6a77577b8bd6cccb6045a9185138a607" Jul 17 01:59:28.209166 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:28.209093 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9aa9e8f-541d-43a9-abba-203cc9c08779-maas-api-service-ca\") pod \"c9aa9e8f-541d-43a9-abba-203cc9c08779\" (UID: \"c9aa9e8f-541d-43a9-abba-203cc9c08779\") " Jul 17 01:59:28.209468 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:28.209445 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9aa9e8f-541d-43a9-abba-203cc9c08779-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c9aa9e8f-541d-43a9-abba-203cc9c08779" (UID: "c9aa9e8f-541d-43a9-abba-203cc9c08779"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:59:28.310394 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:28.310361 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9aa9e8f-541d-43a9-abba-203cc9c08779-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:59:29.001153 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:29.001117 2583 generic.go:356] "Generic (PLEG): container finished" podID="70e3d99d-84d2-4e28-9c92-aa8755145e65" containerID="020d63ecc3e3ac893be488b38e839ec810689c55525a438c12a6b5b66d2661a2" exitCode=6 Jul 17 01:59:29.001612 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:29.001207 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" event={"ID":"70e3d99d-84d2-4e28-9c92-aa8755145e65","Type":"ContainerDied","Data":"020d63ecc3e3ac893be488b38e839ec810689c55525a438c12a6b5b66d2661a2"} Jul 17 01:59:29.001612 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:29.001275 2583 scope.go:117] "RemoveContainer" containerID="c2485b847655a3b058d5dcf78e107e9e6a77577b8bd6cccb6045a9185138a607" Jul 17 01:59:29.001612 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:29.001588 2583 scope.go:117] "RemoveContainer" containerID="020d63ecc3e3ac893be488b38e839ec810689c55525a438c12a6b5b66d2661a2" Jul 17 01:59:29.001853 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:59:29.001830 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt_odh-ai-gateway-infra(70e3d99d-84d2-4e28-9c92-aa8755145e65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" podUID="70e3d99d-84d2-4e28-9c92-aa8755145e65" Jul 17 01:59:29.002632 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:29.002613 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" event={"ID":"c9aa9e8f-541d-43a9-abba-203cc9c08779","Type":"ContainerDied","Data":"94f784aabae79252f4d7c0879cad10f2278ecc8c56dd841ca88f192f1a396038"} Jul 17 01:59:29.002699 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:29.002651 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff" Jul 17 01:59:29.012980 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:29.012961 2583 scope.go:117] "RemoveContainer" containerID="33affe7cdc2b31b6e896218a9118c88da9d59d75e2364c19143d8e6f062a9c57" Jul 17 01:59:29.030143 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:29.030113 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff"] Jul 17 01:59:29.033548 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:29.033526 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zhxff"] Jul 17 01:59:29.170201 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:29.170120 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9aa9e8f-541d-43a9-abba-203cc9c08779" path="/var/lib/kubelet/pods/c9aa9e8f-541d-43a9-abba-203cc9c08779/volumes" Jul 17 01:59:30.025544 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.025508 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt"] Jul 17 01:59:30.160695 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.160668 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" Jul 17 01:59:30.162665 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.162644 2583 scope.go:117] "RemoveContainer" containerID="6734dc374889b035a4921dc11495b902f0f8ef9abd2f9c9a41a0f114b68e34a2" Jul 17 01:59:30.181200 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181169 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz"] Jul 17 01:59:30.183524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181664 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9aa9e8f-541d-43a9-abba-203cc9c08779" containerName="revoke-keys" Jul 17 01:59:30.183524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181688 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9aa9e8f-541d-43a9-abba-203cc9c08779" containerName="revoke-keys" Jul 17 01:59:30.183524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181704 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9aa9e8f-541d-43a9-abba-203cc9c08779" containerName="revoke-keys" Jul 17 01:59:30.183524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181712 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9aa9e8f-541d-43a9-abba-203cc9c08779" containerName="revoke-keys" Jul 17 01:59:30.183524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181734 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70e3d99d-84d2-4e28-9c92-aa8755145e65" containerName="revoke-keys" Jul 17 01:59:30.183524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181743 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="70e3d99d-84d2-4e28-9c92-aa8755145e65" containerName="revoke-keys" Jul 17 01:59:30.183524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181774 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70e3d99d-84d2-4e28-9c92-aa8755145e65" containerName="revoke-keys" Jul 17 01:59:30.183524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181783 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="70e3d99d-84d2-4e28-9c92-aa8755145e65" containerName="revoke-keys" Jul 17 01:59:30.183524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181796 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70e3d99d-84d2-4e28-9c92-aa8755145e65" containerName="revoke-keys" Jul 17 01:59:30.183524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181805 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="70e3d99d-84d2-4e28-9c92-aa8755145e65" containerName="revoke-keys" Jul 17 01:59:30.183524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181885 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9aa9e8f-541d-43a9-abba-203cc9c08779" containerName="revoke-keys" Jul 17 01:59:30.183524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181897 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="70e3d99d-84d2-4e28-9c92-aa8755145e65" containerName="revoke-keys" Jul 17 01:59:30.183524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181906 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9aa9e8f-541d-43a9-abba-203cc9c08779" containerName="revoke-keys" Jul 17 01:59:30.183524 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.181914 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="70e3d99d-84d2-4e28-9c92-aa8755145e65" containerName="revoke-keys" Jul 17 01:59:30.196627 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.188796 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" Jul 17 01:59:30.196627 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.192300 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz"] Jul 17 01:59:30.230069 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.230031 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e3d99d-84d2-4e28-9c92-aa8755145e65-maas-api-service-ca\") pod \"70e3d99d-84d2-4e28-9c92-aa8755145e65\" (UID: \"70e3d99d-84d2-4e28-9c92-aa8755145e65\") " Jul 17 01:59:30.230404 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.230373 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a818bc66-eeb0-4ec5-885d-cc02afde53e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-262pz\" (UID: \"a818bc66-eeb0-4ec5-885d-cc02afde53e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" Jul 17 01:59:30.230554 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.230433 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e3d99d-84d2-4e28-9c92-aa8755145e65-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70e3d99d-84d2-4e28-9c92-aa8755145e65" (UID: "70e3d99d-84d2-4e28-9c92-aa8755145e65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:59:30.331244 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.331210 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a818bc66-eeb0-4ec5-885d-cc02afde53e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-262pz\" (UID: \"a818bc66-eeb0-4ec5-885d-cc02afde53e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" Jul 17 01:59:30.331385 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.331328 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e3d99d-84d2-4e28-9c92-aa8755145e65-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:59:30.331815 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.331798 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a818bc66-eeb0-4ec5-885d-cc02afde53e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-262pz\" (UID: \"a818bc66-eeb0-4ec5-885d-cc02afde53e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" Jul 17 01:59:30.508678 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.508640 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" Jul 17 01:59:30.642302 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:30.642156 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz"] Jul 17 01:59:30.644352 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:59:30.644329 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda818bc66_eeb0_4ec5_885d_cc02afde53e2.slice/crio-4c9ef3345efeccb1f796e788387300be77dfe45064fcaab54aac050f68c40657 WatchSource:0}: Error finding container 4c9ef3345efeccb1f796e788387300be77dfe45064fcaab54aac050f68c40657: Status 404 returned error can't find the container with id 4c9ef3345efeccb1f796e788387300be77dfe45064fcaab54aac050f68c40657 Jul 17 01:59:31.018001 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:31.017974 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" Jul 17 01:59:31.018198 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:31.017973 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt" event={"ID":"70e3d99d-84d2-4e28-9c92-aa8755145e65","Type":"ContainerDied","Data":"45bf1feb88cad367d226b727833b4423a62d5ccf14fd14f06169c8a49181f892"} Jul 17 01:59:31.018198 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:31.018167 2583 scope.go:117] "RemoveContainer" containerID="020d63ecc3e3ac893be488b38e839ec810689c55525a438c12a6b5b66d2661a2" Jul 17 01:59:31.019403 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:31.019373 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" event={"ID":"a818bc66-eeb0-4ec5-885d-cc02afde53e2","Type":"ContainerStarted","Data":"4c9ef3345efeccb1f796e788387300be77dfe45064fcaab54aac050f68c40657"} Jul 17 01:59:31.021597 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:31.021570 2583 generic.go:356] "Generic (PLEG): container finished" podID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" containerID="8b12ec836602b2fa75af65a90070cd10c2e68de927b082160cc2d839fd4cc993" exitCode=6 Jul 17 01:59:31.021696 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:31.021639 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" event={"ID":"7e8e6a61-7a1b-4f29-852a-7dc6183deba2","Type":"ContainerDied","Data":"8b12ec836602b2fa75af65a90070cd10c2e68de927b082160cc2d839fd4cc993"} Jul 17 01:59:31.021989 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:31.021974 2583 scope.go:117] "RemoveContainer" containerID="8b12ec836602b2fa75af65a90070cd10c2e68de927b082160cc2d839fd4cc993" Jul 17 01:59:31.022250 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:59:31.022229 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n_odh-ai-gateway-infra(7e8e6a61-7a1b-4f29-852a-7dc6183deba2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" podUID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" Jul 17 01:59:31.027775 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:31.027613 2583 scope.go:117] "RemoveContainer" containerID="6734dc374889b035a4921dc11495b902f0f8ef9abd2f9c9a41a0f114b68e34a2" Jul 17 01:59:31.054148 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:31.054117 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt"] Jul 17 01:59:31.059836 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:31.059809 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-ww4rt"] Jul 17 01:59:31.167640 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:31.167604 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70e3d99d-84d2-4e28-9c92-aa8755145e65" path="/var/lib/kubelet/pods/70e3d99d-84d2-4e28-9c92-aa8755145e65/volumes" Jul 17 01:59:32.030475 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:32.030438 2583 generic.go:356] "Generic (PLEG): container finished" podID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" containerID="d70b2df0837edc3b7e1d6565a373c2b2c33332c977f75622e50c887f5cb79dcf" exitCode=6 Jul 17 01:59:32.030893 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:32.030501 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" event={"ID":"a818bc66-eeb0-4ec5-885d-cc02afde53e2","Type":"ContainerDied","Data":"d70b2df0837edc3b7e1d6565a373c2b2c33332c977f75622e50c887f5cb79dcf"} Jul 17 01:59:32.030893 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:32.030737 2583 scope.go:117] "RemoveContainer" containerID="d70b2df0837edc3b7e1d6565a373c2b2c33332c977f75622e50c887f5cb79dcf" Jul 17 01:59:32.050020 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:32.049996 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n"] Jul 17 01:59:32.186690 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:32.186660 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" Jul 17 01:59:32.249508 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:32.249470 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8e6a61-7a1b-4f29-852a-7dc6183deba2-maas-api-service-ca\") pod \"7e8e6a61-7a1b-4f29-852a-7dc6183deba2\" (UID: \"7e8e6a61-7a1b-4f29-852a-7dc6183deba2\") " Jul 17 01:59:32.249833 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:32.249810 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8e6a61-7a1b-4f29-852a-7dc6183deba2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e8e6a61-7a1b-4f29-852a-7dc6183deba2" (UID: "7e8e6a61-7a1b-4f29-852a-7dc6183deba2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:59:32.350283 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:32.350253 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8e6a61-7a1b-4f29-852a-7dc6183deba2-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:59:33.036734 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:33.036704 2583 generic.go:356] "Generic (PLEG): container finished" podID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" containerID="552f8d605db614294de8410176fe3270634f6abe35f95fc3479f0fe6f1d66966" exitCode=6 Jul 17 01:59:33.037240 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:33.036787 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" event={"ID":"a818bc66-eeb0-4ec5-885d-cc02afde53e2","Type":"ContainerDied","Data":"552f8d605db614294de8410176fe3270634f6abe35f95fc3479f0fe6f1d66966"} Jul 17 01:59:33.037240 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:33.036853 2583 scope.go:117] "RemoveContainer" containerID="d70b2df0837edc3b7e1d6565a373c2b2c33332c977f75622e50c887f5cb79dcf" Jul 17 01:59:33.037240 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:33.037017 2583 scope.go:117] "RemoveContainer" containerID="552f8d605db614294de8410176fe3270634f6abe35f95fc3479f0fe6f1d66966" Jul 17 01:59:33.037407 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:59:33.037296 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-262pz_odh-ai-gateway-infra(a818bc66-eeb0-4ec5-885d-cc02afde53e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" podUID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" Jul 17 01:59:33.038026 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:33.038017 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" Jul 17 01:59:33.038084 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:33.038036 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n" event={"ID":"7e8e6a61-7a1b-4f29-852a-7dc6183deba2","Type":"ContainerDied","Data":"185cb5327c1c0ee3a0fc6d01cca52e0308b05263ba4e070679696dace4b8c9bc"} Jul 17 01:59:33.047953 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:33.047934 2583 scope.go:117] "RemoveContainer" containerID="8b12ec836602b2fa75af65a90070cd10c2e68de927b082160cc2d839fd4cc993" Jul 17 01:59:33.066274 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:33.066249 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n"] Jul 17 01:59:33.069557 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:33.069532 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-xkc6n"] Jul 17 01:59:33.168510 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:33.168469 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" path="/var/lib/kubelet/pods/7e8e6a61-7a1b-4f29-852a-7dc6183deba2/volumes" Jul 17 01:59:34.044808 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:34.044783 2583 scope.go:117] "RemoveContainer" containerID="552f8d605db614294de8410176fe3270634f6abe35f95fc3479f0fe6f1d66966" Jul 17 01:59:34.045287 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:59:34.045050 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-262pz_odh-ai-gateway-infra(a818bc66-eeb0-4ec5-885d-cc02afde53e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" podUID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" Jul 17 01:59:37.116309 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:37.116262 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:59:46.097104 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:46.097070 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:59:47.095444 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.095414 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:59:47.163144 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.163112 2583 scope.go:117] "RemoveContainer" containerID="552f8d605db614294de8410176fe3270634f6abe35f95fc3479f0fe6f1d66966" Jul 17 01:59:47.205516 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.205486 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz"] Jul 17 01:59:47.205892 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.205878 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" containerName="revoke-keys" Jul 17 01:59:47.205932 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.205897 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" containerName="revoke-keys" Jul 17 01:59:47.205932 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.205909 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" containerName="revoke-keys" Jul 17 01:59:47.205932 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.205915 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" containerName="revoke-keys" Jul 17 01:59:47.206030 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.205933 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9aa9e8f-541d-43a9-abba-203cc9c08779" containerName="revoke-keys" Jul 17 01:59:47.206030 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.205940 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9aa9e8f-541d-43a9-abba-203cc9c08779" containerName="revoke-keys" Jul 17 01:59:47.206030 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.205998 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" containerName="revoke-keys" Jul 17 01:59:47.206030 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.206006 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" containerName="revoke-keys" Jul 17 01:59:47.206030 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.206012 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9aa9e8f-541d-43a9-abba-203cc9c08779" containerName="revoke-keys" Jul 17 01:59:47.206030 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.206017 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="70e3d99d-84d2-4e28-9c92-aa8755145e65" containerName="revoke-keys" Jul 17 01:59:47.206030 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.206025 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" containerName="revoke-keys" Jul 17 01:59:47.210182 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.210165 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz" Jul 17 01:59:47.217947 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.217922 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz"] Jul 17 01:59:47.285630 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.285589 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38dd9c9-2c0b-4681-9986-2c66d32ef99b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz\" (UID: \"e38dd9c9-2c0b-4681-9986-2c66d32ef99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz" Jul 17 01:59:47.387040 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.386942 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38dd9c9-2c0b-4681-9986-2c66d32ef99b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz\" (UID: \"e38dd9c9-2c0b-4681-9986-2c66d32ef99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz" Jul 17 01:59:47.387650 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.387624 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38dd9c9-2c0b-4681-9986-2c66d32ef99b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz\" (UID: \"e38dd9c9-2c0b-4681-9986-2c66d32ef99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz" Jul 17 01:59:47.522388 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.522346 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz" Jul 17 01:59:47.656351 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.656329 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz"] Jul 17 01:59:47.659225 ip-10-0-128-44 kubenswrapper[2583]: W0717 01:59:47.659194 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38dd9c9_2c0b_4681_9986_2c66d32ef99b.slice/crio-a7a98fc33e9f0f23615e1f5af3480a9b92405be2cb2c2f89a1581d164c6c3e29 WatchSource:0}: Error finding container a7a98fc33e9f0f23615e1f5af3480a9b92405be2cb2c2f89a1581d164c6c3e29: Status 404 returned error can't find the container with id a7a98fc33e9f0f23615e1f5af3480a9b92405be2cb2c2f89a1581d164c6c3e29 Jul 17 01:59:47.703528 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.703497 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:59:47.804651 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:47.804611 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:59:48.115691 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:48.115650 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz" event={"ID":"e38dd9c9-2c0b-4681-9986-2c66d32ef99b","Type":"ContainerStarted","Data":"e061b654e0ad33f62c388970a53603345d383b618c9157a0e772eb4e251aaed7"} Jul 17 01:59:48.115894 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:48.115716 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz" event={"ID":"e38dd9c9-2c0b-4681-9986-2c66d32ef99b","Type":"ContainerStarted","Data":"a7a98fc33e9f0f23615e1f5af3480a9b92405be2cb2c2f89a1581d164c6c3e29"} Jul 17 01:59:48.117514 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:48.117484 2583 generic.go:356] "Generic (PLEG): container finished" podID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" containerID="600102b47a3e6934594a43d3fb457788c666ece3fa9fa52f755675d7c4db9290" exitCode=6 Jul 17 01:59:48.117617 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:48.117554 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" event={"ID":"a818bc66-eeb0-4ec5-885d-cc02afde53e2","Type":"ContainerDied","Data":"600102b47a3e6934594a43d3fb457788c666ece3fa9fa52f755675d7c4db9290"} Jul 17 01:59:48.117669 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:48.117596 2583 scope.go:117] "RemoveContainer" containerID="552f8d605db614294de8410176fe3270634f6abe35f95fc3479f0fe6f1d66966" Jul 17 01:59:48.117939 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:48.117918 2583 scope.go:117] "RemoveContainer" containerID="600102b47a3e6934594a43d3fb457788c666ece3fa9fa52f755675d7c4db9290" Jul 17 01:59:48.118252 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:59:48.118226 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-262pz_odh-ai-gateway-infra(a818bc66-eeb0-4ec5-885d-cc02afde53e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" podUID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" Jul 17 01:59:48.130295 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:48.130252 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz" podStartSLOduration=1.130235319 podStartE2EDuration="1.130235319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 01:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 01:59:48.12834095 +0000 UTC m=+2745.523955801" watchObservedRunningTime="2026-07-17 01:59:48.130235319 +0000 UTC m=+2745.525850171" Jul 17 01:59:49.124501 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:49.124464 2583 generic.go:356] "Generic (PLEG): container finished" podID="e38dd9c9-2c0b-4681-9986-2c66d32ef99b" containerID="e061b654e0ad33f62c388970a53603345d383b618c9157a0e772eb4e251aaed7" exitCode=0 Jul 17 01:59:49.124914 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:49.124541 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz" event={"ID":"e38dd9c9-2c0b-4681-9986-2c66d32ef99b","Type":"ContainerDied","Data":"e061b654e0ad33f62c388970a53603345d383b618c9157a0e772eb4e251aaed7"} Jul 17 01:59:49.149512 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:49.149471 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz"] Jul 17 01:59:49.286728 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:49.286704 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" Jul 17 01:59:49.405901 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:49.405810 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a818bc66-eeb0-4ec5-885d-cc02afde53e2-maas-api-service-ca\") pod \"a818bc66-eeb0-4ec5-885d-cc02afde53e2\" (UID: \"a818bc66-eeb0-4ec5-885d-cc02afde53e2\") " Jul 17 01:59:49.406238 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:49.406218 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a818bc66-eeb0-4ec5-885d-cc02afde53e2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a818bc66-eeb0-4ec5-885d-cc02afde53e2" (UID: "a818bc66-eeb0-4ec5-885d-cc02afde53e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:59:49.507383 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:49.507347 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a818bc66-eeb0-4ec5-885d-cc02afde53e2-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:59:50.131944 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:50.131912 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" Jul 17 01:59:50.132455 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:50.131955 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz" event={"ID":"a818bc66-eeb0-4ec5-885d-cc02afde53e2","Type":"ContainerDied","Data":"4c9ef3345efeccb1f796e788387300be77dfe45064fcaab54aac050f68c40657"} Jul 17 01:59:50.132455 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:50.132027 2583 scope.go:117] "RemoveContainer" containerID="600102b47a3e6934594a43d3fb457788c666ece3fa9fa52f755675d7c4db9290" Jul 17 01:59:50.154222 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:50.154193 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz"] Jul 17 01:59:50.163530 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:50.163465 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-262pz"] Jul 17 01:59:50.264996 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:50.264963 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz" Jul 17 01:59:50.415698 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:50.415609 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38dd9c9-2c0b-4681-9986-2c66d32ef99b-maas-api-service-ca\") pod \"e38dd9c9-2c0b-4681-9986-2c66d32ef99b\" (UID: \"e38dd9c9-2c0b-4681-9986-2c66d32ef99b\") " Jul 17 01:59:50.416025 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:50.416005 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e38dd9c9-2c0b-4681-9986-2c66d32ef99b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e38dd9c9-2c0b-4681-9986-2c66d32ef99b" (UID: "e38dd9c9-2c0b-4681-9986-2c66d32ef99b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:59:50.516340 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:50.516308 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38dd9c9-2c0b-4681-9986-2c66d32ef99b-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:59:51.137664 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:51.137638 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz" Jul 17 01:59:51.137664 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:51.137668 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz" event={"ID":"e38dd9c9-2c0b-4681-9986-2c66d32ef99b","Type":"ContainerDied","Data":"a7a98fc33e9f0f23615e1f5af3480a9b92405be2cb2c2f89a1581d164c6c3e29"} Jul 17 01:59:51.138242 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:51.137716 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7a98fc33e9f0f23615e1f5af3480a9b92405be2cb2c2f89a1581d164c6c3e29" Jul 17 01:59:51.167864 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:51.167828 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" path="/var/lib/kubelet/pods/a818bc66-eeb0-4ec5-885d-cc02afde53e2/volumes" Jul 17 01:59:57.589838 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:57.589806 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79"] Jul 17 01:59:57.590412 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:57.590020 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" podUID="256a2841-8135-4ecf-9c61-04fb8b0c9172" containerName="maas-api" containerID="cri-o://183e6127e4e29f58437244e345f276b264420733cbd02f6c2679c28c4bae49fa" gracePeriod=30 Jul 17 01:59:57.617801 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:57.617762 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 01:59:57.804637 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:57.804599 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" podUID="256a2841-8135-4ecf-9c61-04fb8b0c9172" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.67:8443/health\": dial tcp 10.134.0.67:8443: connect: connection refused" Jul 17 01:59:58.038113 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.038082 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:59:58.175372 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.175282 2583 generic.go:356] "Generic (PLEG): container finished" podID="256a2841-8135-4ecf-9c61-04fb8b0c9172" containerID="183e6127e4e29f58437244e345f276b264420733cbd02f6c2679c28c4bae49fa" exitCode=0 Jul 17 01:59:58.175372 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.175328 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" Jul 17 01:59:58.175372 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.175352 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" event={"ID":"256a2841-8135-4ecf-9c61-04fb8b0c9172","Type":"ContainerDied","Data":"183e6127e4e29f58437244e345f276b264420733cbd02f6c2679c28c4bae49fa"} Jul 17 01:59:58.175612 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.175392 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79" event={"ID":"256a2841-8135-4ecf-9c61-04fb8b0c9172","Type":"ContainerDied","Data":"89ec9601a46e735a110e494d1e52aa8054055dd4336e1ec7fe7a591413287cb9"} Jul 17 01:59:58.175612 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.175414 2583 scope.go:117] "RemoveContainer" containerID="183e6127e4e29f58437244e345f276b264420733cbd02f6c2679c28c4bae49fa" Jul 17 01:59:58.177529 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.177507 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5k9x\" (UniqueName: \"kubernetes.io/projected/256a2841-8135-4ecf-9c61-04fb8b0c9172-kube-api-access-c5k9x\") pod \"256a2841-8135-4ecf-9c61-04fb8b0c9172\" (UID: \"256a2841-8135-4ecf-9c61-04fb8b0c9172\") " Jul 17 01:59:58.177660 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.177568 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/256a2841-8135-4ecf-9c61-04fb8b0c9172-maas-api-tls\") pod \"256a2841-8135-4ecf-9c61-04fb8b0c9172\" (UID: \"256a2841-8135-4ecf-9c61-04fb8b0c9172\") " Jul 17 01:59:58.177724 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.177675 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/256a2841-8135-4ecf-9c61-04fb8b0c9172-openshift-service-ca\") pod \"256a2841-8135-4ecf-9c61-04fb8b0c9172\" (UID: \"256a2841-8135-4ecf-9c61-04fb8b0c9172\") " Jul 17 01:59:58.178035 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.178013 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/256a2841-8135-4ecf-9c61-04fb8b0c9172-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "256a2841-8135-4ecf-9c61-04fb8b0c9172" (UID: "256a2841-8135-4ecf-9c61-04fb8b0c9172"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 01:59:58.179691 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.179669 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256a2841-8135-4ecf-9c61-04fb8b0c9172-kube-api-access-c5k9x" (OuterVolumeSpecName: "kube-api-access-c5k9x") pod "256a2841-8135-4ecf-9c61-04fb8b0c9172" (UID: "256a2841-8135-4ecf-9c61-04fb8b0c9172"). InnerVolumeSpecName "kube-api-access-c5k9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 01:59:58.179793 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.179699 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/256a2841-8135-4ecf-9c61-04fb8b0c9172-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "256a2841-8135-4ecf-9c61-04fb8b0c9172" (UID: "256a2841-8135-4ecf-9c61-04fb8b0c9172"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 01:59:58.194519 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.194499 2583 scope.go:117] "RemoveContainer" containerID="183e6127e4e29f58437244e345f276b264420733cbd02f6c2679c28c4bae49fa" Jul 17 01:59:58.194823 ip-10-0-128-44 kubenswrapper[2583]: E0717 01:59:58.194803 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"183e6127e4e29f58437244e345f276b264420733cbd02f6c2679c28c4bae49fa\": container with ID starting with 183e6127e4e29f58437244e345f276b264420733cbd02f6c2679c28c4bae49fa not found: ID does not exist" containerID="183e6127e4e29f58437244e345f276b264420733cbd02f6c2679c28c4bae49fa" Jul 17 01:59:58.194903 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.194831 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"183e6127e4e29f58437244e345f276b264420733cbd02f6c2679c28c4bae49fa"} err="failed to get container status \"183e6127e4e29f58437244e345f276b264420733cbd02f6c2679c28c4bae49fa\": rpc error: code = NotFound desc = could not find container \"183e6127e4e29f58437244e345f276b264420733cbd02f6c2679c28c4bae49fa\": container with ID starting with 183e6127e4e29f58437244e345f276b264420733cbd02f6c2679c28c4bae49fa not found: ID does not exist" Jul 17 01:59:58.278418 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.278382 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/256a2841-8135-4ecf-9c61-04fb8b0c9172-openshift-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:59:58.278418 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.278408 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5k9x\" (UniqueName: \"kubernetes.io/projected/256a2841-8135-4ecf-9c61-04fb8b0c9172-kube-api-access-c5k9x\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:59:58.278418 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.278420 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/256a2841-8135-4ecf-9c61-04fb8b0c9172-maas-api-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 01:59:58.497282 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.497246 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79"] Jul 17 01:59:58.502745 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:58.502719 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-089e15-65b8d89485-5nl79"] Jul 17 01:59:59.167324 ip-10-0-128-44 kubenswrapper[2583]: I0717 01:59:59.167287 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="256a2841-8135-4ecf-9c61-04fb8b0c9172" path="/var/lib/kubelet/pods/256a2841-8135-4ecf-9c61-04fb8b0c9172/volumes" Jul 17 02:00:00.129509 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129476 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737560-qqs98"] Jul 17 02:00:00.129830 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129818 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="256a2841-8135-4ecf-9c61-04fb8b0c9172" containerName="maas-api" Jul 17 02:00:00.129878 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129833 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="256a2841-8135-4ecf-9c61-04fb8b0c9172" containerName="maas-api" Jul 17 02:00:00.129878 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129849 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" containerName="revoke-keys" Jul 17 02:00:00.129878 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129856 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" containerName="revoke-keys" Jul 17 02:00:00.129878 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129864 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" containerName="revoke-keys" Jul 17 02:00:00.129878 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129869 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" containerName="revoke-keys" Jul 17 02:00:00.129878 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129875 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" containerName="revoke-keys" Jul 17 02:00:00.129878 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129880 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" containerName="revoke-keys" Jul 17 02:00:00.130194 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129886 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e38dd9c9-2c0b-4681-9986-2c66d32ef99b" containerName="revoke-keys" Jul 17 02:00:00.130194 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129891 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e38dd9c9-2c0b-4681-9986-2c66d32ef99b" containerName="revoke-keys" Jul 17 02:00:00.130194 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129905 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" containerName="revoke-keys" Jul 17 02:00:00.130194 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129912 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8e6a61-7a1b-4f29-852a-7dc6183deba2" containerName="revoke-keys" Jul 17 02:00:00.130194 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129956 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" containerName="revoke-keys" Jul 17 02:00:00.130194 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129962 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" containerName="revoke-keys" Jul 17 02:00:00.130194 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129971 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="256a2841-8135-4ecf-9c61-04fb8b0c9172" containerName="maas-api" Jul 17 02:00:00.130194 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.129977 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e38dd9c9-2c0b-4681-9986-2c66d32ef99b" containerName="revoke-keys" Jul 17 02:00:00.134935 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.134913 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7f72-29737560-tt9d4"] Jul 17 02:00:00.135054 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.135035 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737560-qqs98" Jul 17 02:00:00.135448 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.135426 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a818bc66-eeb0-4ec5-885d-cc02afde53e2" containerName="revoke-keys" Jul 17 02:00:00.137659 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.137638 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 02:00:00.138197 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.138180 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7f72-29737560-tt9d4" Jul 17 02:00:00.142018 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.141996 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737560-qqs98"] Jul 17 02:00:00.144486 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.144089 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7f72-29737560-tt9d4"] Jul 17 02:00:00.148448 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.148429 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737560-qqs98" Jul 17 02:00:00.157359 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.157334 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7f72-29737560-tt9d4" Jul 17 02:00:00.298968 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.298940 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737560-qqs98"] Jul 17 02:00:00.302329 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:00:00.302284 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d5bc47_89fc_4447_af12_be3281ad0001.slice/crio-512ef4172acea8636da7043413236ace997e76cc1da489aeabcb58a5c09e857e WatchSource:0}: Error finding container 512ef4172acea8636da7043413236ace997e76cc1da489aeabcb58a5c09e857e: Status 404 returned error can't find the container with id 512ef4172acea8636da7043413236ace997e76cc1da489aeabcb58a5c09e857e Jul 17 02:00:00.323300 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:00.323259 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7f72-29737560-tt9d4"] Jul 17 02:00:00.327590 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:00:00.327563 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f03564b_ae75_4400_bbe3_c8e4a08b4e48.slice/crio-d1b1af4bf1776289ff46d11fbd8d01c72f340fa274f3e763c48120a8b98738c2 WatchSource:0}: Error finding container d1b1af4bf1776289ff46d11fbd8d01c72f340fa274f3e763c48120a8b98738c2: Status 404 returned error can't find the container with id d1b1af4bf1776289ff46d11fbd8d01c72f340fa274f3e763c48120a8b98738c2 Jul 17 02:00:01.193691 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:01.193654 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7f72-29737560-tt9d4" event={"ID":"3f03564b-ae75-4400-bbe3-c8e4a08b4e48","Type":"ContainerStarted","Data":"c709fbaa6755d7e8aa738314a5f9dff82ba6c5a6152db9168792bd9f5caa9c46"} Jul 17 02:00:01.193908 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:01.193716 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7f72-29737560-tt9d4" event={"ID":"3f03564b-ae75-4400-bbe3-c8e4a08b4e48","Type":"ContainerStarted","Data":"d1b1af4bf1776289ff46d11fbd8d01c72f340fa274f3e763c48120a8b98738c2"} Jul 17 02:00:01.195068 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:01.195046 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737560-qqs98" event={"ID":"26d5bc47-89fc-4447-af12-be3281ad0001","Type":"ContainerStarted","Data":"5e8bb189aa045b7d93e07adf36b9ff36490e3fe7462b18cacf444351289cff49"} Jul 17 02:00:01.195206 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:01.195083 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737560-qqs98" event={"ID":"26d5bc47-89fc-4447-af12-be3281ad0001","Type":"ContainerStarted","Data":"512ef4172acea8636da7043413236ace997e76cc1da489aeabcb58a5c09e857e"} Jul 17 02:00:01.209508 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:01.209457 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7f72-29737560-tt9d4" podStartSLOduration=1.209442024 podStartE2EDuration="1.209442024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:00:01.208270103 +0000 UTC m=+2758.603884953" watchObservedRunningTime="2026-07-17 02:00:01.209442024 +0000 UTC m=+2758.605056874" Jul 17 02:00:02.200977 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:02.200946 2583 generic.go:356] "Generic (PLEG): container finished" podID="3f03564b-ae75-4400-bbe3-c8e4a08b4e48" containerID="c709fbaa6755d7e8aa738314a5f9dff82ba6c5a6152db9168792bd9f5caa9c46" exitCode=0 Jul 17 02:00:02.201433 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:02.201033 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7f72-29737560-tt9d4" event={"ID":"3f03564b-ae75-4400-bbe3-c8e4a08b4e48","Type":"ContainerDied","Data":"c709fbaa6755d7e8aa738314a5f9dff82ba6c5a6152db9168792bd9f5caa9c46"} Jul 17 02:00:02.202532 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:02.202512 2583 generic.go:356] "Generic (PLEG): container finished" podID="26d5bc47-89fc-4447-af12-be3281ad0001" containerID="5e8bb189aa045b7d93e07adf36b9ff36490e3fe7462b18cacf444351289cff49" exitCode=0 Jul 17 02:00:02.202616 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:02.202597 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737560-qqs98" event={"ID":"26d5bc47-89fc-4447-af12-be3281ad0001","Type":"ContainerDied","Data":"5e8bb189aa045b7d93e07adf36b9ff36490e3fe7462b18cacf444351289cff49"} Jul 17 02:00:02.217112 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:02.217036 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737560-qqs98" podStartSLOduration=2.217018823 podStartE2EDuration="2.217018823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:00:01.222373416 +0000 UTC m=+2758.617988269" watchObservedRunningTime="2026-07-17 02:00:02.217018823 +0000 UTC m=+2759.612633673" Jul 17 02:00:02.286711 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:02.286681 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:03.345296 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.345266 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7f72-29737560-tt9d4" Jul 17 02:00:03.368777 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.368753 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737560-qqs98" Jul 17 02:00:03.655113 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.655072 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24"] Jul 17 02:00:03.655431 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.655418 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f03564b-ae75-4400-bbe3-c8e4a08b4e48" containerName="cleanup" Jul 17 02:00:03.655476 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.655433 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f03564b-ae75-4400-bbe3-c8e4a08b4e48" containerName="cleanup" Jul 17 02:00:03.655476 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.655442 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d5bc47-89fc-4447-af12-be3281ad0001" containerName="cleanup" Jul 17 02:00:03.655476 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.655447 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d5bc47-89fc-4447-af12-be3281ad0001" containerName="cleanup" Jul 17 02:00:03.655571 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.655504 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f03564b-ae75-4400-bbe3-c8e4a08b4e48" containerName="cleanup" Jul 17 02:00:03.655571 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.655514 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d5bc47-89fc-4447-af12-be3281ad0001" containerName="cleanup" Jul 17 02:00:03.659841 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.659824 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" Jul 17 02:00:03.664906 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.664875 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24"] Jul 17 02:00:03.722620 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.722523 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc05e058-5f07-44a9-9e37-6bcb476ec68b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24\" (UID: \"fc05e058-5f07-44a9-9e37-6bcb476ec68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" Jul 17 02:00:03.823514 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.823473 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc05e058-5f07-44a9-9e37-6bcb476ec68b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24\" (UID: \"fc05e058-5f07-44a9-9e37-6bcb476ec68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" Jul 17 02:00:03.824066 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.824046 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc05e058-5f07-44a9-9e37-6bcb476ec68b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24\" (UID: \"fc05e058-5f07-44a9-9e37-6bcb476ec68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" Jul 17 02:00:03.972355 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:03.972294 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" Jul 17 02:00:04.107582 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:04.107561 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24"] Jul 17 02:00:04.109726 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:00:04.109702 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc05e058_5f07_44a9_9e37_6bcb476ec68b.slice/crio-bdfb9fee25afd0abcab185a2ebea08e5a967f5160e928577d58daaaac68e7507 WatchSource:0}: Error finding container bdfb9fee25afd0abcab185a2ebea08e5a967f5160e928577d58daaaac68e7507: Status 404 returned error can't find the container with id bdfb9fee25afd0abcab185a2ebea08e5a967f5160e928577d58daaaac68e7507 Jul 17 02:00:04.212880 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:04.212853 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7f72-29737560-tt9d4" Jul 17 02:00:04.212880 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:04.212867 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7f72-29737560-tt9d4" event={"ID":"3f03564b-ae75-4400-bbe3-c8e4a08b4e48","Type":"ContainerDied","Data":"d1b1af4bf1776289ff46d11fbd8d01c72f340fa274f3e763c48120a8b98738c2"} Jul 17 02:00:04.213127 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:04.212911 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1b1af4bf1776289ff46d11fbd8d01c72f340fa274f3e763c48120a8b98738c2" Jul 17 02:00:04.214206 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:04.214178 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" event={"ID":"fc05e058-5f07-44a9-9e37-6bcb476ec68b","Type":"ContainerStarted","Data":"bdfb9fee25afd0abcab185a2ebea08e5a967f5160e928577d58daaaac68e7507"} Jul 17 02:00:04.215515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:04.215494 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737560-qqs98" event={"ID":"26d5bc47-89fc-4447-af12-be3281ad0001","Type":"ContainerDied","Data":"512ef4172acea8636da7043413236ace997e76cc1da489aeabcb58a5c09e857e"} Jul 17 02:00:04.215634 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:04.215529 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="512ef4172acea8636da7043413236ace997e76cc1da489aeabcb58a5c09e857e" Jul 17 02:00:04.215634 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:04.215532 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737560-qqs98" Jul 17 02:00:05.221477 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:05.221437 2583 generic.go:356] "Generic (PLEG): container finished" podID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" containerID="46cd8e00668a28d5ba773a9ff59105eaecdfd245e0ef399eaa9c53e6c0e9783f" exitCode=6 Jul 17 02:00:05.221878 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:05.221516 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" event={"ID":"fc05e058-5f07-44a9-9e37-6bcb476ec68b","Type":"ContainerDied","Data":"46cd8e00668a28d5ba773a9ff59105eaecdfd245e0ef399eaa9c53e6c0e9783f"} Jul 17 02:00:05.221878 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:05.221649 2583 scope.go:117] "RemoveContainer" containerID="46cd8e00668a28d5ba773a9ff59105eaecdfd245e0ef399eaa9c53e6c0e9783f" Jul 17 02:00:06.228298 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:06.228212 2583 generic.go:356] "Generic (PLEG): container finished" podID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" containerID="24ed65fec22f1a7c79fc7a28278afb148f4a09c666f91bbff58899be9766bf9f" exitCode=6 Jul 17 02:00:06.228298 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:06.228270 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" event={"ID":"fc05e058-5f07-44a9-9e37-6bcb476ec68b","Type":"ContainerDied","Data":"24ed65fec22f1a7c79fc7a28278afb148f4a09c666f91bbff58899be9766bf9f"} Jul 17 02:00:06.228771 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:06.228316 2583 scope.go:117] "RemoveContainer" containerID="46cd8e00668a28d5ba773a9ff59105eaecdfd245e0ef399eaa9c53e6c0e9783f" Jul 17 02:00:06.228771 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:06.228459 2583 scope.go:117] "RemoveContainer" containerID="24ed65fec22f1a7c79fc7a28278afb148f4a09c666f91bbff58899be9766bf9f" Jul 17 02:00:06.228771 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:06.228675 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24_odh-ai-gateway-infra(fc05e058-5f07-44a9-9e37-6bcb476ec68b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" podUID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" Jul 17 02:00:06.247446 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:06.247421 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv"] Jul 17 02:00:06.251974 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:06.251956 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" Jul 17 02:00:06.257420 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:06.257370 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv"] Jul 17 02:00:06.341799 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:06.341755 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe01066-d70f-4be8-b79c-d9eef5feab5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv\" (UID: \"abe01066-d70f-4be8-b79c-d9eef5feab5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" Jul 17 02:00:06.442560 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:06.442492 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe01066-d70f-4be8-b79c-d9eef5feab5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv\" (UID: \"abe01066-d70f-4be8-b79c-d9eef5feab5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" Jul 17 02:00:06.443045 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:06.443024 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe01066-d70f-4be8-b79c-d9eef5feab5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv\" (UID: \"abe01066-d70f-4be8-b79c-d9eef5feab5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" Jul 17 02:00:06.563968 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:06.563938 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" Jul 17 02:00:06.699478 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:06.699321 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv"] Jul 17 02:00:06.702208 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:00:06.702182 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe01066_d70f_4be8_b79c_d9eef5feab5e.slice/crio-9dcf7a7a23410f03dad2462b7bf70e29dcb0bfa9ee7d6e3e090ffee2eef80349 WatchSource:0}: Error finding container 9dcf7a7a23410f03dad2462b7bf70e29dcb0bfa9ee7d6e3e090ffee2eef80349: Status 404 returned error can't find the container with id 9dcf7a7a23410f03dad2462b7bf70e29dcb0bfa9ee7d6e3e090ffee2eef80349 Jul 17 02:00:07.234927 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:07.234830 2583 scope.go:117] "RemoveContainer" containerID="24ed65fec22f1a7c79fc7a28278afb148f4a09c666f91bbff58899be9766bf9f" Jul 17 02:00:07.235391 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:07.235068 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24_odh-ai-gateway-infra(fc05e058-5f07-44a9-9e37-6bcb476ec68b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" podUID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" Jul 17 02:00:07.236287 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:07.236255 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" event={"ID":"abe01066-d70f-4be8-b79c-d9eef5feab5e","Type":"ContainerStarted","Data":"9f2c175ea6b1f7317755f4ebd6b1d0892c23105524d5811801300481f6540509"} Jul 17 02:00:07.236419 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:07.236302 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" event={"ID":"abe01066-d70f-4be8-b79c-d9eef5feab5e","Type":"ContainerStarted","Data":"9dcf7a7a23410f03dad2462b7bf70e29dcb0bfa9ee7d6e3e090ffee2eef80349"} Jul 17 02:00:07.265522 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:07.265466 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" podStartSLOduration=1.265449556 podStartE2EDuration="1.265449556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:00:07.264920336 +0000 UTC m=+2764.660535187" watchObservedRunningTime="2026-07-17 02:00:07.265449556 +0000 UTC m=+2764.661064471" Jul 17 02:00:08.242912 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:08.242875 2583 generic.go:356] "Generic (PLEG): container finished" podID="abe01066-d70f-4be8-b79c-d9eef5feab5e" containerID="9f2c175ea6b1f7317755f4ebd6b1d0892c23105524d5811801300481f6540509" exitCode=6 Jul 17 02:00:08.243449 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:08.242964 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" event={"ID":"abe01066-d70f-4be8-b79c-d9eef5feab5e","Type":"ContainerDied","Data":"9f2c175ea6b1f7317755f4ebd6b1d0892c23105524d5811801300481f6540509"} Jul 17 02:00:08.243449 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:08.243230 2583 scope.go:117] "RemoveContainer" containerID="9f2c175ea6b1f7317755f4ebd6b1d0892c23105524d5811801300481f6540509" Jul 17 02:00:08.294933 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:08.294902 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj"] Jul 17 02:00:08.299526 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:08.299509 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" Jul 17 02:00:08.307860 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:08.307834 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj"] Jul 17 02:00:08.355459 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:08.355419 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f256a-5cce-4a10-b131-f985a0a2b55d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj\" (UID: \"1c7f256a-5cce-4a10-b131-f985a0a2b55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" Jul 17 02:00:08.456392 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:08.456355 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f256a-5cce-4a10-b131-f985a0a2b55d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj\" (UID: \"1c7f256a-5cce-4a10-b131-f985a0a2b55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" Jul 17 02:00:08.457175 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:08.457144 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f256a-5cce-4a10-b131-f985a0a2b55d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj\" (UID: \"1c7f256a-5cce-4a10-b131-f985a0a2b55d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" Jul 17 02:00:08.504845 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:08.504718 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:08.604257 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:08.604224 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:08.610542 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:08.610522 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" Jul 17 02:00:08.741620 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:08.741492 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj"] Jul 17 02:00:08.744220 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:00:08.744191 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7f256a_5cce_4a10_b131_f985a0a2b55d.slice/crio-34bd96b642dc5208db49b9af09c3c54aecfe577b229504fe93be416f652fab37 WatchSource:0}: Error finding container 34bd96b642dc5208db49b9af09c3c54aecfe577b229504fe93be416f652fab37: Status 404 returned error can't find the container with id 34bd96b642dc5208db49b9af09c3c54aecfe577b229504fe93be416f652fab37 Jul 17 02:00:09.248588 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:09.248549 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" event={"ID":"1c7f256a-5cce-4a10-b131-f985a0a2b55d","Type":"ContainerStarted","Data":"3637c5b6f6382a925639722434b02aca40c1c60ff560cf156eae91e16878ca0e"} Jul 17 02:00:09.249043 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:09.248610 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" event={"ID":"1c7f256a-5cce-4a10-b131-f985a0a2b55d","Type":"ContainerStarted","Data":"34bd96b642dc5208db49b9af09c3c54aecfe577b229504fe93be416f652fab37"} Jul 17 02:00:09.250540 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:09.250517 2583 generic.go:356] "Generic (PLEG): container finished" podID="abe01066-d70f-4be8-b79c-d9eef5feab5e" containerID="8566b52574678cea9d9ad8d1c130e7279e142e9f5412a5a78f7a1681e57cff7c" exitCode=6 Jul 17 02:00:09.250658 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:09.250549 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" event={"ID":"abe01066-d70f-4be8-b79c-d9eef5feab5e","Type":"ContainerDied","Data":"8566b52574678cea9d9ad8d1c130e7279e142e9f5412a5a78f7a1681e57cff7c"} Jul 17 02:00:09.250658 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:09.250583 2583 scope.go:117] "RemoveContainer" containerID="9f2c175ea6b1f7317755f4ebd6b1d0892c23105524d5811801300481f6540509" Jul 17 02:00:09.250812 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:09.250793 2583 scope.go:117] "RemoveContainer" containerID="8566b52574678cea9d9ad8d1c130e7279e142e9f5412a5a78f7a1681e57cff7c" Jul 17 02:00:09.251001 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:09.250984 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv_odh-ai-gateway-infra(abe01066-d70f-4be8-b79c-d9eef5feab5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" podUID="abe01066-d70f-4be8-b79c-d9eef5feab5e" Jul 17 02:00:09.264421 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:09.264380 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" podStartSLOduration=1.264363598 podStartE2EDuration="1.264363598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:00:09.262752892 +0000 UTC m=+2766.658367741" watchObservedRunningTime="2026-07-17 02:00:09.264363598 +0000 UTC m=+2766.659978448" Jul 17 02:00:10.257754 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:10.257714 2583 scope.go:117] "RemoveContainer" containerID="8566b52574678cea9d9ad8d1c130e7279e142e9f5412a5a78f7a1681e57cff7c" Jul 17 02:00:10.258194 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:10.257926 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv_odh-ai-gateway-infra(abe01066-d70f-4be8-b79c-d9eef5feab5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" podUID="abe01066-d70f-4be8-b79c-d9eef5feab5e" Jul 17 02:00:10.259053 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:10.259029 2583 generic.go:356] "Generic (PLEG): container finished" podID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" containerID="3637c5b6f6382a925639722434b02aca40c1c60ff560cf156eae91e16878ca0e" exitCode=6 Jul 17 02:00:10.259198 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:10.259082 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" event={"ID":"1c7f256a-5cce-4a10-b131-f985a0a2b55d","Type":"ContainerDied","Data":"3637c5b6f6382a925639722434b02aca40c1c60ff560cf156eae91e16878ca0e"} Jul 17 02:00:10.259242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:10.259232 2583 scope.go:117] "RemoveContainer" containerID="3637c5b6f6382a925639722434b02aca40c1c60ff560cf156eae91e16878ca0e" Jul 17 02:00:11.265872 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:11.265786 2583 generic.go:356] "Generic (PLEG): container finished" podID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" containerID="80db3e21221372a297c7cb41c0b02ef51bdcc25ac7732c7d7f69acc5ff8069ea" exitCode=6 Jul 17 02:00:11.265872 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:11.265830 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" event={"ID":"1c7f256a-5cce-4a10-b131-f985a0a2b55d","Type":"ContainerDied","Data":"80db3e21221372a297c7cb41c0b02ef51bdcc25ac7732c7d7f69acc5ff8069ea"} Jul 17 02:00:11.266371 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:11.265893 2583 scope.go:117] "RemoveContainer" containerID="3637c5b6f6382a925639722434b02aca40c1c60ff560cf156eae91e16878ca0e" Jul 17 02:00:11.266371 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:11.266131 2583 scope.go:117] "RemoveContainer" containerID="80db3e21221372a297c7cb41c0b02ef51bdcc25ac7732c7d7f69acc5ff8069ea" Jul 17 02:00:11.266371 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:11.266359 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj_odh-ai-gateway-infra(1c7f256a-5cce-4a10-b131-f985a0a2b55d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" podUID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" Jul 17 02:00:12.273173 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:12.273146 2583 scope.go:117] "RemoveContainer" containerID="80db3e21221372a297c7cb41c0b02ef51bdcc25ac7732c7d7f69acc5ff8069ea" Jul 17 02:00:12.273545 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:12.273350 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj_odh-ai-gateway-infra(1c7f256a-5cce-4a10-b131-f985a0a2b55d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" podUID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" Jul 17 02:00:13.328057 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:13.328020 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg"] Jul 17 02:00:13.332515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:13.332495 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg" Jul 17 02:00:13.339623 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:13.339585 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg"] Jul 17 02:00:13.388375 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:13.388346 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6fc007-10a9-4bdc-89a3-8fff93799964-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg\" (UID: \"dd6fc007-10a9-4bdc-89a3-8fff93799964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg" Jul 17 02:00:13.489221 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:13.489178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6fc007-10a9-4bdc-89a3-8fff93799964-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg\" (UID: \"dd6fc007-10a9-4bdc-89a3-8fff93799964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg" Jul 17 02:00:13.489810 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:13.489788 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6fc007-10a9-4bdc-89a3-8fff93799964-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg\" (UID: \"dd6fc007-10a9-4bdc-89a3-8fff93799964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg" Jul 17 02:00:13.645384 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:13.645310 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg" Jul 17 02:00:13.680871 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:13.680832 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:13.776348 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:13.776314 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg"] Jul 17 02:00:13.778345 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:13.778315 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:13.778791 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:00:13.778764 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6fc007_10a9_4bdc_89a3_8fff93799964.slice/crio-85d8bc2576b8dc215b3fe5d5ce93d462357dd8c1914266a86557e49d1dbf4dbc WatchSource:0}: Error finding container 85d8bc2576b8dc215b3fe5d5ce93d462357dd8c1914266a86557e49d1dbf4dbc: Status 404 returned error can't find the container with id 85d8bc2576b8dc215b3fe5d5ce93d462357dd8c1914266a86557e49d1dbf4dbc Jul 17 02:00:13.906291 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:13.906205 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:14.010119 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:14.010066 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:14.283818 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:14.283775 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg" event={"ID":"dd6fc007-10a9-4bdc-89a3-8fff93799964","Type":"ContainerStarted","Data":"1e5a4cfbe2739195accc2e0b3f7d181d6485d73d357eb50534d552fca6922273"} Jul 17 02:00:14.284033 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:14.283845 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg" event={"ID":"dd6fc007-10a9-4bdc-89a3-8fff93799964","Type":"ContainerStarted","Data":"85d8bc2576b8dc215b3fe5d5ce93d462357dd8c1914266a86557e49d1dbf4dbc"} Jul 17 02:00:14.301991 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:14.301936 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg" podStartSLOduration=1.301919379 podStartE2EDuration="1.301919379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:00:14.301802321 +0000 UTC m=+2771.697417171" watchObservedRunningTime="2026-07-17 02:00:14.301919379 +0000 UTC m=+2771.697534230" Jul 17 02:00:15.289759 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:15.289726 2583 generic.go:356] "Generic (PLEG): container finished" podID="dd6fc007-10a9-4bdc-89a3-8fff93799964" containerID="1e5a4cfbe2739195accc2e0b3f7d181d6485d73d357eb50534d552fca6922273" exitCode=0 Jul 17 02:00:15.290155 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:15.289794 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg" event={"ID":"dd6fc007-10a9-4bdc-89a3-8fff93799964","Type":"ContainerDied","Data":"1e5a4cfbe2739195accc2e0b3f7d181d6485d73d357eb50534d552fca6922273"} Jul 17 02:00:16.432446 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:16.432424 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg" Jul 17 02:00:16.511422 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:16.511378 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6fc007-10a9-4bdc-89a3-8fff93799964-maas-api-service-ca\") pod \"dd6fc007-10a9-4bdc-89a3-8fff93799964\" (UID: \"dd6fc007-10a9-4bdc-89a3-8fff93799964\") " Jul 17 02:00:16.511772 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:16.511749 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6fc007-10a9-4bdc-89a3-8fff93799964-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd6fc007-10a9-4bdc-89a3-8fff93799964" (UID: "dd6fc007-10a9-4bdc-89a3-8fff93799964"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:00:16.612010 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:16.611918 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd6fc007-10a9-4bdc-89a3-8fff93799964-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:00:17.301139 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:17.301112 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg" Jul 17 02:00:17.301295 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:17.301084 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg" event={"ID":"dd6fc007-10a9-4bdc-89a3-8fff93799964","Type":"ContainerDied","Data":"85d8bc2576b8dc215b3fe5d5ce93d462357dd8c1914266a86557e49d1dbf4dbc"} Jul 17 02:00:17.301295 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:17.301233 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85d8bc2576b8dc215b3fe5d5ce93d462357dd8c1914266a86557e49d1dbf4dbc" Jul 17 02:00:19.163000 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:19.162961 2583 scope.go:117] "RemoveContainer" containerID="24ed65fec22f1a7c79fc7a28278afb148f4a09c666f91bbff58899be9766bf9f" Jul 17 02:00:20.215867 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.215831 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz"] Jul 17 02:00:20.216313 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.216203 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd6fc007-10a9-4bdc-89a3-8fff93799964" containerName="revoke-keys" Jul 17 02:00:20.216313 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.216215 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd6fc007-10a9-4bdc-89a3-8fff93799964" containerName="revoke-keys" Jul 17 02:00:20.216313 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.216277 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd6fc007-10a9-4bdc-89a3-8fff93799964" containerName="revoke-keys" Jul 17 02:00:20.219361 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.219339 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" Jul 17 02:00:20.227136 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.227110 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz"] Jul 17 02:00:20.237737 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.237707 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/826c391e-0829-4645-bba7-a0d8857dc637-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz\" (UID: \"826c391e-0829-4645-bba7-a0d8857dc637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" Jul 17 02:00:20.318844 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.318812 2583 generic.go:356] "Generic (PLEG): container finished" podID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" containerID="fb03a790dc0578c47c9d8357f63ef86371cc98c63938753604ff44082a169c40" exitCode=6 Jul 17 02:00:20.319048 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.318896 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" event={"ID":"fc05e058-5f07-44a9-9e37-6bcb476ec68b","Type":"ContainerDied","Data":"fb03a790dc0578c47c9d8357f63ef86371cc98c63938753604ff44082a169c40"} Jul 17 02:00:20.319048 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.318972 2583 scope.go:117] "RemoveContainer" containerID="24ed65fec22f1a7c79fc7a28278afb148f4a09c666f91bbff58899be9766bf9f" Jul 17 02:00:20.319319 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.319297 2583 scope.go:117] "RemoveContainer" containerID="fb03a790dc0578c47c9d8357f63ef86371cc98c63938753604ff44082a169c40" Jul 17 02:00:20.319540 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:20.319503 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24_odh-ai-gateway-infra(fc05e058-5f07-44a9-9e37-6bcb476ec68b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" podUID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" Jul 17 02:00:20.338949 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.338915 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/826c391e-0829-4645-bba7-a0d8857dc637-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz\" (UID: \"826c391e-0829-4645-bba7-a0d8857dc637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" Jul 17 02:00:20.339531 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.339511 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/826c391e-0829-4645-bba7-a0d8857dc637-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz\" (UID: \"826c391e-0829-4645-bba7-a0d8857dc637\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" Jul 17 02:00:20.530730 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.530700 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" Jul 17 02:00:20.662988 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:20.662965 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz"] Jul 17 02:00:20.665077 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:00:20.665050 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826c391e_0829_4645_bba7_a0d8857dc637.slice/crio-2b06f5a61fb3fa4d935244bbbd058b5cd787d4a50d79d4af538bdd34f0fc2794 WatchSource:0}: Error finding container 2b06f5a61fb3fa4d935244bbbd058b5cd787d4a50d79d4af538bdd34f0fc2794: Status 404 returned error can't find the container with id 2b06f5a61fb3fa4d935244bbbd058b5cd787d4a50d79d4af538bdd34f0fc2794 Jul 17 02:00:21.326369 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:21.326330 2583 generic.go:356] "Generic (PLEG): container finished" podID="826c391e-0829-4645-bba7-a0d8857dc637" containerID="fdeec07236e99ce1a23279003cae7aeb6fc79ae45c3d5ee649b07ca1ff529c3c" exitCode=6 Jul 17 02:00:21.326849 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:21.326408 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" event={"ID":"826c391e-0829-4645-bba7-a0d8857dc637","Type":"ContainerDied","Data":"fdeec07236e99ce1a23279003cae7aeb6fc79ae45c3d5ee649b07ca1ff529c3c"} Jul 17 02:00:21.326849 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:21.326453 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" event={"ID":"826c391e-0829-4645-bba7-a0d8857dc637","Type":"ContainerStarted","Data":"2b06f5a61fb3fa4d935244bbbd058b5cd787d4a50d79d4af538bdd34f0fc2794"} Jul 17 02:00:21.326849 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:21.326618 2583 scope.go:117] "RemoveContainer" containerID="fdeec07236e99ce1a23279003cae7aeb6fc79ae45c3d5ee649b07ca1ff529c3c" Jul 17 02:00:21.346382 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:21.346356 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24"] Jul 17 02:00:21.469447 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:21.469418 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" Jul 17 02:00:21.547742 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:21.547706 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc05e058-5f07-44a9-9e37-6bcb476ec68b-maas-api-service-ca\") pod \"fc05e058-5f07-44a9-9e37-6bcb476ec68b\" (UID: \"fc05e058-5f07-44a9-9e37-6bcb476ec68b\") " Jul 17 02:00:21.548086 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:21.548062 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc05e058-5f07-44a9-9e37-6bcb476ec68b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fc05e058-5f07-44a9-9e37-6bcb476ec68b" (UID: "fc05e058-5f07-44a9-9e37-6bcb476ec68b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:00:21.649180 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:21.649151 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc05e058-5f07-44a9-9e37-6bcb476ec68b-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:00:22.339140 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:22.339085 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" Jul 17 02:00:22.339140 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:22.339129 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24" event={"ID":"fc05e058-5f07-44a9-9e37-6bcb476ec68b","Type":"ContainerDied","Data":"bdfb9fee25afd0abcab185a2ebea08e5a967f5160e928577d58daaaac68e7507"} Jul 17 02:00:22.339704 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:22.339216 2583 scope.go:117] "RemoveContainer" containerID="fb03a790dc0578c47c9d8357f63ef86371cc98c63938753604ff44082a169c40" Jul 17 02:00:22.341015 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:22.340990 2583 generic.go:356] "Generic (PLEG): container finished" podID="826c391e-0829-4645-bba7-a0d8857dc637" containerID="4d9f95b0f352e25d49e5d88a89e1f7cb34495c26b8c3028a8060157893f076bd" exitCode=6 Jul 17 02:00:22.341149 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:22.341047 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" event={"ID":"826c391e-0829-4645-bba7-a0d8857dc637","Type":"ContainerDied","Data":"4d9f95b0f352e25d49e5d88a89e1f7cb34495c26b8c3028a8060157893f076bd"} Jul 17 02:00:22.341341 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:22.341324 2583 scope.go:117] "RemoveContainer" containerID="4d9f95b0f352e25d49e5d88a89e1f7cb34495c26b8c3028a8060157893f076bd" Jul 17 02:00:22.341547 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:22.341527 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz_odh-ai-gateway-infra(826c391e-0829-4645-bba7-a0d8857dc637)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" podUID="826c391e-0829-4645-bba7-a0d8857dc637" Jul 17 02:00:22.348974 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:22.348957 2583 scope.go:117] "RemoveContainer" containerID="fdeec07236e99ce1a23279003cae7aeb6fc79ae45c3d5ee649b07ca1ff529c3c" Jul 17 02:00:22.370998 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:22.370970 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24"] Jul 17 02:00:22.374797 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:22.374764 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-9vm24"] Jul 17 02:00:23.168535 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:23.168501 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" path="/var/lib/kubelet/pods/fc05e058-5f07-44a9-9e37-6bcb476ec68b/volumes" Jul 17 02:00:23.349262 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:23.349233 2583 scope.go:117] "RemoveContainer" containerID="4d9f95b0f352e25d49e5d88a89e1f7cb34495c26b8c3028a8060157893f076bd" Jul 17 02:00:23.349617 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:23.349416 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz_odh-ai-gateway-infra(826c391e-0829-4645-bba7-a0d8857dc637)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" podUID="826c391e-0829-4645-bba7-a0d8857dc637" Jul 17 02:00:23.691002 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:23.690914 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7"] Jul 17 02:00:23.691183 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:23.691148 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" podUID="b579fca9-f19f-4090-973d-f8bea3e360df" containerName="maas-api" containerID="cri-o://b0c4cf132cff540574ddb33456dd8db42fa739e748f05865206c7cbb10cbb1d8" gracePeriod=30 Jul 17 02:00:23.723990 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:23.723961 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7f72-29737560-tt9d4"] Jul 17 02:00:23.726383 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:23.726346 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ab7f72-29737560-tt9d4"] Jul 17 02:00:23.729545 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:23.729518 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:23.940648 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:23.940624 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 02:00:23.963988 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:23.963907 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b579fca9-f19f-4090-973d-f8bea3e360df-openshift-service-ca\") pod \"b579fca9-f19f-4090-973d-f8bea3e360df\" (UID: \"b579fca9-f19f-4090-973d-f8bea3e360df\") " Jul 17 02:00:23.963988 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:23.963951 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppktv\" (UniqueName: \"kubernetes.io/projected/b579fca9-f19f-4090-973d-f8bea3e360df-kube-api-access-ppktv\") pod \"b579fca9-f19f-4090-973d-f8bea3e360df\" (UID: \"b579fca9-f19f-4090-973d-f8bea3e360df\") " Jul 17 02:00:23.964339 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:23.964313 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b579fca9-f19f-4090-973d-f8bea3e360df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b579fca9-f19f-4090-973d-f8bea3e360df" (UID: "b579fca9-f19f-4090-973d-f8bea3e360df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:00:23.966241 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:23.966216 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b579fca9-f19f-4090-973d-f8bea3e360df-kube-api-access-ppktv" (OuterVolumeSpecName: "kube-api-access-ppktv") pod "b579fca9-f19f-4090-973d-f8bea3e360df" (UID: "b579fca9-f19f-4090-973d-f8bea3e360df"). InnerVolumeSpecName "kube-api-access-ppktv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 02:00:24.065033 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.064995 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b579fca9-f19f-4090-973d-f8bea3e360df-maas-api-tls\") pod \"b579fca9-f19f-4090-973d-f8bea3e360df\" (UID: \"b579fca9-f19f-4090-973d-f8bea3e360df\") " Jul 17 02:00:24.065351 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.065328 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b579fca9-f19f-4090-973d-f8bea3e360df-openshift-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:00:24.065414 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.065358 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppktv\" (UniqueName: \"kubernetes.io/projected/b579fca9-f19f-4090-973d-f8bea3e360df-kube-api-access-ppktv\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:00:24.067322 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.067293 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b579fca9-f19f-4090-973d-f8bea3e360df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b579fca9-f19f-4090-973d-f8bea3e360df" (UID: "b579fca9-f19f-4090-973d-f8bea3e360df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 02:00:24.163089 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.163055 2583 scope.go:117] "RemoveContainer" containerID="8566b52574678cea9d9ad8d1c130e7279e142e9f5412a5a78f7a1681e57cff7c" Jul 17 02:00:24.166264 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.166246 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b579fca9-f19f-4090-973d-f8bea3e360df-maas-api-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:00:24.355306 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.355273 2583 generic.go:356] "Generic (PLEG): container finished" podID="b579fca9-f19f-4090-973d-f8bea3e360df" containerID="b0c4cf132cff540574ddb33456dd8db42fa739e748f05865206c7cbb10cbb1d8" exitCode=0 Jul 17 02:00:24.355749 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.355319 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" Jul 17 02:00:24.355749 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.355363 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" event={"ID":"b579fca9-f19f-4090-973d-f8bea3e360df","Type":"ContainerDied","Data":"b0c4cf132cff540574ddb33456dd8db42fa739e748f05865206c7cbb10cbb1d8"} Jul 17 02:00:24.355749 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.355424 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7" event={"ID":"b579fca9-f19f-4090-973d-f8bea3e360df","Type":"ContainerDied","Data":"d0e82c5bbbff306b7ba7f73704a01b95b5678eefb8cd15fb88cec3ea364f5e1b"} Jul 17 02:00:24.355749 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.355454 2583 scope.go:117] "RemoveContainer" containerID="b0c4cf132cff540574ddb33456dd8db42fa739e748f05865206c7cbb10cbb1d8" Jul 17 02:00:24.357491 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.357462 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" event={"ID":"abe01066-d70f-4be8-b79c-d9eef5feab5e","Type":"ContainerStarted","Data":"0ad7a4796528d26301c07463e381c97a0b1620247f540c0086ed8254ddea51e2"} Jul 17 02:00:24.364852 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.364837 2583 scope.go:117] "RemoveContainer" containerID="b0c4cf132cff540574ddb33456dd8db42fa739e748f05865206c7cbb10cbb1d8" Jul 17 02:00:24.365141 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:24.365120 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c4cf132cff540574ddb33456dd8db42fa739e748f05865206c7cbb10cbb1d8\": container with ID starting with b0c4cf132cff540574ddb33456dd8db42fa739e748f05865206c7cbb10cbb1d8 not found: ID does not exist" containerID="b0c4cf132cff540574ddb33456dd8db42fa739e748f05865206c7cbb10cbb1d8" Jul 17 02:00:24.365228 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.365150 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c4cf132cff540574ddb33456dd8db42fa739e748f05865206c7cbb10cbb1d8"} err="failed to get container status \"b0c4cf132cff540574ddb33456dd8db42fa739e748f05865206c7cbb10cbb1d8\": rpc error: code = NotFound desc = could not find container \"b0c4cf132cff540574ddb33456dd8db42fa739e748f05865206c7cbb10cbb1d8\": container with ID starting with b0c4cf132cff540574ddb33456dd8db42fa739e748f05865206c7cbb10cbb1d8 not found: ID does not exist" Jul 17 02:00:24.385125 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.385081 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7"] Jul 17 02:00:24.388710 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:24.388685 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7f72-6478755fc7-n44d7"] Jul 17 02:00:25.168211 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:25.168181 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f03564b-ae75-4400-bbe3-c8e4a08b4e48" path="/var/lib/kubelet/pods/3f03564b-ae75-4400-bbe3-c8e4a08b4e48/volumes" Jul 17 02:00:25.168538 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:25.168525 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b579fca9-f19f-4090-973d-f8bea3e360df" path="/var/lib/kubelet/pods/b579fca9-f19f-4090-973d-f8bea3e360df/volumes" Jul 17 02:00:25.365359 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:25.365326 2583 generic.go:356] "Generic (PLEG): container finished" podID="abe01066-d70f-4be8-b79c-d9eef5feab5e" containerID="0ad7a4796528d26301c07463e381c97a0b1620247f540c0086ed8254ddea51e2" exitCode=6 Jul 17 02:00:25.365771 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:25.365383 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" event={"ID":"abe01066-d70f-4be8-b79c-d9eef5feab5e","Type":"ContainerDied","Data":"0ad7a4796528d26301c07463e381c97a0b1620247f540c0086ed8254ddea51e2"} Jul 17 02:00:25.365771 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:25.365439 2583 scope.go:117] "RemoveContainer" containerID="8566b52574678cea9d9ad8d1c130e7279e142e9f5412a5a78f7a1681e57cff7c" Jul 17 02:00:25.365771 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:25.365720 2583 scope.go:117] "RemoveContainer" containerID="0ad7a4796528d26301c07463e381c97a0b1620247f540c0086ed8254ddea51e2" Jul 17 02:00:25.365940 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:25.365921 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv_odh-ai-gateway-infra(abe01066-d70f-4be8-b79c-d9eef5feab5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" podUID="abe01066-d70f-4be8-b79c-d9eef5feab5e" Jul 17 02:00:25.389907 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:25.389872 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv"] Jul 17 02:00:26.163385 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:26.163356 2583 scope.go:117] "RemoveContainer" containerID="80db3e21221372a297c7cb41c0b02ef51bdcc25ac7732c7d7f69acc5ff8069ea" Jul 17 02:00:26.533292 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:26.533266 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" Jul 17 02:00:26.587218 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:26.587175 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe01066-d70f-4be8-b79c-d9eef5feab5e-maas-api-service-ca\") pod \"abe01066-d70f-4be8-b79c-d9eef5feab5e\" (UID: \"abe01066-d70f-4be8-b79c-d9eef5feab5e\") " Jul 17 02:00:26.587501 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:26.587475 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe01066-d70f-4be8-b79c-d9eef5feab5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abe01066-d70f-4be8-b79c-d9eef5feab5e" (UID: "abe01066-d70f-4be8-b79c-d9eef5feab5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:00:26.688344 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:26.688263 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe01066-d70f-4be8-b79c-d9eef5feab5e-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:00:27.378259 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:27.378230 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" Jul 17 02:00:27.378259 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:27.378243 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv" event={"ID":"abe01066-d70f-4be8-b79c-d9eef5feab5e","Type":"ContainerDied","Data":"9dcf7a7a23410f03dad2462b7bf70e29dcb0bfa9ee7d6e3e090ffee2eef80349"} Jul 17 02:00:27.378533 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:27.378311 2583 scope.go:117] "RemoveContainer" containerID="0ad7a4796528d26301c07463e381c97a0b1620247f540c0086ed8254ddea51e2" Jul 17 02:00:27.380223 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:27.380192 2583 generic.go:356] "Generic (PLEG): container finished" podID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" containerID="9ef3a92be589bc40a8573641bc84bdc25409cafe97f4a76df800544fc0b8a503" exitCode=6 Jul 17 02:00:27.380340 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:27.380240 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" event={"ID":"1c7f256a-5cce-4a10-b131-f985a0a2b55d","Type":"ContainerDied","Data":"9ef3a92be589bc40a8573641bc84bdc25409cafe97f4a76df800544fc0b8a503"} Jul 17 02:00:27.380548 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:27.380530 2583 scope.go:117] "RemoveContainer" containerID="9ef3a92be589bc40a8573641bc84bdc25409cafe97f4a76df800544fc0b8a503" Jul 17 02:00:27.380761 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:27.380740 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj_odh-ai-gateway-infra(1c7f256a-5cce-4a10-b131-f985a0a2b55d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" podUID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" Jul 17 02:00:27.386979 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:27.386963 2583 scope.go:117] "RemoveContainer" containerID="80db3e21221372a297c7cb41c0b02ef51bdcc25ac7732c7d7f69acc5ff8069ea" Jul 17 02:00:27.394415 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:27.394385 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv"] Jul 17 02:00:27.399289 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:27.399264 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-vqxnv"] Jul 17 02:00:28.410920 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:28.410890 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:28.418777 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:28.418745 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj"] Jul 17 02:00:28.552932 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:28.552905 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" Jul 17 02:00:28.607921 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:28.607875 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f256a-5cce-4a10-b131-f985a0a2b55d-maas-api-service-ca\") pod \"1c7f256a-5cce-4a10-b131-f985a0a2b55d\" (UID: \"1c7f256a-5cce-4a10-b131-f985a0a2b55d\") " Jul 17 02:00:28.608286 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:28.608264 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7f256a-5cce-4a10-b131-f985a0a2b55d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c7f256a-5cce-4a10-b131-f985a0a2b55d" (UID: "1c7f256a-5cce-4a10-b131-f985a0a2b55d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:00:28.709619 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:28.709531 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f256a-5cce-4a10-b131-f985a0a2b55d-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:00:29.169270 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:29.169237 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe01066-d70f-4be8-b79c-d9eef5feab5e" path="/var/lib/kubelet/pods/abe01066-d70f-4be8-b79c-d9eef5feab5e/volumes" Jul 17 02:00:29.393863 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:29.393835 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" Jul 17 02:00:29.394035 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:29.393831 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj" event={"ID":"1c7f256a-5cce-4a10-b131-f985a0a2b55d","Type":"ContainerDied","Data":"34bd96b642dc5208db49b9af09c3c54aecfe577b229504fe93be416f652fab37"} Jul 17 02:00:29.394035 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:29.393986 2583 scope.go:117] "RemoveContainer" containerID="9ef3a92be589bc40a8573641bc84bdc25409cafe97f4a76df800544fc0b8a503" Jul 17 02:00:29.409402 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:29.409371 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj"] Jul 17 02:00:29.412939 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:29.412911 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-hnlcj"] Jul 17 02:00:31.167216 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:31.167183 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" path="/var/lib/kubelet/pods/1c7f256a-5cce-4a10-b131-f985a0a2b55d/volumes" Jul 17 02:00:34.434693 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:34.434657 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:34.536339 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:34.536306 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:38.162690 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:38.162658 2583 scope.go:117] "RemoveContainer" containerID="4d9f95b0f352e25d49e5d88a89e1f7cb34495c26b8c3028a8060157893f076bd" Jul 17 02:00:38.444412 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:38.444382 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" event={"ID":"826c391e-0829-4645-bba7-a0d8857dc637","Type":"ContainerStarted","Data":"6ec73fc1550c8b47caf05c4634063a97044baac713772b16a8050239ff41d0b7"} Jul 17 02:00:38.460741 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:38.460691 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" podStartSLOduration=18.460672451 podStartE2EDuration="18.460672451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:00:38.45857391 +0000 UTC m=+2795.854188763" watchObservedRunningTime="2026-07-17 02:00:38.460672451 +0000 UTC m=+2795.856287301" Jul 17 02:00:39.091808 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:39.091779 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:39.319498 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:39.319467 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:39.447037 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:39.446948 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:39.450792 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:39.450759 2583 generic.go:356] "Generic (PLEG): container finished" podID="826c391e-0829-4645-bba7-a0d8857dc637" containerID="6ec73fc1550c8b47caf05c4634063a97044baac713772b16a8050239ff41d0b7" exitCode=6 Jul 17 02:00:39.450926 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:39.450830 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" event={"ID":"826c391e-0829-4645-bba7-a0d8857dc637","Type":"ContainerDied","Data":"6ec73fc1550c8b47caf05c4634063a97044baac713772b16a8050239ff41d0b7"} Jul 17 02:00:39.450926 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:39.450887 2583 scope.go:117] "RemoveContainer" containerID="4d9f95b0f352e25d49e5d88a89e1f7cb34495c26b8c3028a8060157893f076bd" Jul 17 02:00:39.451198 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:39.451178 2583 scope.go:117] "RemoveContainer" containerID="6ec73fc1550c8b47caf05c4634063a97044baac713772b16a8050239ff41d0b7" Jul 17 02:00:39.451379 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:39.451358 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz_odh-ai-gateway-infra(826c391e-0829-4645-bba7-a0d8857dc637)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" podUID="826c391e-0829-4645-bba7-a0d8857dc637" Jul 17 02:00:39.497308 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:39.497275 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz"] Jul 17 02:00:39.673781 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:39.673745 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:40.034693 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:40.034655 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:40.272675 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:40.272645 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:40.590362 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:40.590338 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" Jul 17 02:00:40.718869 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:40.718775 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/826c391e-0829-4645-bba7-a0d8857dc637-maas-api-service-ca\") pod \"826c391e-0829-4645-bba7-a0d8857dc637\" (UID: \"826c391e-0829-4645-bba7-a0d8857dc637\") " Jul 17 02:00:40.719192 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:40.719168 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826c391e-0829-4645-bba7-a0d8857dc637-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "826c391e-0829-4645-bba7-a0d8857dc637" (UID: "826c391e-0829-4645-bba7-a0d8857dc637"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:00:40.819792 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:40.819751 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/826c391e-0829-4645-bba7-a0d8857dc637-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:00:41.463652 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:41.463563 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" event={"ID":"826c391e-0829-4645-bba7-a0d8857dc637","Type":"ContainerDied","Data":"2b06f5a61fb3fa4d935244bbbd058b5cd787d4a50d79d4af538bdd34f0fc2794"} Jul 17 02:00:41.463652 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:41.463586 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz" Jul 17 02:00:41.463652 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:41.463619 2583 scope.go:117] "RemoveContainer" containerID="6ec73fc1550c8b47caf05c4634063a97044baac713772b16a8050239ff41d0b7" Jul 17 02:00:41.481910 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:41.481849 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz"] Jul 17 02:00:41.483580 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:41.483560 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-lnfcz"] Jul 17 02:00:42.476583 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.476543 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:42.564941 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.564899 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:42.565987 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.565967 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l"] Jul 17 02:00:42.566318 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566304 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" containerName="revoke-keys" Jul 17 02:00:42.566368 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566321 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" containerName="revoke-keys" Jul 17 02:00:42.566368 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566330 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe01066-d70f-4be8-b79c-d9eef5feab5e" containerName="revoke-keys" Jul 17 02:00:42.566368 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566336 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe01066-d70f-4be8-b79c-d9eef5feab5e" containerName="revoke-keys" Jul 17 02:00:42.566368 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566344 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" containerName="revoke-keys" Jul 17 02:00:42.566368 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566352 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" containerName="revoke-keys" Jul 17 02:00:42.566368 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566367 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566373 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566380 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566385 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566391 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="826c391e-0829-4645-bba7-a0d8857dc637" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566397 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="826c391e-0829-4645-bba7-a0d8857dc637" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566409 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b579fca9-f19f-4090-973d-f8bea3e360df" containerName="maas-api" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566414 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b579fca9-f19f-4090-973d-f8bea3e360df" containerName="maas-api" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566420 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe01066-d70f-4be8-b79c-d9eef5feab5e" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566425 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe01066-d70f-4be8-b79c-d9eef5feab5e" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566433 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566441 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566447 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="826c391e-0829-4645-bba7-a0d8857dc637" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566452 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="826c391e-0829-4645-bba7-a0d8857dc637" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566461 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566465 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566522 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe01066-d70f-4be8-b79c-d9eef5feab5e" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566534 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566542 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="826c391e-0829-4645-bba7-a0d8857dc637" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566551 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566559 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" containerName="revoke-keys" Jul 17 02:00:42.566563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566566 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b579fca9-f19f-4090-973d-f8bea3e360df" containerName="maas-api" Jul 17 02:00:42.567193 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566572 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="826c391e-0829-4645-bba7-a0d8857dc637" containerName="revoke-keys" Jul 17 02:00:42.567193 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566592 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" containerName="revoke-keys" Jul 17 02:00:42.567193 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.566600 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe01066-d70f-4be8-b79c-d9eef5feab5e" containerName="revoke-keys" Jul 17 02:00:42.570698 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.570681 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:00:42.573023 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.572997 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3413e7-serving-cert\"" Jul 17 02:00:42.573157 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.573047 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 02:00:42.573950 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.573933 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5l8p8\"" Jul 17 02:00:42.574048 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.573946 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 02:00:42.579854 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.579825 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l"] Jul 17 02:00:42.636772 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.636741 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17919854-14cf-4b8f-a63d-01e2617a71cb-openshift-service-ca\") pod \"maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l\" (UID: \"17919854-14cf-4b8f-a63d-01e2617a71cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:00:42.636943 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.636788 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nffwk\" (UniqueName: \"kubernetes.io/projected/17919854-14cf-4b8f-a63d-01e2617a71cb-kube-api-access-nffwk\") pod \"maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l\" (UID: \"17919854-14cf-4b8f-a63d-01e2617a71cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:00:42.636943 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.636871 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17919854-14cf-4b8f-a63d-01e2617a71cb-maas-api-tls\") pod \"maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l\" (UID: \"17919854-14cf-4b8f-a63d-01e2617a71cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:00:42.670791 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.670751 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:42.738338 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.738251 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17919854-14cf-4b8f-a63d-01e2617a71cb-maas-api-tls\") pod \"maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l\" (UID: \"17919854-14cf-4b8f-a63d-01e2617a71cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:00:42.738338 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.738309 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17919854-14cf-4b8f-a63d-01e2617a71cb-openshift-service-ca\") pod \"maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l\" (UID: \"17919854-14cf-4b8f-a63d-01e2617a71cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:00:42.738568 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.738357 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nffwk\" (UniqueName: \"kubernetes.io/projected/17919854-14cf-4b8f-a63d-01e2617a71cb-kube-api-access-nffwk\") pod \"maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l\" (UID: \"17919854-14cf-4b8f-a63d-01e2617a71cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:00:42.738568 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:42.738424 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-serving-cert: secret "maas-api-e2e-api-a-3413e7-serving-cert" not found Jul 17 02:00:42.738568 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:42.738498 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17919854-14cf-4b8f-a63d-01e2617a71cb-maas-api-tls podName:17919854-14cf-4b8f-a63d-01e2617a71cb nodeName:}" failed. No retries permitted until 2026-07-17 02:00:43.238481217 +0000 UTC m=+2800.634096049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17919854-14cf-4b8f-a63d-01e2617a71cb-maas-api-tls") pod "maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" (UID: "17919854-14cf-4b8f-a63d-01e2617a71cb") : secret "maas-api-e2e-api-a-3413e7-serving-cert" not found Jul 17 02:00:42.739043 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.739025 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17919854-14cf-4b8f-a63d-01e2617a71cb-openshift-service-ca\") pod \"maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l\" (UID: \"17919854-14cf-4b8f-a63d-01e2617a71cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:00:42.750302 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.750275 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nffwk\" (UniqueName: \"kubernetes.io/projected/17919854-14cf-4b8f-a63d-01e2617a71cb-kube-api-access-nffwk\") pod \"maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l\" (UID: \"17919854-14cf-4b8f-a63d-01e2617a71cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:00:42.868271 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:42.868246 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:43.168046 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:43.168012 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="826c391e-0829-4645-bba7-a0d8857dc637" path="/var/lib/kubelet/pods/826c391e-0829-4645-bba7-a0d8857dc637/volumes" Jul 17 02:00:43.218822 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:43.218779 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:43.242490 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:43.242449 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17919854-14cf-4b8f-a63d-01e2617a71cb-maas-api-tls\") pod \"maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l\" (UID: \"17919854-14cf-4b8f-a63d-01e2617a71cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:00:43.245062 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:43.245036 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17919854-14cf-4b8f-a63d-01e2617a71cb-maas-api-tls\") pod \"maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l\" (UID: \"17919854-14cf-4b8f-a63d-01e2617a71cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:00:43.314620 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:43.314587 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:43.482618 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:43.482530 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:00:43.619000 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:43.618855 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l"] Jul 17 02:00:43.621309 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:00:43.621283 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17919854_14cf_4b8f_a63d_01e2617a71cb.slice/crio-66aa1f1d03db6ea708f9d09d5dab374dac41f88fb7ea7c8401bc1c61b54a4e83 WatchSource:0}: Error finding container 66aa1f1d03db6ea708f9d09d5dab374dac41f88fb7ea7c8401bc1c61b54a4e83: Status 404 returned error can't find the container with id 66aa1f1d03db6ea708f9d09d5dab374dac41f88fb7ea7c8401bc1c61b54a4e83 Jul 17 02:00:43.622585 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:43.622569 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 02:00:44.484789 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:44.484754 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" event={"ID":"17919854-14cf-4b8f-a63d-01e2617a71cb","Type":"ContainerStarted","Data":"5f960f88c9a9d6278d6d93e2e932e3e0e239a339ae4c861980dc73b1951eeca4"} Jul 17 02:00:44.485216 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:44.484826 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" event={"ID":"17919854-14cf-4b8f-a63d-01e2617a71cb","Type":"ContainerStarted","Data":"66aa1f1d03db6ea708f9d09d5dab374dac41f88fb7ea7c8401bc1c61b54a4e83"} Jul 17 02:00:44.485216 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:44.484860 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:00:44.514984 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:44.514927 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" podStartSLOduration=2.143057165 podStartE2EDuration="2.514911587s" totalImagesPullingTime="371.854422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:00:44.513185623 +0000 UTC m=+2801.908800474" watchObservedRunningTime="2026-07-17 02:00:44.514911587 +0000 UTC m=+2801.910526470" Jul 17 02:00:46.955362 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:46.955322 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:47.411413 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:47.411382 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:47.709823 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:47.709746 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:47.809828 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:47.809796 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:50.495150 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.495125 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:00:50.702638 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.702607 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:50.800275 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.800237 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:50.800897 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.800873 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf"] Jul 17 02:00:50.801284 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.801266 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="826c391e-0829-4645-bba7-a0d8857dc637" containerName="revoke-keys" Jul 17 02:00:50.801349 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.801288 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="826c391e-0829-4645-bba7-a0d8857dc637" containerName="revoke-keys" Jul 17 02:00:50.801349 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.801338 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe01066-d70f-4be8-b79c-d9eef5feab5e" containerName="revoke-keys" Jul 17 02:00:50.801415 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.801350 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe01066-d70f-4be8-b79c-d9eef5feab5e" containerName="revoke-keys" Jul 17 02:00:50.801475 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.801464 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c7f256a-5cce-4a10-b131-f985a0a2b55d" containerName="revoke-keys" Jul 17 02:00:50.801518 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.801478 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="826c391e-0829-4645-bba7-a0d8857dc637" containerName="revoke-keys" Jul 17 02:00:50.801518 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.801488 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc05e058-5f07-44a9-9e37-6bcb476ec68b" containerName="revoke-keys" Jul 17 02:00:50.801518 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.801494 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe01066-d70f-4be8-b79c-d9eef5feab5e" containerName="revoke-keys" Jul 17 02:00:50.804594 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.804578 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:00:50.806741 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.806719 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d8cc4a-serving-cert\"" Jul 17 02:00:50.813028 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.813002 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf"] Jul 17 02:00:50.869272 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.869238 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:50.911947 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.911912 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wll5f\" (UniqueName: \"kubernetes.io/projected/66324c41-643e-4f3a-bdce-bcfb127afd86-kube-api-access-wll5f\") pod \"maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf\" (UID: \"66324c41-643e-4f3a-bdce-bcfb127afd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:00:50.912167 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.911966 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66324c41-643e-4f3a-bdce-bcfb127afd86-openshift-service-ca\") pod \"maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf\" (UID: \"66324c41-643e-4f3a-bdce-bcfb127afd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:00:50.912167 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:50.912033 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66324c41-643e-4f3a-bdce-bcfb127afd86-maas-api-tls\") pod \"maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf\" (UID: \"66324c41-643e-4f3a-bdce-bcfb127afd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:00:51.012833 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:51.012792 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wll5f\" (UniqueName: \"kubernetes.io/projected/66324c41-643e-4f3a-bdce-bcfb127afd86-kube-api-access-wll5f\") pod \"maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf\" (UID: \"66324c41-643e-4f3a-bdce-bcfb127afd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:00:51.013013 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:51.012839 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66324c41-643e-4f3a-bdce-bcfb127afd86-openshift-service-ca\") pod \"maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf\" (UID: \"66324c41-643e-4f3a-bdce-bcfb127afd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:00:51.013066 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:51.013041 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66324c41-643e-4f3a-bdce-bcfb127afd86-maas-api-tls\") pod \"maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf\" (UID: \"66324c41-643e-4f3a-bdce-bcfb127afd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:00:51.013493 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:51.013472 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66324c41-643e-4f3a-bdce-bcfb127afd86-openshift-service-ca\") pod \"maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf\" (UID: \"66324c41-643e-4f3a-bdce-bcfb127afd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:00:51.015665 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:51.015639 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66324c41-643e-4f3a-bdce-bcfb127afd86-maas-api-tls\") pod \"maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf\" (UID: \"66324c41-643e-4f3a-bdce-bcfb127afd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:00:51.022212 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:51.022179 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wll5f\" (UniqueName: \"kubernetes.io/projected/66324c41-643e-4f3a-bdce-bcfb127afd86-kube-api-access-wll5f\") pod \"maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf\" (UID: \"66324c41-643e-4f3a-bdce-bcfb127afd86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:00:51.038776 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:51.038749 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:51.116974 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:51.116874 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:00:51.253829 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:51.253805 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf"] Jul 17 02:00:51.256439 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:00:51.256408 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66324c41_643e_4f3a_bdce_bcfb127afd86.slice/crio-a2c851b382a6005b4c7c735a7802ab7560d44bee856929b16ba8887d3809a01f WatchSource:0}: Error finding container a2c851b382a6005b4c7c735a7802ab7560d44bee856929b16ba8887d3809a01f: Status 404 returned error can't find the container with id a2c851b382a6005b4c7c735a7802ab7560d44bee856929b16ba8887d3809a01f Jul 17 02:00:51.477878 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:51.477786 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:51.518475 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:51.518439 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" event={"ID":"66324c41-643e-4f3a-bdce-bcfb127afd86","Type":"ContainerStarted","Data":"a2c851b382a6005b4c7c735a7802ab7560d44bee856929b16ba8887d3809a01f"} Jul 17 02:00:51.578050 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:51.578020 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:52.524464 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:52.524429 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" event={"ID":"66324c41-643e-4f3a-bdce-bcfb127afd86","Type":"ContainerStarted","Data":"bbf606f5e1b3fafce03c76db619aa8fbbdbde372f40991b8e0d9b44c5397e7de"} Jul 17 02:00:52.524850 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:52.524553 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:00:52.541881 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:52.541831 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" podStartSLOduration=1.9804545839999999 podStartE2EDuration="2.54181689s" totalImagesPullingTime="561.362306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:00:52.54002216 +0000 UTC m=+2809.935637009" watchObservedRunningTime="2026-07-17 02:00:52.54181689 +0000 UTC m=+2809.937431801" Jul 17 02:00:53.691347 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:53.691313 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv"] Jul 17 02:00:53.694616 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:53.694600 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" Jul 17 02:00:53.696801 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:53.696786 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 02:00:53.702957 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:53.702932 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv"] Jul 17 02:00:53.838312 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:53.838274 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec3bb55-291f-494d-a66c-74a857e2626e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv\" (UID: \"eec3bb55-291f-494d-a66c-74a857e2626e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" Jul 17 02:00:53.938701 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:53.938655 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec3bb55-291f-494d-a66c-74a857e2626e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv\" (UID: \"eec3bb55-291f-494d-a66c-74a857e2626e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" Jul 17 02:00:53.939314 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:53.939293 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec3bb55-291f-494d-a66c-74a857e2626e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv\" (UID: \"eec3bb55-291f-494d-a66c-74a857e2626e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" Jul 17 02:00:54.006839 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:54.006747 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" Jul 17 02:00:54.136897 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:54.136871 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv"] Jul 17 02:00:54.138544 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:00:54.138518 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec3bb55_291f_494d_a66c_74a857e2626e.slice/crio-bce35bd2421d0397db4693f861acfe451640196ab1fb872aeebbdaa3eb04ded4 WatchSource:0}: Error finding container bce35bd2421d0397db4693f861acfe451640196ab1fb872aeebbdaa3eb04ded4: Status 404 returned error can't find the container with id bce35bd2421d0397db4693f861acfe451640196ab1fb872aeebbdaa3eb04ded4 Jul 17 02:00:54.535054 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:54.535016 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" event={"ID":"eec3bb55-291f-494d-a66c-74a857e2626e","Type":"ContainerStarted","Data":"09dc615f22858786a1c6b360495aedcb9495befe17362a5cb9195bb0a859bc3a"} Jul 17 02:00:54.535300 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:54.535066 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" event={"ID":"eec3bb55-291f-494d-a66c-74a857e2626e","Type":"ContainerStarted","Data":"bce35bd2421d0397db4693f861acfe451640196ab1fb872aeebbdaa3eb04ded4"} Jul 17 02:00:54.550293 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:54.550234 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" podStartSLOduration=1.5502170830000002 podStartE2EDuration="1.550217083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:00:54.548408661 +0000 UTC m=+2811.944023513" watchObservedRunningTime="2026-07-17 02:00:54.550217083 +0000 UTC m=+2811.945831934" Jul 17 02:00:55.188194 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:55.188117 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:00:55.542627 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:55.542591 2583 generic.go:356] "Generic (PLEG): container finished" podID="eec3bb55-291f-494d-a66c-74a857e2626e" containerID="09dc615f22858786a1c6b360495aedcb9495befe17362a5cb9195bb0a859bc3a" exitCode=6 Jul 17 02:00:55.542842 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:55.542676 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" event={"ID":"eec3bb55-291f-494d-a66c-74a857e2626e","Type":"ContainerDied","Data":"09dc615f22858786a1c6b360495aedcb9495befe17362a5cb9195bb0a859bc3a"} Jul 17 02:00:55.542919 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:55.542862 2583 scope.go:117] "RemoveContainer" containerID="09dc615f22858786a1c6b360495aedcb9495befe17362a5cb9195bb0a859bc3a" Jul 17 02:00:56.550077 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:56.550031 2583 generic.go:356] "Generic (PLEG): container finished" podID="eec3bb55-291f-494d-a66c-74a857e2626e" containerID="aaa3464d07f20df99567ea27e4f8da5ddb95b07991212b881b1be2957e57fde4" exitCode=6 Jul 17 02:00:56.550573 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:56.550092 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" event={"ID":"eec3bb55-291f-494d-a66c-74a857e2626e","Type":"ContainerDied","Data":"aaa3464d07f20df99567ea27e4f8da5ddb95b07991212b881b1be2957e57fde4"} Jul 17 02:00:56.550573 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:56.550214 2583 scope.go:117] "RemoveContainer" containerID="09dc615f22858786a1c6b360495aedcb9495befe17362a5cb9195bb0a859bc3a" Jul 17 02:00:56.550573 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:56.550476 2583 scope.go:117] "RemoveContainer" containerID="aaa3464d07f20df99567ea27e4f8da5ddb95b07991212b881b1be2957e57fde4" Jul 17 02:00:56.550722 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:56.550705 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv_odh-ai-gateway-infra(eec3bb55-291f-494d-a66c-74a857e2626e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" podUID="eec3bb55-291f-494d-a66c-74a857e2626e" Jul 17 02:00:57.556940 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:57.556912 2583 scope.go:117] "RemoveContainer" containerID="aaa3464d07f20df99567ea27e4f8da5ddb95b07991212b881b1be2957e57fde4" Jul 17 02:00:57.557354 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:00:57.557085 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv_odh-ai-gateway-infra(eec3bb55-291f-494d-a66c-74a857e2626e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" podUID="eec3bb55-291f-494d-a66c-74a857e2626e" Jul 17 02:00:58.534663 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:00:58.534635 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:01:03.207020 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:03.206986 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8"] Jul 17 02:01:03.211690 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:03.211674 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8" Jul 17 02:01:03.217948 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:03.217917 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8"] Jul 17 02:01:03.325731 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:03.325690 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d34eef-a2de-4df8-aa17-5c2958a86ba6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8\" (UID: \"50d34eef-a2de-4df8-aa17-5c2958a86ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8" Jul 17 02:01:03.426490 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:03.426454 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d34eef-a2de-4df8-aa17-5c2958a86ba6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8\" (UID: \"50d34eef-a2de-4df8-aa17-5c2958a86ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8" Jul 17 02:01:03.427071 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:03.427051 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d34eef-a2de-4df8-aa17-5c2958a86ba6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8\" (UID: \"50d34eef-a2de-4df8-aa17-5c2958a86ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8" Jul 17 02:01:03.524318 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:03.524289 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8" Jul 17 02:01:03.660618 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:03.660595 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8"] Jul 17 02:01:03.662991 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:01:03.662965 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d34eef_a2de_4df8_aa17_5c2958a86ba6.slice/crio-316ff8e3f3660ecce4f6d05a198af323b6f67f085202ff03fbc0a663aee0e46d WatchSource:0}: Error finding container 316ff8e3f3660ecce4f6d05a198af323b6f67f085202ff03fbc0a663aee0e46d: Status 404 returned error can't find the container with id 316ff8e3f3660ecce4f6d05a198af323b6f67f085202ff03fbc0a663aee0e46d Jul 17 02:01:03.796474 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:03.796439 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:03.859402 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:03.859364 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:04.595741 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:04.595652 2583 generic.go:356] "Generic (PLEG): container finished" podID="50d34eef-a2de-4df8-aa17-5c2958a86ba6" containerID="3b6f18ef19a6406aae62139c8018dfccc2cdd3ecf3b2f08391f4d05a45c1703a" exitCode=0 Jul 17 02:01:04.595741 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:04.595730 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8" event={"ID":"50d34eef-a2de-4df8-aa17-5c2958a86ba6","Type":"ContainerDied","Data":"3b6f18ef19a6406aae62139c8018dfccc2cdd3ecf3b2f08391f4d05a45c1703a"} Jul 17 02:01:04.596173 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:04.595765 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8" event={"ID":"50d34eef-a2de-4df8-aa17-5c2958a86ba6","Type":"ContainerStarted","Data":"316ff8e3f3660ecce4f6d05a198af323b6f67f085202ff03fbc0a663aee0e46d"} Jul 17 02:01:05.735268 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:05.735242 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8" Jul 17 02:01:05.848287 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:05.848256 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d34eef-a2de-4df8-aa17-5c2958a86ba6-maas-api-service-ca\") pod \"50d34eef-a2de-4df8-aa17-5c2958a86ba6\" (UID: \"50d34eef-a2de-4df8-aa17-5c2958a86ba6\") " Jul 17 02:01:05.848620 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:05.848597 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d34eef-a2de-4df8-aa17-5c2958a86ba6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50d34eef-a2de-4df8-aa17-5c2958a86ba6" (UID: "50d34eef-a2de-4df8-aa17-5c2958a86ba6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:01:05.949663 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:05.949560 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d34eef-a2de-4df8-aa17-5c2958a86ba6-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:01:06.290545 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:06.290506 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl"] Jul 17 02:01:06.290884 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:06.290870 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d34eef-a2de-4df8-aa17-5c2958a86ba6" containerName="revoke-keys" Jul 17 02:01:06.290930 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:06.290886 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d34eef-a2de-4df8-aa17-5c2958a86ba6" containerName="revoke-keys" Jul 17 02:01:06.290969 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:06.290956 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d34eef-a2de-4df8-aa17-5c2958a86ba6" containerName="revoke-keys" Jul 17 02:01:06.294299 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:06.294284 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" Jul 17 02:01:06.300838 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:06.300806 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl"] Jul 17 02:01:06.454308 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:06.454269 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl\" (UID: \"a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" Jul 17 02:01:06.555249 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:06.555156 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl\" (UID: \"a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" Jul 17 02:01:06.555861 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:06.555833 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl\" (UID: \"a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" Jul 17 02:01:06.604985 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:06.604960 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" Jul 17 02:01:06.606210 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:06.606188 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8" event={"ID":"50d34eef-a2de-4df8-aa17-5c2958a86ba6","Type":"ContainerDied","Data":"316ff8e3f3660ecce4f6d05a198af323b6f67f085202ff03fbc0a663aee0e46d"} Jul 17 02:01:06.606317 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:06.606224 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8" Jul 17 02:01:06.606317 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:06.606231 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="316ff8e3f3660ecce4f6d05a198af323b6f67f085202ff03fbc0a663aee0e46d" Jul 17 02:01:06.737963 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:06.737936 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl"] Jul 17 02:01:06.739943 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:01:06.739910 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27841d8_0ea1_44f1_bfaf_28eeeb3bfe02.slice/crio-b1514fa024451afd14933ddf7e3b0dc867584c0e044ab87fb8db4e0916c7c9db WatchSource:0}: Error finding container b1514fa024451afd14933ddf7e3b0dc867584c0e044ab87fb8db4e0916c7c9db: Status 404 returned error can't find the container with id b1514fa024451afd14933ddf7e3b0dc867584c0e044ab87fb8db4e0916c7c9db Jul 17 02:01:07.612118 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:07.612053 2583 generic.go:356] "Generic (PLEG): container finished" podID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" containerID="7bf4db2e9ce77e541d3f4ab56af02edbecb628603fe49a09226ba2a3d3e27f95" exitCode=6 Jul 17 02:01:07.612289 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:07.612143 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" event={"ID":"a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02","Type":"ContainerDied","Data":"7bf4db2e9ce77e541d3f4ab56af02edbecb628603fe49a09226ba2a3d3e27f95"} Jul 17 02:01:07.612289 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:07.612226 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" event={"ID":"a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02","Type":"ContainerStarted","Data":"b1514fa024451afd14933ddf7e3b0dc867584c0e044ab87fb8db4e0916c7c9db"} Jul 17 02:01:07.612362 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:07.612339 2583 scope.go:117] "RemoveContainer" containerID="7bf4db2e9ce77e541d3f4ab56af02edbecb628603fe49a09226ba2a3d3e27f95" Jul 17 02:01:08.333048 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:08.333006 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf"] Jul 17 02:01:08.336672 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:08.336648 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" Jul 17 02:01:08.342228 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:08.342182 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf"] Jul 17 02:01:08.377859 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:08.377819 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ea8893-8f0f-44d8-9786-aabe7f9c5e16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf\" (UID: \"a3ea8893-8f0f-44d8-9786-aabe7f9c5e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" Jul 17 02:01:08.479200 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:08.479093 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ea8893-8f0f-44d8-9786-aabe7f9c5e16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf\" (UID: \"a3ea8893-8f0f-44d8-9786-aabe7f9c5e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" Jul 17 02:01:08.479713 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:08.479695 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ea8893-8f0f-44d8-9786-aabe7f9c5e16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf\" (UID: \"a3ea8893-8f0f-44d8-9786-aabe7f9c5e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" Jul 17 02:01:08.621513 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:08.621474 2583 generic.go:356] "Generic (PLEG): container finished" podID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" containerID="c4d30f91b35536a19787c042e39baa7ebe34a73cce1dae2cefcdfbb07988bdec" exitCode=6 Jul 17 02:01:08.621659 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:08.621529 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" event={"ID":"a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02","Type":"ContainerDied","Data":"c4d30f91b35536a19787c042e39baa7ebe34a73cce1dae2cefcdfbb07988bdec"} Jul 17 02:01:08.621659 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:08.621591 2583 scope.go:117] "RemoveContainer" containerID="7bf4db2e9ce77e541d3f4ab56af02edbecb628603fe49a09226ba2a3d3e27f95" Jul 17 02:01:08.621816 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:08.621798 2583 scope.go:117] "RemoveContainer" containerID="c4d30f91b35536a19787c042e39baa7ebe34a73cce1dae2cefcdfbb07988bdec" Jul 17 02:01:08.622052 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:08.622031 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl_odh-ai-gateway-infra(a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" podUID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" Jul 17 02:01:08.649436 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:08.649401 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" Jul 17 02:01:08.784446 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:08.784421 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf"] Jul 17 02:01:08.786583 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:01:08.786554 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ea8893_8f0f_44d8_9786_aabe7f9c5e16.slice/crio-ade4ce76ae3b0304088641f69d8af48212049c36a991653415173148fd8b6bf0 WatchSource:0}: Error finding container ade4ce76ae3b0304088641f69d8af48212049c36a991653415173148fd8b6bf0: Status 404 returned error can't find the container with id ade4ce76ae3b0304088641f69d8af48212049c36a991653415173148fd8b6bf0 Jul 17 02:01:09.628615 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:09.628584 2583 scope.go:117] "RemoveContainer" containerID="c4d30f91b35536a19787c042e39baa7ebe34a73cce1dae2cefcdfbb07988bdec" Jul 17 02:01:09.629051 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:09.628857 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl_odh-ai-gateway-infra(a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" podUID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" Jul 17 02:01:09.629951 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:09.629927 2583 generic.go:356] "Generic (PLEG): container finished" podID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" containerID="c010cb52c4c83b75a22291b0953435abc4d6e58f43bff9ac908c3a3bcb42f55e" exitCode=6 Jul 17 02:01:09.630052 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:09.630012 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" event={"ID":"a3ea8893-8f0f-44d8-9786-aabe7f9c5e16","Type":"ContainerDied","Data":"c010cb52c4c83b75a22291b0953435abc4d6e58f43bff9ac908c3a3bcb42f55e"} Jul 17 02:01:09.630115 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:09.630059 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" event={"ID":"a3ea8893-8f0f-44d8-9786-aabe7f9c5e16","Type":"ContainerStarted","Data":"ade4ce76ae3b0304088641f69d8af48212049c36a991653415173148fd8b6bf0"} Jul 17 02:01:09.630245 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:09.630231 2583 scope.go:117] "RemoveContainer" containerID="c010cb52c4c83b75a22291b0953435abc4d6e58f43bff9ac908c3a3bcb42f55e" Jul 17 02:01:10.636784 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:10.636694 2583 generic.go:356] "Generic (PLEG): container finished" podID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" containerID="f4cbdd51b7e3e6e80f23a57e169ad39b373ae1ab4ee70e4ca4ed7c1587ab11cc" exitCode=6 Jul 17 02:01:10.637260 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:10.636784 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" event={"ID":"a3ea8893-8f0f-44d8-9786-aabe7f9c5e16","Type":"ContainerDied","Data":"f4cbdd51b7e3e6e80f23a57e169ad39b373ae1ab4ee70e4ca4ed7c1587ab11cc"} Jul 17 02:01:10.637260 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:10.636849 2583 scope.go:117] "RemoveContainer" containerID="c010cb52c4c83b75a22291b0953435abc4d6e58f43bff9ac908c3a3bcb42f55e" Jul 17 02:01:10.637260 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:10.637036 2583 scope.go:117] "RemoveContainer" containerID="f4cbdd51b7e3e6e80f23a57e169ad39b373ae1ab4ee70e4ca4ed7c1587ab11cc" Jul 17 02:01:10.637431 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:10.637282 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf_odh-ai-gateway-infra(a3ea8893-8f0f-44d8-9786-aabe7f9c5e16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" podUID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" Jul 17 02:01:11.644783 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:11.644752 2583 scope.go:117] "RemoveContainer" containerID="f4cbdd51b7e3e6e80f23a57e169ad39b373ae1ab4ee70e4ca4ed7c1587ab11cc" Jul 17 02:01:11.645282 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:11.645002 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf_odh-ai-gateway-infra(a3ea8893-8f0f-44d8-9786-aabe7f9c5e16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" podUID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" Jul 17 02:01:12.162656 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:12.162628 2583 scope.go:117] "RemoveContainer" containerID="aaa3464d07f20df99567ea27e4f8da5ddb95b07991212b881b1be2957e57fde4" Jul 17 02:01:12.198935 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:12.198913 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:12.299384 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:12.299350 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:12.652012 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:12.651978 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" event={"ID":"eec3bb55-291f-494d-a66c-74a857e2626e","Type":"ContainerStarted","Data":"b930785dd1449ba0cc330ba6961ff82d42e54afff7a45a9f668dc31a6a6951d1"} Jul 17 02:01:13.658789 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:13.658743 2583 generic.go:356] "Generic (PLEG): container finished" podID="eec3bb55-291f-494d-a66c-74a857e2626e" containerID="b930785dd1449ba0cc330ba6961ff82d42e54afff7a45a9f668dc31a6a6951d1" exitCode=6 Jul 17 02:01:13.659231 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:13.658819 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" event={"ID":"eec3bb55-291f-494d-a66c-74a857e2626e","Type":"ContainerDied","Data":"b930785dd1449ba0cc330ba6961ff82d42e54afff7a45a9f668dc31a6a6951d1"} Jul 17 02:01:13.659231 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:13.658879 2583 scope.go:117] "RemoveContainer" containerID="aaa3464d07f20df99567ea27e4f8da5ddb95b07991212b881b1be2957e57fde4" Jul 17 02:01:13.659231 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:13.659165 2583 scope.go:117] "RemoveContainer" containerID="b930785dd1449ba0cc330ba6961ff82d42e54afff7a45a9f668dc31a6a6951d1" Jul 17 02:01:13.659416 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:13.659395 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv_odh-ai-gateway-infra(eec3bb55-291f-494d-a66c-74a857e2626e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" podUID="eec3bb55-291f-494d-a66c-74a857e2626e" Jul 17 02:01:13.682767 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:13.682732 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv"] Jul 17 02:01:13.985333 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:13.985223 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:14.087201 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.087171 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:14.214503 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.214459 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l"] Jul 17 02:01:14.214804 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.214772 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" podUID="17919854-14cf-4b8f-a63d-01e2617a71cb" containerName="maas-api" containerID="cri-o://5f960f88c9a9d6278d6d93e2e932e3e0e239a339ae4c861980dc73b1951eeca4" gracePeriod=30 Jul 17 02:01:14.232024 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.231998 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:14.461356 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.461327 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:01:14.533077 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.533039 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17919854-14cf-4b8f-a63d-01e2617a71cb-maas-api-tls\") pod \"17919854-14cf-4b8f-a63d-01e2617a71cb\" (UID: \"17919854-14cf-4b8f-a63d-01e2617a71cb\") " Jul 17 02:01:14.533257 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.533136 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nffwk\" (UniqueName: \"kubernetes.io/projected/17919854-14cf-4b8f-a63d-01e2617a71cb-kube-api-access-nffwk\") pod \"17919854-14cf-4b8f-a63d-01e2617a71cb\" (UID: \"17919854-14cf-4b8f-a63d-01e2617a71cb\") " Jul 17 02:01:14.533257 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.533173 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17919854-14cf-4b8f-a63d-01e2617a71cb-openshift-service-ca\") pod \"17919854-14cf-4b8f-a63d-01e2617a71cb\" (UID: \"17919854-14cf-4b8f-a63d-01e2617a71cb\") " Jul 17 02:01:14.533578 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.533556 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17919854-14cf-4b8f-a63d-01e2617a71cb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17919854-14cf-4b8f-a63d-01e2617a71cb" (UID: "17919854-14cf-4b8f-a63d-01e2617a71cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:01:14.535422 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.535395 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17919854-14cf-4b8f-a63d-01e2617a71cb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17919854-14cf-4b8f-a63d-01e2617a71cb" (UID: "17919854-14cf-4b8f-a63d-01e2617a71cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 02:01:14.535477 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.535406 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17919854-14cf-4b8f-a63d-01e2617a71cb-kube-api-access-nffwk" (OuterVolumeSpecName: "kube-api-access-nffwk") pod "17919854-14cf-4b8f-a63d-01e2617a71cb" (UID: "17919854-14cf-4b8f-a63d-01e2617a71cb"). InnerVolumeSpecName "kube-api-access-nffwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 02:01:14.634010 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.633929 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17919854-14cf-4b8f-a63d-01e2617a71cb-openshift-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:01:14.634010 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.633960 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17919854-14cf-4b8f-a63d-01e2617a71cb-maas-api-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:01:14.634010 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.633969 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nffwk\" (UniqueName: \"kubernetes.io/projected/17919854-14cf-4b8f-a63d-01e2617a71cb-kube-api-access-nffwk\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:01:14.666536 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.666505 2583 generic.go:356] "Generic (PLEG): container finished" podID="17919854-14cf-4b8f-a63d-01e2617a71cb" containerID="5f960f88c9a9d6278d6d93e2e932e3e0e239a339ae4c861980dc73b1951eeca4" exitCode=0 Jul 17 02:01:14.666950 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.666556 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" Jul 17 02:01:14.666950 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.666547 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" event={"ID":"17919854-14cf-4b8f-a63d-01e2617a71cb","Type":"ContainerDied","Data":"5f960f88c9a9d6278d6d93e2e932e3e0e239a339ae4c861980dc73b1951eeca4"} Jul 17 02:01:14.666950 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.666694 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l" event={"ID":"17919854-14cf-4b8f-a63d-01e2617a71cb","Type":"ContainerDied","Data":"66aa1f1d03db6ea708f9d09d5dab374dac41f88fb7ea7c8401bc1c61b54a4e83"} Jul 17 02:01:14.666950 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.666732 2583 scope.go:117] "RemoveContainer" containerID="5f960f88c9a9d6278d6d93e2e932e3e0e239a339ae4c861980dc73b1951eeca4" Jul 17 02:01:14.675943 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.675921 2583 scope.go:117] "RemoveContainer" containerID="5f960f88c9a9d6278d6d93e2e932e3e0e239a339ae4c861980dc73b1951eeca4" Jul 17 02:01:14.676329 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:14.676304 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f960f88c9a9d6278d6d93e2e932e3e0e239a339ae4c861980dc73b1951eeca4\": container with ID starting with 5f960f88c9a9d6278d6d93e2e932e3e0e239a339ae4c861980dc73b1951eeca4 not found: ID does not exist" containerID="5f960f88c9a9d6278d6d93e2e932e3e0e239a339ae4c861980dc73b1951eeca4" Jul 17 02:01:14.676418 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.676340 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f960f88c9a9d6278d6d93e2e932e3e0e239a339ae4c861980dc73b1951eeca4"} err="failed to get container status \"5f960f88c9a9d6278d6d93e2e932e3e0e239a339ae4c861980dc73b1951eeca4\": rpc error: code = NotFound desc = could not find container \"5f960f88c9a9d6278d6d93e2e932e3e0e239a339ae4c861980dc73b1951eeca4\": container with ID starting with 5f960f88c9a9d6278d6d93e2e932e3e0e239a339ae4c861980dc73b1951eeca4 not found: ID does not exist" Jul 17 02:01:14.691847 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.691804 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l"] Jul 17 02:01:14.695874 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.695842 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3413e7-668c85dc67-lvd7l"] Jul 17 02:01:14.801283 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.801258 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" Jul 17 02:01:14.835247 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.835215 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec3bb55-291f-494d-a66c-74a857e2626e-maas-api-service-ca\") pod \"eec3bb55-291f-494d-a66c-74a857e2626e\" (UID: \"eec3bb55-291f-494d-a66c-74a857e2626e\") " Jul 17 02:01:14.835562 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.835537 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec3bb55-291f-494d-a66c-74a857e2626e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eec3bb55-291f-494d-a66c-74a857e2626e" (UID: "eec3bb55-291f-494d-a66c-74a857e2626e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:01:14.936753 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:14.936661 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec3bb55-291f-494d-a66c-74a857e2626e-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:01:15.173546 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:15.173509 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17919854-14cf-4b8f-a63d-01e2617a71cb" path="/var/lib/kubelet/pods/17919854-14cf-4b8f-a63d-01e2617a71cb/volumes" Jul 17 02:01:15.672684 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:15.672648 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" Jul 17 02:01:15.673189 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:15.672649 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv" event={"ID":"eec3bb55-291f-494d-a66c-74a857e2626e","Type":"ContainerDied","Data":"bce35bd2421d0397db4693f861acfe451640196ab1fb872aeebbdaa3eb04ded4"} Jul 17 02:01:15.673189 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:15.672784 2583 scope.go:117] "RemoveContainer" containerID="b930785dd1449ba0cc330ba6961ff82d42e54afff7a45a9f668dc31a6a6951d1" Jul 17 02:01:15.688552 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:15.688519 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv"] Jul 17 02:01:15.698113 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:15.698070 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2k7sv"] Jul 17 02:01:17.168125 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:17.168065 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eec3bb55-291f-494d-a66c-74a857e2626e" path="/var/lib/kubelet/pods/eec3bb55-291f-494d-a66c-74a857e2626e/volumes" Jul 17 02:01:18.970421 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:18.970388 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:20.276480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.273309 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk"] Jul 17 02:01:20.276480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.274271 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17919854-14cf-4b8f-a63d-01e2617a71cb" containerName="maas-api" Jul 17 02:01:20.276480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.274295 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="17919854-14cf-4b8f-a63d-01e2617a71cb" containerName="maas-api" Jul 17 02:01:20.276480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.274321 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec3bb55-291f-494d-a66c-74a857e2626e" containerName="revoke-keys" Jul 17 02:01:20.276480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.274330 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec3bb55-291f-494d-a66c-74a857e2626e" containerName="revoke-keys" Jul 17 02:01:20.276480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.274363 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec3bb55-291f-494d-a66c-74a857e2626e" containerName="revoke-keys" Jul 17 02:01:20.276480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.274373 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec3bb55-291f-494d-a66c-74a857e2626e" containerName="revoke-keys" Jul 17 02:01:20.276480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.274383 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec3bb55-291f-494d-a66c-74a857e2626e" containerName="revoke-keys" Jul 17 02:01:20.276480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.274392 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec3bb55-291f-494d-a66c-74a857e2626e" containerName="revoke-keys" Jul 17 02:01:20.276480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.274600 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec3bb55-291f-494d-a66c-74a857e2626e" containerName="revoke-keys" Jul 17 02:01:20.276480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.274619 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec3bb55-291f-494d-a66c-74a857e2626e" containerName="revoke-keys" Jul 17 02:01:20.276480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.274630 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec3bb55-291f-494d-a66c-74a857e2626e" containerName="revoke-keys" Jul 17 02:01:20.276480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.274660 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="17919854-14cf-4b8f-a63d-01e2617a71cb" containerName="maas-api" Jul 17 02:01:20.279150 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.279130 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" Jul 17 02:01:20.282850 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.282822 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk"] Jul 17 02:01:20.387407 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.387361 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa023726-a5f4-4153-9817-e18334b764d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk\" (UID: \"aa023726-a5f4-4153-9817-e18334b764d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" Jul 17 02:01:20.487926 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.487881 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa023726-a5f4-4153-9817-e18334b764d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk\" (UID: \"aa023726-a5f4-4153-9817-e18334b764d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" Jul 17 02:01:20.488528 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.488509 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa023726-a5f4-4153-9817-e18334b764d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk\" (UID: \"aa023726-a5f4-4153-9817-e18334b764d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" Jul 17 02:01:20.591034 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.591002 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" Jul 17 02:01:20.729223 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:20.729194 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk"] Jul 17 02:01:20.731904 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:01:20.731877 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa023726_a5f4_4153_9817_e18334b764d5.slice/crio-5f2741defde21429f4cd8ab649361b95599009fe759c35852c816d01bf850bc3 WatchSource:0}: Error finding container 5f2741defde21429f4cd8ab649361b95599009fe759c35852c816d01bf850bc3: Status 404 returned error can't find the container with id 5f2741defde21429f4cd8ab649361b95599009fe759c35852c816d01bf850bc3 Jul 17 02:01:21.707460 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:21.707350 2583 generic.go:356] "Generic (PLEG): container finished" podID="aa023726-a5f4-4153-9817-e18334b764d5" containerID="a537b513e7799811ce4ce674bca7607ef262af44d9651f4d5d47703a9de7e403" exitCode=6 Jul 17 02:01:21.707460 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:21.707445 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" event={"ID":"aa023726-a5f4-4153-9817-e18334b764d5","Type":"ContainerDied","Data":"a537b513e7799811ce4ce674bca7607ef262af44d9651f4d5d47703a9de7e403"} Jul 17 02:01:21.707995 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:21.707499 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" event={"ID":"aa023726-a5f4-4153-9817-e18334b764d5","Type":"ContainerStarted","Data":"5f2741defde21429f4cd8ab649361b95599009fe759c35852c816d01bf850bc3"} Jul 17 02:01:21.707995 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:21.707696 2583 scope.go:117] "RemoveContainer" containerID="a537b513e7799811ce4ce674bca7607ef262af44d9651f4d5d47703a9de7e403" Jul 17 02:01:22.716325 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:22.716243 2583 generic.go:356] "Generic (PLEG): container finished" podID="aa023726-a5f4-4153-9817-e18334b764d5" containerID="303a0c726d44a395a4619b11a98db1eddfc09995251ebe27f7977161b76d5ea7" exitCode=6 Jul 17 02:01:22.716325 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:22.716295 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" event={"ID":"aa023726-a5f4-4153-9817-e18334b764d5","Type":"ContainerDied","Data":"303a0c726d44a395a4619b11a98db1eddfc09995251ebe27f7977161b76d5ea7"} Jul 17 02:01:22.716845 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:22.716342 2583 scope.go:117] "RemoveContainer" containerID="a537b513e7799811ce4ce674bca7607ef262af44d9651f4d5d47703a9de7e403" Jul 17 02:01:22.716845 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:22.716513 2583 scope.go:117] "RemoveContainer" containerID="303a0c726d44a395a4619b11a98db1eddfc09995251ebe27f7977161b76d5ea7" Jul 17 02:01:22.716845 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:22.716719 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk_odh-ai-gateway-infra(aa023726-a5f4-4153-9817-e18334b764d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" podUID="aa023726-a5f4-4153-9817-e18334b764d5" Jul 17 02:01:23.164534 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:23.164499 2583 scope.go:117] "RemoveContainer" containerID="c4d30f91b35536a19787c042e39baa7ebe34a73cce1dae2cefcdfbb07988bdec" Jul 17 02:01:23.723927 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:23.723900 2583 scope.go:117] "RemoveContainer" containerID="303a0c726d44a395a4619b11a98db1eddfc09995251ebe27f7977161b76d5ea7" Jul 17 02:01:23.724388 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:23.724181 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk_odh-ai-gateway-infra(aa023726-a5f4-4153-9817-e18334b764d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" podUID="aa023726-a5f4-4153-9817-e18334b764d5" Jul 17 02:01:23.725731 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:23.725703 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" event={"ID":"a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02","Type":"ContainerStarted","Data":"4ab6150582ae6c4f90238602de3feda66dd36944bb0648a657b07da4f63d6f13"} Jul 17 02:01:23.752478 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:23.752413 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" podStartSLOduration=17.75239219 podStartE2EDuration="17.75239219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:01:23.751736971 +0000 UTC m=+2841.147351834" watchObservedRunningTime="2026-07-17 02:01:23.75239219 +0000 UTC m=+2841.148007041" Jul 17 02:01:24.092467 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:24.092436 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:24.192810 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:24.192777 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:24.732994 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:24.732955 2583 generic.go:356] "Generic (PLEG): container finished" podID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" containerID="4ab6150582ae6c4f90238602de3feda66dd36944bb0648a657b07da4f63d6f13" exitCode=6 Jul 17 02:01:24.733474 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:24.733027 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" event={"ID":"a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02","Type":"ContainerDied","Data":"4ab6150582ae6c4f90238602de3feda66dd36944bb0648a657b07da4f63d6f13"} Jul 17 02:01:24.733474 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:24.733088 2583 scope.go:117] "RemoveContainer" containerID="c4d30f91b35536a19787c042e39baa7ebe34a73cce1dae2cefcdfbb07988bdec" Jul 17 02:01:24.733474 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:24.733422 2583 scope.go:117] "RemoveContainer" containerID="4ab6150582ae6c4f90238602de3feda66dd36944bb0648a657b07da4f63d6f13" Jul 17 02:01:24.733688 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:24.733665 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl_odh-ai-gateway-infra(a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" podUID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" Jul 17 02:01:24.760885 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:24.760850 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl"] Jul 17 02:01:25.891376 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:25.891347 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" Jul 17 02:01:25.933152 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:25.933110 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02-maas-api-service-ca\") pod \"a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02\" (UID: \"a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02\") " Jul 17 02:01:25.933517 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:25.933491 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" (UID: "a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:01:26.033916 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:26.033893 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:01:26.163238 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:26.163205 2583 scope.go:117] "RemoveContainer" containerID="f4cbdd51b7e3e6e80f23a57e169ad39b373ae1ab4ee70e4ca4ed7c1587ab11cc" Jul 17 02:01:26.746586 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:26.746493 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" event={"ID":"a3ea8893-8f0f-44d8-9786-aabe7f9c5e16","Type":"ContainerStarted","Data":"c678c67a9739ddeb0db08be139ee5a68aca1ae646d4281557c3a6f4c79235b7a"} Jul 17 02:01:26.747850 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:26.747824 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" event={"ID":"a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02","Type":"ContainerDied","Data":"b1514fa024451afd14933ddf7e3b0dc867584c0e044ab87fb8db4e0916c7c9db"} Jul 17 02:01:26.747987 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:26.747874 2583 scope.go:117] "RemoveContainer" containerID="4ab6150582ae6c4f90238602de3feda66dd36944bb0648a657b07da4f63d6f13" Jul 17 02:01:26.747987 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:26.747836 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl" Jul 17 02:01:26.763922 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:26.763875 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" podStartSLOduration=18.763858484 podStartE2EDuration="18.763858484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:01:26.762254622 +0000 UTC m=+2844.157869476" watchObservedRunningTime="2026-07-17 02:01:26.763858484 +0000 UTC m=+2844.159473384" Jul 17 02:01:26.774515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:26.774483 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl"] Jul 17 02:01:26.779976 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:26.779951 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-pzjtl"] Jul 17 02:01:27.168489 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:27.168453 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" path="/var/lib/kubelet/pods/a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02/volumes" Jul 17 02:01:27.753919 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:27.753887 2583 generic.go:356] "Generic (PLEG): container finished" podID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" containerID="c678c67a9739ddeb0db08be139ee5a68aca1ae646d4281557c3a6f4c79235b7a" exitCode=6 Jul 17 02:01:27.754081 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:27.753954 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" event={"ID":"a3ea8893-8f0f-44d8-9786-aabe7f9c5e16","Type":"ContainerDied","Data":"c678c67a9739ddeb0db08be139ee5a68aca1ae646d4281557c3a6f4c79235b7a"} Jul 17 02:01:27.754081 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:27.754014 2583 scope.go:117] "RemoveContainer" containerID="f4cbdd51b7e3e6e80f23a57e169ad39b373ae1ab4ee70e4ca4ed7c1587ab11cc" Jul 17 02:01:27.754304 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:27.754288 2583 scope.go:117] "RemoveContainer" containerID="c678c67a9739ddeb0db08be139ee5a68aca1ae646d4281557c3a6f4c79235b7a" Jul 17 02:01:27.754535 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:27.754512 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf_odh-ai-gateway-infra(a3ea8893-8f0f-44d8-9786-aabe7f9c5e16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" podUID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" Jul 17 02:01:27.777523 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:27.777497 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf"] Jul 17 02:01:28.747868 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:28.747832 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:28.900187 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:28.900157 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" Jul 17 02:01:28.958671 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:28.958633 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ea8893-8f0f-44d8-9786-aabe7f9c5e16-maas-api-service-ca\") pod \"a3ea8893-8f0f-44d8-9786-aabe7f9c5e16\" (UID: \"a3ea8893-8f0f-44d8-9786-aabe7f9c5e16\") " Jul 17 02:01:28.959067 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:28.959044 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ea8893-8f0f-44d8-9786-aabe7f9c5e16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" (UID: "a3ea8893-8f0f-44d8-9786-aabe7f9c5e16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:01:28.977206 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:28.977161 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:29.059657 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.059623 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ea8893-8f0f-44d8-9786-aabe7f9c5e16-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:01:29.213383 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.213353 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:29.332710 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.332636 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg"] Jul 17 02:01:29.332976 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.332964 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" containerName="revoke-keys" Jul 17 02:01:29.333017 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.332978 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" containerName="revoke-keys" Jul 17 02:01:29.333017 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.332986 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" containerName="revoke-keys" Jul 17 02:01:29.333017 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.332991 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" containerName="revoke-keys" Jul 17 02:01:29.333017 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.333007 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" containerName="revoke-keys" Jul 17 02:01:29.333017 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.333013 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" containerName="revoke-keys" Jul 17 02:01:29.333200 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.333022 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" containerName="revoke-keys" Jul 17 02:01:29.333200 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.333027 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" containerName="revoke-keys" Jul 17 02:01:29.333200 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.333072 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" containerName="revoke-keys" Jul 17 02:01:29.333200 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.333080 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" containerName="revoke-keys" Jul 17 02:01:29.333200 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.333088 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" containerName="revoke-keys" Jul 17 02:01:29.333200 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.333107 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" containerName="revoke-keys" Jul 17 02:01:29.335321 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.335303 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg" Jul 17 02:01:29.343604 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.343563 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg"] Jul 17 02:01:29.462760 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.462713 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg\" (UID: \"9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg" Jul 17 02:01:29.563715 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.563675 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg\" (UID: \"9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg" Jul 17 02:01:29.564378 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.564351 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg\" (UID: \"9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg" Jul 17 02:01:29.646836 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.646712 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg" Jul 17 02:01:29.776294 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.776264 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" event={"ID":"a3ea8893-8f0f-44d8-9786-aabe7f9c5e16","Type":"ContainerDied","Data":"ade4ce76ae3b0304088641f69d8af48212049c36a991653415173148fd8b6bf0"} Jul 17 02:01:29.776768 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.776330 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf" Jul 17 02:01:29.776768 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.776352 2583 scope.go:117] "RemoveContainer" containerID="c678c67a9739ddeb0db08be139ee5a68aca1ae646d4281557c3a6f4c79235b7a" Jul 17 02:01:29.790900 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.790872 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg"] Jul 17 02:01:29.792004 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:01:29.791974 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad84ef6_4a2b_45ae_8d5f_9c1c3d062dfa.slice/crio-a9064b9aae6dd40f5fafab261e86f58c78a955bd29963b9eb111f36b8d8c63f6 WatchSource:0}: Error finding container a9064b9aae6dd40f5fafab261e86f58c78a955bd29963b9eb111f36b8d8c63f6: Status 404 returned error can't find the container with id a9064b9aae6dd40f5fafab261e86f58c78a955bd29963b9eb111f36b8d8c63f6 Jul 17 02:01:29.816476 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.816450 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf"] Jul 17 02:01:29.825492 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.825467 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zjssf"] Jul 17 02:01:29.880269 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.880233 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:29.982966 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:29.982884 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:30.782316 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:30.782286 2583 generic.go:356] "Generic (PLEG): container finished" podID="9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa" containerID="bd3528e0286783f932dfb7aaa19b3c3acfdbc7a769a4fe4da366d6b56ebcc4dc" exitCode=0 Jul 17 02:01:30.782710 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:30.782376 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg" event={"ID":"9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa","Type":"ContainerDied","Data":"bd3528e0286783f932dfb7aaa19b3c3acfdbc7a769a4fe4da366d6b56ebcc4dc"} Jul 17 02:01:30.782710 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:30.782430 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg" event={"ID":"9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa","Type":"ContainerStarted","Data":"a9064b9aae6dd40f5fafab261e86f58c78a955bd29963b9eb111f36b8d8c63f6"} Jul 17 02:01:31.168074 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:31.167999 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" path="/var/lib/kubelet/pods/a3ea8893-8f0f-44d8-9786-aabe7f9c5e16/volumes" Jul 17 02:01:31.925048 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:31.925017 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg" Jul 17 02:01:31.986743 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:31.986708 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa-maas-api-service-ca\") pod \"9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa\" (UID: \"9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa\") " Jul 17 02:01:31.987090 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:31.987065 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa" (UID: "9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:01:32.088226 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:32.088131 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:01:32.794755 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:32.794713 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg" event={"ID":"9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa","Type":"ContainerDied","Data":"a9064b9aae6dd40f5fafab261e86f58c78a955bd29963b9eb111f36b8d8c63f6"} Jul 17 02:01:32.794755 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:32.794736 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg" Jul 17 02:01:32.794967 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:32.794769 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9064b9aae6dd40f5fafab261e86f58c78a955bd29963b9eb111f36b8d8c63f6" Jul 17 02:01:35.162514 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:35.162478 2583 scope.go:117] "RemoveContainer" containerID="303a0c726d44a395a4619b11a98db1eddfc09995251ebe27f7977161b76d5ea7" Jul 17 02:01:35.817849 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:35.817813 2583 generic.go:356] "Generic (PLEG): container finished" podID="aa023726-a5f4-4153-9817-e18334b764d5" containerID="95a1d18599dc9244a3b32e8d57b20f4d51fd88c679eb2d9d23d940523fd4651a" exitCode=6 Jul 17 02:01:35.818048 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:35.817896 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" event={"ID":"aa023726-a5f4-4153-9817-e18334b764d5","Type":"ContainerDied","Data":"95a1d18599dc9244a3b32e8d57b20f4d51fd88c679eb2d9d23d940523fd4651a"} Jul 17 02:01:35.818048 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:35.817997 2583 scope.go:117] "RemoveContainer" containerID="303a0c726d44a395a4619b11a98db1eddfc09995251ebe27f7977161b76d5ea7" Jul 17 02:01:35.818361 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:35.818335 2583 scope.go:117] "RemoveContainer" containerID="95a1d18599dc9244a3b32e8d57b20f4d51fd88c679eb2d9d23d940523fd4651a" Jul 17 02:01:35.818570 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:35.818548 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk_odh-ai-gateway-infra(aa023726-a5f4-4153-9817-e18334b764d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" podUID="aa023726-a5f4-4153-9817-e18334b764d5" Jul 17 02:01:36.841657 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:36.841620 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk"] Jul 17 02:01:36.977765 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:36.977740 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" Jul 17 02:01:37.130227 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:37.130130 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa023726-a5f4-4153-9817-e18334b764d5-maas-api-service-ca\") pod \"aa023726-a5f4-4153-9817-e18334b764d5\" (UID: \"aa023726-a5f4-4153-9817-e18334b764d5\") " Jul 17 02:01:37.130498 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:37.130466 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa023726-a5f4-4153-9817-e18334b764d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa023726-a5f4-4153-9817-e18334b764d5" (UID: "aa023726-a5f4-4153-9817-e18334b764d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:01:37.231067 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:37.231033 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa023726-a5f4-4153-9817-e18334b764d5-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:01:37.832557 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:37.832526 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" Jul 17 02:01:37.832557 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:37.832537 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk" event={"ID":"aa023726-a5f4-4153-9817-e18334b764d5","Type":"ContainerDied","Data":"5f2741defde21429f4cd8ab649361b95599009fe759c35852c816d01bf850bc3"} Jul 17 02:01:37.832798 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:37.832586 2583 scope.go:117] "RemoveContainer" containerID="95a1d18599dc9244a3b32e8d57b20f4d51fd88c679eb2d9d23d940523fd4651a" Jul 17 02:01:37.849285 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:37.849254 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk"] Jul 17 02:01:37.857176 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:37.857132 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-ffkhk"] Jul 17 02:01:39.168020 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:39.167978 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa023726-a5f4-4153-9817-e18334b764d5" path="/var/lib/kubelet/pods/aa023726-a5f4-4153-9817-e18334b764d5/volumes" Jul 17 02:01:39.715354 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:39.715321 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf"] Jul 17 02:01:39.715670 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:39.715618 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" podUID="66324c41-643e-4f3a-bdce-bcfb127afd86" containerName="maas-api" containerID="cri-o://bbf606f5e1b3fafce03c76db619aa8fbbdbde372f40991b8e0d9b44c5397e7de" gracePeriod=30 Jul 17 02:01:39.737371 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:39.737343 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:39.846842 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:39.846805 2583 generic.go:356] "Generic (PLEG): container finished" podID="66324c41-643e-4f3a-bdce-bcfb127afd86" containerID="bbf606f5e1b3fafce03c76db619aa8fbbdbde372f40991b8e0d9b44c5397e7de" exitCode=0 Jul 17 02:01:39.847034 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:39.846878 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" event={"ID":"66324c41-643e-4f3a-bdce-bcfb127afd86","Type":"ContainerDied","Data":"bbf606f5e1b3fafce03c76db619aa8fbbdbde372f40991b8e0d9b44c5397e7de"} Jul 17 02:01:39.976194 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:39.976131 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:01:40.056461 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:40.056423 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66324c41-643e-4f3a-bdce-bcfb127afd86-openshift-service-ca\") pod \"66324c41-643e-4f3a-bdce-bcfb127afd86\" (UID: \"66324c41-643e-4f3a-bdce-bcfb127afd86\") " Jul 17 02:01:40.056633 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:40.056470 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wll5f\" (UniqueName: \"kubernetes.io/projected/66324c41-643e-4f3a-bdce-bcfb127afd86-kube-api-access-wll5f\") pod \"66324c41-643e-4f3a-bdce-bcfb127afd86\" (UID: \"66324c41-643e-4f3a-bdce-bcfb127afd86\") " Jul 17 02:01:40.056633 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:40.056535 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66324c41-643e-4f3a-bdce-bcfb127afd86-maas-api-tls\") pod \"66324c41-643e-4f3a-bdce-bcfb127afd86\" (UID: \"66324c41-643e-4f3a-bdce-bcfb127afd86\") " Jul 17 02:01:40.056862 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:40.056842 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66324c41-643e-4f3a-bdce-bcfb127afd86-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66324c41-643e-4f3a-bdce-bcfb127afd86" (UID: "66324c41-643e-4f3a-bdce-bcfb127afd86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:01:40.058827 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:40.058803 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66324c41-643e-4f3a-bdce-bcfb127afd86-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66324c41-643e-4f3a-bdce-bcfb127afd86" (UID: "66324c41-643e-4f3a-bdce-bcfb127afd86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 02:01:40.058926 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:40.058824 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66324c41-643e-4f3a-bdce-bcfb127afd86-kube-api-access-wll5f" (OuterVolumeSpecName: "kube-api-access-wll5f") pod "66324c41-643e-4f3a-bdce-bcfb127afd86" (UID: "66324c41-643e-4f3a-bdce-bcfb127afd86"). InnerVolumeSpecName "kube-api-access-wll5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 02:01:40.157824 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:40.157792 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66324c41-643e-4f3a-bdce-bcfb127afd86-openshift-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:01:40.157824 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:40.157818 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wll5f\" (UniqueName: \"kubernetes.io/projected/66324c41-643e-4f3a-bdce-bcfb127afd86-kube-api-access-wll5f\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:01:40.157824 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:40.157828 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66324c41-643e-4f3a-bdce-bcfb127afd86-maas-api-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:01:40.852801 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:40.852769 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" event={"ID":"66324c41-643e-4f3a-bdce-bcfb127afd86","Type":"ContainerDied","Data":"a2c851b382a6005b4c7c735a7802ab7560d44bee856929b16ba8887d3809a01f"} Jul 17 02:01:40.852801 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:40.852799 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf" Jul 17 02:01:40.853283 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:40.852832 2583 scope.go:117] "RemoveContainer" containerID="bbf606f5e1b3fafce03c76db619aa8fbbdbde372f40991b8e0d9b44c5397e7de" Jul 17 02:01:40.873691 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:40.873645 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf"] Jul 17 02:01:40.877240 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:40.877206 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d8cc4a-664dc75dcd-zxhdf"] Jul 17 02:01:41.168245 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:41.168159 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66324c41-643e-4f3a-bdce-bcfb127afd86" path="/var/lib/kubelet/pods/66324c41-643e-4f3a-bdce-bcfb127afd86/volumes" Jul 17 02:01:44.427892 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:44.427840 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:50.020311 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:50.020276 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:50.087645 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:50.087611 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:53.733568 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733490 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s"] Jul 17 02:01:53.733918 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733851 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa023726-a5f4-4153-9817-e18334b764d5" containerName="revoke-keys" Jul 17 02:01:53.733918 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733865 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa023726-a5f4-4153-9817-e18334b764d5" containerName="revoke-keys" Jul 17 02:01:53.733918 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733877 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa023726-a5f4-4153-9817-e18334b764d5" containerName="revoke-keys" Jul 17 02:01:53.733918 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733882 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa023726-a5f4-4153-9817-e18334b764d5" containerName="revoke-keys" Jul 17 02:01:53.733918 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733893 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66324c41-643e-4f3a-bdce-bcfb127afd86" containerName="maas-api" Jul 17 02:01:53.733918 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733899 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="66324c41-643e-4f3a-bdce-bcfb127afd86" containerName="maas-api" Jul 17 02:01:53.733918 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733905 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" containerName="revoke-keys" Jul 17 02:01:53.733918 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733909 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" containerName="revoke-keys" Jul 17 02:01:53.733918 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733918 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" containerName="revoke-keys" Jul 17 02:01:53.733918 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733923 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" containerName="revoke-keys" Jul 17 02:01:53.734270 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733930 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa" containerName="revoke-keys" Jul 17 02:01:53.734270 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733935 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa" containerName="revoke-keys" Jul 17 02:01:53.734270 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733993 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27841d8-0ea1-44f1-bfaf-28eeeb3bfe02" containerName="revoke-keys" Jul 17 02:01:53.734270 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.733999 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="66324c41-643e-4f3a-bdce-bcfb127afd86" containerName="maas-api" Jul 17 02:01:53.734270 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.734007 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa" containerName="revoke-keys" Jul 17 02:01:53.734270 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.734012 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa023726-a5f4-4153-9817-e18334b764d5" containerName="revoke-keys" Jul 17 02:01:53.734270 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.734019 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3ea8893-8f0f-44d8-9786-aabe7f9c5e16" containerName="revoke-keys" Jul 17 02:01:53.734270 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.734028 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa023726-a5f4-4153-9817-e18334b764d5" containerName="revoke-keys" Jul 17 02:01:53.737549 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.737531 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" Jul 17 02:01:53.739790 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.739774 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 02:01:53.739874 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.739825 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 02:01:53.744116 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.744065 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s"] Jul 17 02:01:53.878857 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.878816 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0236a35f-9a7b-4945-abc9-efc1cf41f8c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s\" (UID: \"0236a35f-9a7b-4945-abc9-efc1cf41f8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" Jul 17 02:01:53.980392 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.980344 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0236a35f-9a7b-4945-abc9-efc1cf41f8c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s\" (UID: \"0236a35f-9a7b-4945-abc9-efc1cf41f8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" Jul 17 02:01:53.980971 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:53.980953 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0236a35f-9a7b-4945-abc9-efc1cf41f8c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s\" (UID: \"0236a35f-9a7b-4945-abc9-efc1cf41f8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" Jul 17 02:01:54.049533 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:54.049502 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" Jul 17 02:01:54.184792 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:54.184756 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s"] Jul 17 02:01:54.187874 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:01:54.187835 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0236a35f_9a7b_4945_abc9_efc1cf41f8c6.slice/crio-25c98b001410e01d5f33eb8ee7da0a6b26c2cd58131b08dc5dc0185686d97b91 WatchSource:0}: Error finding container 25c98b001410e01d5f33eb8ee7da0a6b26c2cd58131b08dc5dc0185686d97b91: Status 404 returned error can't find the container with id 25c98b001410e01d5f33eb8ee7da0a6b26c2cd58131b08dc5dc0185686d97b91 Jul 17 02:01:54.924068 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:54.924031 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" event={"ID":"0236a35f-9a7b-4945-abc9-efc1cf41f8c6","Type":"ContainerStarted","Data":"17cc977e989ea27c0989e173fea3ad1a6ed7e0cde6029b3c5d3bbeaa47cc5867"} Jul 17 02:01:54.924421 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:54.924130 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" event={"ID":"0236a35f-9a7b-4945-abc9-efc1cf41f8c6","Type":"ContainerStarted","Data":"25c98b001410e01d5f33eb8ee7da0a6b26c2cd58131b08dc5dc0185686d97b91"} Jul 17 02:01:54.938497 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:54.938420 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" podStartSLOduration=1.938402087 podStartE2EDuration="1.938402087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:01:54.937512974 +0000 UTC m=+2872.333127829" watchObservedRunningTime="2026-07-17 02:01:54.938402087 +0000 UTC m=+2872.334016936" Jul 17 02:01:55.098379 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:55.098350 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:55.328634 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:55.328605 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:55.676479 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:55.676386 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:55.931700 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:55.931612 2583 generic.go:356] "Generic (PLEG): container finished" podID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" containerID="17cc977e989ea27c0989e173fea3ad1a6ed7e0cde6029b3c5d3bbeaa47cc5867" exitCode=6 Jul 17 02:01:55.931700 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:55.931651 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" event={"ID":"0236a35f-9a7b-4945-abc9-efc1cf41f8c6","Type":"ContainerDied","Data":"17cc977e989ea27c0989e173fea3ad1a6ed7e0cde6029b3c5d3bbeaa47cc5867"} Jul 17 02:01:55.932130 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:55.931886 2583 scope.go:117] "RemoveContainer" containerID="17cc977e989ea27c0989e173fea3ad1a6ed7e0cde6029b3c5d3bbeaa47cc5867" Jul 17 02:01:55.997741 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:55.997703 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:56.219367 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:56.219250 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:56.939257 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:56.939223 2583 generic.go:356] "Generic (PLEG): container finished" podID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" containerID="fbb71e3161db335d174287cf0656a38b5e7118b2894df72d1decf11e57223c11" exitCode=6 Jul 17 02:01:56.939683 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:56.939305 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" event={"ID":"0236a35f-9a7b-4945-abc9-efc1cf41f8c6","Type":"ContainerDied","Data":"fbb71e3161db335d174287cf0656a38b5e7118b2894df72d1decf11e57223c11"} Jul 17 02:01:56.939683 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:56.939365 2583 scope.go:117] "RemoveContainer" containerID="17cc977e989ea27c0989e173fea3ad1a6ed7e0cde6029b3c5d3bbeaa47cc5867" Jul 17 02:01:56.939683 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:56.939574 2583 scope.go:117] "RemoveContainer" containerID="fbb71e3161db335d174287cf0656a38b5e7118b2894df72d1decf11e57223c11" Jul 17 02:01:56.939812 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:56.939783 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s_odh-ai-gateway-infra(0236a35f-9a7b-4945-abc9-efc1cf41f8c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" podUID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" Jul 17 02:01:57.946605 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:57.946581 2583 scope.go:117] "RemoveContainer" containerID="fbb71e3161db335d174287cf0656a38b5e7118b2894df72d1decf11e57223c11" Jul 17 02:01:57.947011 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:57.946769 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s_odh-ai-gateway-infra(0236a35f-9a7b-4945-abc9-efc1cf41f8c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" podUID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" Jul 17 02:01:58.883606 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:58.883571 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:58.982947 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:58.982914 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:59.048327 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.048296 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k"] Jul 17 02:01:59.048668 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.048655 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa023726-a5f4-4153-9817-e18334b764d5" containerName="revoke-keys" Jul 17 02:01:59.048709 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.048669 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa023726-a5f4-4153-9817-e18334b764d5" containerName="revoke-keys" Jul 17 02:01:59.048748 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.048723 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa023726-a5f4-4153-9817-e18334b764d5" containerName="revoke-keys" Jul 17 02:01:59.052226 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.052200 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:01:59.055244 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.055219 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-fd4ae7-serving-cert\"" Jul 17 02:01:59.055413 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.055280 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 02:01:59.055413 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.055314 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5l8p8\"" Jul 17 02:01:59.063638 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.063612 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k"] Jul 17 02:01:59.083638 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.083595 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:59.118892 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.118853 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25794c4-a21c-4449-9feb-4c3682e818d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k\" (UID: \"b25794c4-a21c-4449-9feb-4c3682e818d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:01:59.119093 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.118921 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgnvk\" (UniqueName: \"kubernetes.io/projected/b25794c4-a21c-4449-9feb-4c3682e818d0-kube-api-access-jgnvk\") pod \"maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k\" (UID: \"b25794c4-a21c-4449-9feb-4c3682e818d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:01:59.119093 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.118950 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25794c4-a21c-4449-9feb-4c3682e818d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k\" (UID: \"b25794c4-a21c-4449-9feb-4c3682e818d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:01:59.219476 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.219381 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25794c4-a21c-4449-9feb-4c3682e818d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k\" (UID: \"b25794c4-a21c-4449-9feb-4c3682e818d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:01:59.219476 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.219467 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgnvk\" (UniqueName: \"kubernetes.io/projected/b25794c4-a21c-4449-9feb-4c3682e818d0-kube-api-access-jgnvk\") pod \"maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k\" (UID: \"b25794c4-a21c-4449-9feb-4c3682e818d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:01:59.219713 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.219508 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25794c4-a21c-4449-9feb-4c3682e818d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k\" (UID: \"b25794c4-a21c-4449-9feb-4c3682e818d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:01:59.219713 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:59.219540 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-serving-cert: secret "maas-api-e2e-shared-a-fd4ae7-serving-cert" not found Jul 17 02:01:59.219713 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:01:59.219606 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b25794c4-a21c-4449-9feb-4c3682e818d0-maas-api-tls podName:b25794c4-a21c-4449-9feb-4c3682e818d0 nodeName:}" failed. No retries permitted until 2026-07-17 02:01:59.719588548 +0000 UTC m=+2877.115203378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b25794c4-a21c-4449-9feb-4c3682e818d0-maas-api-tls") pod "maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" (UID: "b25794c4-a21c-4449-9feb-4c3682e818d0") : secret "maas-api-e2e-shared-a-fd4ae7-serving-cert" not found Jul 17 02:01:59.220230 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.220209 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25794c4-a21c-4449-9feb-4c3682e818d0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k\" (UID: \"b25794c4-a21c-4449-9feb-4c3682e818d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:01:59.229547 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.229520 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgnvk\" (UniqueName: \"kubernetes.io/projected/b25794c4-a21c-4449-9feb-4c3682e818d0-kube-api-access-jgnvk\") pod \"maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k\" (UID: \"b25794c4-a21c-4449-9feb-4c3682e818d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:01:59.272784 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.272743 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:59.531496 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.531468 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:59.630089 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.630056 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:01:59.722612 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.722576 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25794c4-a21c-4449-9feb-4c3682e818d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k\" (UID: \"b25794c4-a21c-4449-9feb-4c3682e818d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:01:59.725110 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.725082 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25794c4-a21c-4449-9feb-4c3682e818d0-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k\" (UID: \"b25794c4-a21c-4449-9feb-4c3682e818d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:01:59.963979 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:01:59.963883 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:02:00.106004 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:00.105965 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k"] Jul 17 02:02:00.107911 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:02:00.107870 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25794c4_a21c_4449_9feb_4c3682e818d0.slice/crio-9744ed62321911db1b6e78f5ab4aed8b5ae20601504ddf84431e6d6f57735e6b WatchSource:0}: Error finding container 9744ed62321911db1b6e78f5ab4aed8b5ae20601504ddf84431e6d6f57735e6b: Status 404 returned error can't find the container with id 9744ed62321911db1b6e78f5ab4aed8b5ae20601504ddf84431e6d6f57735e6b Jul 17 02:02:00.961638 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:00.961608 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" event={"ID":"b25794c4-a21c-4449-9feb-4c3682e818d0","Type":"ContainerStarted","Data":"2097882d52911338420bcb1590228ad05a91500da893d4bed4b0285800ed4b8e"} Jul 17 02:02:00.961825 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:00.961659 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" event={"ID":"b25794c4-a21c-4449-9feb-4c3682e818d0","Type":"ContainerStarted","Data":"9744ed62321911db1b6e78f5ab4aed8b5ae20601504ddf84431e6d6f57735e6b"} Jul 17 02:02:00.961825 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:00.961709 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:02:00.977640 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:00.977561 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" podStartSLOduration=1.6259983 podStartE2EDuration="1.977543176s" totalImagesPullingTime="351.544876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:02:00.977437726 +0000 UTC m=+2878.373052577" watchObservedRunningTime="2026-07-17 02:02:00.977543176 +0000 UTC m=+2878.373158026" Jul 17 02:02:03.391392 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:03.391324 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:03.725285 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:03.725193 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:04.051675 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:04.051643 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:04.238962 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:04.238930 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:06.350910 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:06.350871 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk"] Jul 17 02:02:06.356291 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:06.356274 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" Jul 17 02:02:06.361958 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:06.361929 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk"] Jul 17 02:02:06.387285 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:06.387249 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e71ed35-bd9f-4f04-9a91-90c92c552156-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk\" (UID: \"6e71ed35-bd9f-4f04-9a91-90c92c552156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" Jul 17 02:02:06.488358 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:06.488321 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e71ed35-bd9f-4f04-9a91-90c92c552156-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk\" (UID: \"6e71ed35-bd9f-4f04-9a91-90c92c552156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" Jul 17 02:02:06.488966 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:06.488943 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e71ed35-bd9f-4f04-9a91-90c92c552156-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk\" (UID: \"6e71ed35-bd9f-4f04-9a91-90c92c552156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" Jul 17 02:02:06.669043 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:06.668936 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" Jul 17 02:02:06.810241 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:06.810212 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk"] Jul 17 02:02:06.811813 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:02:06.811781 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e71ed35_bd9f_4f04_9a91_90c92c552156.slice/crio-b127efdf03df451949f581c74f7d35cb6d00518d0634a3eb2cc1072948a26f77 WatchSource:0}: Error finding container b127efdf03df451949f581c74f7d35cb6d00518d0634a3eb2cc1072948a26f77: Status 404 returned error can't find the container with id b127efdf03df451949f581c74f7d35cb6d00518d0634a3eb2cc1072948a26f77 Jul 17 02:02:06.972806 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:06.972724 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:02:06.994903 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:06.994861 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" event={"ID":"6e71ed35-bd9f-4f04-9a91-90c92c552156","Type":"ContainerStarted","Data":"b127efdf03df451949f581c74f7d35cb6d00518d0634a3eb2cc1072948a26f77"} Jul 17 02:02:07.054302 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.054269 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:07.133998 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.133973 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:07.134978 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.134958 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g"] Jul 17 02:02:07.138612 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.138595 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:02:07.141453 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.141436 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-1f2720-serving-cert\"" Jul 17 02:02:07.150996 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.150966 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g"] Jul 17 02:02:07.194944 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.194915 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3225281f-d7ca-4f64-b363-6f330d6c6a4f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g\" (UID: \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:02:07.195150 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.194965 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3225281f-d7ca-4f64-b363-6f330d6c6a4f-maas-api-tls\") pod \"maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g\" (UID: \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:02:07.195150 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.195031 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn8zz\" (UniqueName: \"kubernetes.io/projected/3225281f-d7ca-4f64-b363-6f330d6c6a4f-kube-api-access-tn8zz\") pod \"maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g\" (UID: \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:02:07.199446 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.199412 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:07.296286 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.296254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3225281f-d7ca-4f64-b363-6f330d6c6a4f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g\" (UID: \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:02:07.296473 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.296300 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3225281f-d7ca-4f64-b363-6f330d6c6a4f-maas-api-tls\") pod \"maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g\" (UID: \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:02:07.296473 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.296340 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8zz\" (UniqueName: \"kubernetes.io/projected/3225281f-d7ca-4f64-b363-6f330d6c6a4f-kube-api-access-tn8zz\") pod \"maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g\" (UID: \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:02:07.297032 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.297001 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3225281f-d7ca-4f64-b363-6f330d6c6a4f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g\" (UID: \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:02:07.299088 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.299064 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3225281f-d7ca-4f64-b363-6f330d6c6a4f-maas-api-tls\") pod \"maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g\" (UID: \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:02:07.302539 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.302505 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:07.304902 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.304880 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn8zz\" (UniqueName: \"kubernetes.io/projected/3225281f-d7ca-4f64-b363-6f330d6c6a4f-kube-api-access-tn8zz\") pod \"maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g\" (UID: \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:02:07.450176 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.450126 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:02:07.599924 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.599898 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g"] Jul 17 02:02:07.601982 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:02:07.601958 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3225281f_d7ca_4f64_b363_6f330d6c6a4f.slice/crio-cd5627976e96a10085dac67b26bbb7db38a5b876452ef5b2bee827513b214ca9 WatchSource:0}: Error finding container cd5627976e96a10085dac67b26bbb7db38a5b876452ef5b2bee827513b214ca9: Status 404 returned error can't find the container with id cd5627976e96a10085dac67b26bbb7db38a5b876452ef5b2bee827513b214ca9 Jul 17 02:02:07.847302 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.847267 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:07.944632 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:07.944549 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:08.004347 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:08.004312 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" event={"ID":"3225281f-d7ca-4f64-b363-6f330d6c6a4f","Type":"ContainerStarted","Data":"cd5627976e96a10085dac67b26bbb7db38a5b876452ef5b2bee827513b214ca9"} Jul 17 02:02:08.006027 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:08.006004 2583 generic.go:356] "Generic (PLEG): container finished" podID="6e71ed35-bd9f-4f04-9a91-90c92c552156" containerID="d53812acdef4ece659043f3f9d642465654207ef9fb2a2495b3ea38c0d1d79f7" exitCode=6 Jul 17 02:02:08.006167 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:08.006052 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" event={"ID":"6e71ed35-bd9f-4f04-9a91-90c92c552156","Type":"ContainerDied","Data":"d53812acdef4ece659043f3f9d642465654207ef9fb2a2495b3ea38c0d1d79f7"} Jul 17 02:02:08.006340 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:08.006322 2583 scope.go:117] "RemoveContainer" containerID="d53812acdef4ece659043f3f9d642465654207ef9fb2a2495b3ea38c0d1d79f7" Jul 17 02:02:08.162725 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:08.162682 2583 scope.go:117] "RemoveContainer" containerID="fbb71e3161db335d174287cf0656a38b5e7118b2894df72d1decf11e57223c11" Jul 17 02:02:09.014718 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:09.014617 2583 generic.go:356] "Generic (PLEG): container finished" podID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" containerID="0b9027899b49134db78fea2fb30623c9dbcbc5ae94b5c9ec0725d34081b46e24" exitCode=6 Jul 17 02:02:09.014718 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:09.014696 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" event={"ID":"0236a35f-9a7b-4945-abc9-efc1cf41f8c6","Type":"ContainerDied","Data":"0b9027899b49134db78fea2fb30623c9dbcbc5ae94b5c9ec0725d34081b46e24"} Jul 17 02:02:09.015274 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:09.014771 2583 scope.go:117] "RemoveContainer" containerID="fbb71e3161db335d174287cf0656a38b5e7118b2894df72d1decf11e57223c11" Jul 17 02:02:09.015274 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:09.015133 2583 scope.go:117] "RemoveContainer" containerID="0b9027899b49134db78fea2fb30623c9dbcbc5ae94b5c9ec0725d34081b46e24" Jul 17 02:02:09.015391 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:02:09.015363 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s_odh-ai-gateway-infra(0236a35f-9a7b-4945-abc9-efc1cf41f8c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" podUID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" Jul 17 02:02:09.016655 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:09.016626 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" event={"ID":"3225281f-d7ca-4f64-b363-6f330d6c6a4f","Type":"ContainerStarted","Data":"6b9ed85784052b80b1ba4d45b88313287d58994d8610ca341c519e6c73465ef4"} Jul 17 02:02:09.016759 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:09.016746 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:02:09.018436 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:09.018416 2583 generic.go:356] "Generic (PLEG): container finished" podID="6e71ed35-bd9f-4f04-9a91-90c92c552156" containerID="818f7bd77834cc4056cbb30b1e6f15c18a8107d165c2ea0f678999db04b99a7e" exitCode=6 Jul 17 02:02:09.018534 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:09.018449 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" event={"ID":"6e71ed35-bd9f-4f04-9a91-90c92c552156","Type":"ContainerDied","Data":"818f7bd77834cc4056cbb30b1e6f15c18a8107d165c2ea0f678999db04b99a7e"} Jul 17 02:02:09.018679 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:09.018665 2583 scope.go:117] "RemoveContainer" containerID="818f7bd77834cc4056cbb30b1e6f15c18a8107d165c2ea0f678999db04b99a7e" Jul 17 02:02:09.018840 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:02:09.018823 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk_odh-ai-gateway-infra(6e71ed35-bd9f-4f04-9a91-90c92c552156)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" podUID="6e71ed35-bd9f-4f04-9a91-90c92c552156" Jul 17 02:02:09.026307 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:09.026285 2583 scope.go:117] "RemoveContainer" containerID="d53812acdef4ece659043f3f9d642465654207ef9fb2a2495b3ea38c0d1d79f7" Jul 17 02:02:09.043686 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:09.043640 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" podStartSLOduration=1.700811792 podStartE2EDuration="2.043623836s" totalImagesPullingTime="342.812044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:02:09.042023527 +0000 UTC m=+2886.437638391" watchObservedRunningTime="2026-07-17 02:02:09.043623836 +0000 UTC m=+2886.439238709" Jul 17 02:02:10.025057 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.025020 2583 scope.go:117] "RemoveContainer" containerID="818f7bd77834cc4056cbb30b1e6f15c18a8107d165c2ea0f678999db04b99a7e" Jul 17 02:02:10.025519 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:02:10.025282 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk_odh-ai-gateway-infra(6e71ed35-bd9f-4f04-9a91-90c92c552156)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" podUID="6e71ed35-bd9f-4f04-9a91-90c92c552156" Jul 17 02:02:10.037537 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.037498 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s"] Jul 17 02:02:10.178579 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.178547 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" Jul 17 02:02:10.219345 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.219309 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0236a35f-9a7b-4945-abc9-efc1cf41f8c6-maas-api-service-ca\") pod \"0236a35f-9a7b-4945-abc9-efc1cf41f8c6\" (UID: \"0236a35f-9a7b-4945-abc9-efc1cf41f8c6\") " Jul 17 02:02:10.219700 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.219677 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0236a35f-9a7b-4945-abc9-efc1cf41f8c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0236a35f-9a7b-4945-abc9-efc1cf41f8c6" (UID: "0236a35f-9a7b-4945-abc9-efc1cf41f8c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:02:10.316779 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.316743 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh"] Jul 17 02:02:10.317142 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.317129 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" containerName="revoke-keys" Jul 17 02:02:10.317202 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.317144 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" containerName="revoke-keys" Jul 17 02:02:10.317202 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.317159 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" containerName="revoke-keys" Jul 17 02:02:10.317202 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.317165 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" containerName="revoke-keys" Jul 17 02:02:10.317311 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.317211 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" containerName="revoke-keys" Jul 17 02:02:10.317311 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.317222 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" containerName="revoke-keys" Jul 17 02:02:10.321766 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.321705 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0236a35f-9a7b-4945-abc9-efc1cf41f8c6-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:02:10.324175 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.324148 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" Jul 17 02:02:10.328000 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.327972 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh"] Jul 17 02:02:10.422914 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.422877 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8843e040-f0cc-4a2f-b33c-dc2bf5768ce2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh\" (UID: \"8843e040-f0cc-4a2f-b33c-dc2bf5768ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" Jul 17 02:02:10.524345 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.524304 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8843e040-f0cc-4a2f-b33c-dc2bf5768ce2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh\" (UID: \"8843e040-f0cc-4a2f-b33c-dc2bf5768ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" Jul 17 02:02:10.524889 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.524868 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8843e040-f0cc-4a2f-b33c-dc2bf5768ce2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh\" (UID: \"8843e040-f0cc-4a2f-b33c-dc2bf5768ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" Jul 17 02:02:10.636511 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.636415 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" Jul 17 02:02:10.765397 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:10.765367 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh"] Jul 17 02:02:10.767299 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:02:10.767272 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8843e040_f0cc_4a2f_b33c_dc2bf5768ce2.slice/crio-8cc0d5f3889d540ca222208afb86a39d7f28d6d4d570ddf4f0283946c8c12d4d WatchSource:0}: Error finding container 8cc0d5f3889d540ca222208afb86a39d7f28d6d4d570ddf4f0283946c8c12d4d: Status 404 returned error can't find the container with id 8cc0d5f3889d540ca222208afb86a39d7f28d6d4d570ddf4f0283946c8c12d4d Jul 17 02:02:11.034710 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:11.034681 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" Jul 17 02:02:11.035229 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:11.034704 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s" event={"ID":"0236a35f-9a7b-4945-abc9-efc1cf41f8c6","Type":"ContainerDied","Data":"25c98b001410e01d5f33eb8ee7da0a6b26c2cd58131b08dc5dc0185686d97b91"} Jul 17 02:02:11.035229 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:11.034775 2583 scope.go:117] "RemoveContainer" containerID="0b9027899b49134db78fea2fb30623c9dbcbc5ae94b5c9ec0725d34081b46e24" Jul 17 02:02:11.036538 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:11.036510 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" event={"ID":"8843e040-f0cc-4a2f-b33c-dc2bf5768ce2","Type":"ContainerStarted","Data":"d5e811c34d270ff596f89cbc0a9c6fbdb45bee0ac7cde7f080f4c3283d834818"} Jul 17 02:02:11.036672 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:11.036554 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" event={"ID":"8843e040-f0cc-4a2f-b33c-dc2bf5768ce2","Type":"ContainerStarted","Data":"8cc0d5f3889d540ca222208afb86a39d7f28d6d4d570ddf4f0283946c8c12d4d"} Jul 17 02:02:11.055677 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:11.055632 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" podStartSLOduration=1.055616066 podStartE2EDuration="1.055616066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:02:11.054161199 +0000 UTC m=+2888.449776052" watchObservedRunningTime="2026-07-17 02:02:11.055616066 +0000 UTC m=+2888.451230917" Jul 17 02:02:11.069149 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:11.069114 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s"] Jul 17 02:02:11.075151 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:11.075122 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5w42s"] Jul 17 02:02:11.077448 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:11.077425 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:11.139117 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:11.139069 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:11.169780 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:11.169738 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" path="/var/lib/kubelet/pods/0236a35f-9a7b-4945-abc9-efc1cf41f8c6/volumes" Jul 17 02:02:11.559633 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:11.559605 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:12.044363 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:12.044322 2583 generic.go:356] "Generic (PLEG): container finished" podID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" containerID="d5e811c34d270ff596f89cbc0a9c6fbdb45bee0ac7cde7f080f4c3283d834818" exitCode=6 Jul 17 02:02:12.044751 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:12.044414 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" event={"ID":"8843e040-f0cc-4a2f-b33c-dc2bf5768ce2","Type":"ContainerDied","Data":"d5e811c34d270ff596f89cbc0a9c6fbdb45bee0ac7cde7f080f4c3283d834818"} Jul 17 02:02:12.044751 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:12.044539 2583 scope.go:117] "RemoveContainer" containerID="d5e811c34d270ff596f89cbc0a9c6fbdb45bee0ac7cde7f080f4c3283d834818" Jul 17 02:02:13.051855 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:13.051817 2583 generic.go:356] "Generic (PLEG): container finished" podID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" containerID="f650d1e59c8858682ca7bf3061d9e8b13ae43cbd598414f46ab69d492bf16f18" exitCode=6 Jul 17 02:02:13.051855 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:13.051857 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" event={"ID":"8843e040-f0cc-4a2f-b33c-dc2bf5768ce2","Type":"ContainerDied","Data":"f650d1e59c8858682ca7bf3061d9e8b13ae43cbd598414f46ab69d492bf16f18"} Jul 17 02:02:13.052361 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:13.051926 2583 scope.go:117] "RemoveContainer" containerID="d5e811c34d270ff596f89cbc0a9c6fbdb45bee0ac7cde7f080f4c3283d834818" Jul 17 02:02:13.052361 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:13.052089 2583 scope.go:117] "RemoveContainer" containerID="f650d1e59c8858682ca7bf3061d9e8b13ae43cbd598414f46ab69d492bf16f18" Jul 17 02:02:13.052361 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:02:13.052338 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh_odh-ai-gateway-infra(8843e040-f0cc-4a2f-b33c-dc2bf5768ce2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" podUID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" Jul 17 02:02:14.058866 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:14.058842 2583 scope.go:117] "RemoveContainer" containerID="f650d1e59c8858682ca7bf3061d9e8b13ae43cbd598414f46ab69d492bf16f18" Jul 17 02:02:14.059281 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:02:14.059029 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh_odh-ai-gateway-infra(8843e040-f0cc-4a2f-b33c-dc2bf5768ce2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" podUID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" Jul 17 02:02:15.032141 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:15.032113 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:02:20.402946 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:20.402906 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:20.503948 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:20.503917 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:22.163003 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:22.162972 2583 scope.go:117] "RemoveContainer" containerID="818f7bd77834cc4056cbb30b1e6f15c18a8107d165c2ea0f678999db04b99a7e" Jul 17 02:02:23.110403 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:23.110311 2583 generic.go:356] "Generic (PLEG): container finished" podID="6e71ed35-bd9f-4f04-9a91-90c92c552156" containerID="2b862ab8f7a93f55373e789d905beade9ec560c3fd568e180f0a8a89cc7b480b" exitCode=6 Jul 17 02:02:23.110403 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:23.110388 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" event={"ID":"6e71ed35-bd9f-4f04-9a91-90c92c552156","Type":"ContainerDied","Data":"2b862ab8f7a93f55373e789d905beade9ec560c3fd568e180f0a8a89cc7b480b"} Jul 17 02:02:23.110626 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:23.110461 2583 scope.go:117] "RemoveContainer" containerID="818f7bd77834cc4056cbb30b1e6f15c18a8107d165c2ea0f678999db04b99a7e" Jul 17 02:02:23.110827 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:23.110810 2583 scope.go:117] "RemoveContainer" containerID="2b862ab8f7a93f55373e789d905beade9ec560c3fd568e180f0a8a89cc7b480b" Jul 17 02:02:23.111022 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:02:23.111000 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk_odh-ai-gateway-infra(6e71ed35-bd9f-4f04-9a91-90c92c552156)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" podUID="6e71ed35-bd9f-4f04-9a91-90c92c552156" Jul 17 02:02:24.014863 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:24.014835 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:24.135701 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:24.135666 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk"] Jul 17 02:02:24.295884 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:24.295857 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" Jul 17 02:02:24.465033 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:24.464993 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e71ed35-bd9f-4f04-9a91-90c92c552156-maas-api-service-ca\") pod \"6e71ed35-bd9f-4f04-9a91-90c92c552156\" (UID: \"6e71ed35-bd9f-4f04-9a91-90c92c552156\") " Jul 17 02:02:24.465367 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:24.465344 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e71ed35-bd9f-4f04-9a91-90c92c552156-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e71ed35-bd9f-4f04-9a91-90c92c552156" (UID: "6e71ed35-bd9f-4f04-9a91-90c92c552156"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:02:24.565806 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:24.565758 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e71ed35-bd9f-4f04-9a91-90c92c552156-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:02:25.123948 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:25.123917 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" Jul 17 02:02:25.124420 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:25.123949 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk" event={"ID":"6e71ed35-bd9f-4f04-9a91-90c92c552156","Type":"ContainerDied","Data":"b127efdf03df451949f581c74f7d35cb6d00518d0634a3eb2cc1072948a26f77"} Jul 17 02:02:25.124420 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:25.124010 2583 scope.go:117] "RemoveContainer" containerID="2b862ab8f7a93f55373e789d905beade9ec560c3fd568e180f0a8a89cc7b480b" Jul 17 02:02:25.146773 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:25.146744 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk"] Jul 17 02:02:25.152510 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:25.152482 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-85ksk"] Jul 17 02:02:25.163004 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:25.162977 2583 scope.go:117] "RemoveContainer" containerID="f650d1e59c8858682ca7bf3061d9e8b13ae43cbd598414f46ab69d492bf16f18" Jul 17 02:02:25.169743 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:25.169720 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e71ed35-bd9f-4f04-9a91-90c92c552156" path="/var/lib/kubelet/pods/6e71ed35-bd9f-4f04-9a91-90c92c552156/volumes" Jul 17 02:02:26.131919 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:26.131887 2583 generic.go:356] "Generic (PLEG): container finished" podID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" containerID="eb2ce109d527b57d10ef09a67a69050e1ad5e350f40f18ca0bbda9d2d8da47b9" exitCode=6 Jul 17 02:02:26.132343 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:26.131961 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" event={"ID":"8843e040-f0cc-4a2f-b33c-dc2bf5768ce2","Type":"ContainerDied","Data":"eb2ce109d527b57d10ef09a67a69050e1ad5e350f40f18ca0bbda9d2d8da47b9"} Jul 17 02:02:26.132343 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:26.132038 2583 scope.go:117] "RemoveContainer" containerID="f650d1e59c8858682ca7bf3061d9e8b13ae43cbd598414f46ab69d492bf16f18" Jul 17 02:02:26.132343 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:26.132338 2583 scope.go:117] "RemoveContainer" containerID="eb2ce109d527b57d10ef09a67a69050e1ad5e350f40f18ca0bbda9d2d8da47b9" Jul 17 02:02:26.132590 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:02:26.132564 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh_odh-ai-gateway-infra(8843e040-f0cc-4a2f-b33c-dc2bf5768ce2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" podUID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" Jul 17 02:02:27.156730 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:27.156689 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh"] Jul 17 02:02:27.292325 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:27.292302 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" Jul 17 02:02:27.388459 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:27.388421 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8843e040-f0cc-4a2f-b33c-dc2bf5768ce2-maas-api-service-ca\") pod \"8843e040-f0cc-4a2f-b33c-dc2bf5768ce2\" (UID: \"8843e040-f0cc-4a2f-b33c-dc2bf5768ce2\") " Jul 17 02:02:27.388795 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:27.388771 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8843e040-f0cc-4a2f-b33c-dc2bf5768ce2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" (UID: "8843e040-f0cc-4a2f-b33c-dc2bf5768ce2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:02:27.489780 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:27.489699 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8843e040-f0cc-4a2f-b33c-dc2bf5768ce2-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:02:28.144505 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:28.144478 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" Jul 17 02:02:28.144669 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:28.144507 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh" event={"ID":"8843e040-f0cc-4a2f-b33c-dc2bf5768ce2","Type":"ContainerDied","Data":"8cc0d5f3889d540ca222208afb86a39d7f28d6d4d570ddf4f0283946c8c12d4d"} Jul 17 02:02:28.144669 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:28.144566 2583 scope.go:117] "RemoveContainer" containerID="eb2ce109d527b57d10ef09a67a69050e1ad5e350f40f18ca0bbda9d2d8da47b9" Jul 17 02:02:28.170271 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:28.170234 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh"] Jul 17 02:02:28.175902 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:28.175872 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-sw7fh"] Jul 17 02:02:28.599753 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:28.599716 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:28.697887 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:28.697856 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:29.167218 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:29.167185 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" path="/var/lib/kubelet/pods/8843e040-f0cc-4a2f-b33c-dc2bf5768ce2/volumes" Jul 17 02:02:43.780966 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.780919 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7"] Jul 17 02:02:43.781573 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781469 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" containerName="revoke-keys" Jul 17 02:02:43.781573 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781489 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" containerName="revoke-keys" Jul 17 02:02:43.781573 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781521 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e71ed35-bd9f-4f04-9a91-90c92c552156" containerName="revoke-keys" Jul 17 02:02:43.781573 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781530 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e71ed35-bd9f-4f04-9a91-90c92c552156" containerName="revoke-keys" Jul 17 02:02:43.781573 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781542 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" containerName="revoke-keys" Jul 17 02:02:43.781573 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781551 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" containerName="revoke-keys" Jul 17 02:02:43.781573 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781576 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e71ed35-bd9f-4f04-9a91-90c92c552156" containerName="revoke-keys" Jul 17 02:02:43.781934 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781584 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e71ed35-bd9f-4f04-9a91-90c92c552156" containerName="revoke-keys" Jul 17 02:02:43.781934 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781594 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e71ed35-bd9f-4f04-9a91-90c92c552156" containerName="revoke-keys" Jul 17 02:02:43.781934 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781601 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e71ed35-bd9f-4f04-9a91-90c92c552156" containerName="revoke-keys" Jul 17 02:02:43.781934 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781614 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" containerName="revoke-keys" Jul 17 02:02:43.781934 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781622 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" containerName="revoke-keys" Jul 17 02:02:43.781934 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781702 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e71ed35-bd9f-4f04-9a91-90c92c552156" containerName="revoke-keys" Jul 17 02:02:43.781934 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781713 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" containerName="revoke-keys" Jul 17 02:02:43.781934 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781724 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0236a35f-9a7b-4945-abc9-efc1cf41f8c6" containerName="revoke-keys" Jul 17 02:02:43.781934 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781735 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e71ed35-bd9f-4f04-9a91-90c92c552156" containerName="revoke-keys" Jul 17 02:02:43.781934 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781746 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e71ed35-bd9f-4f04-9a91-90c92c552156" containerName="revoke-keys" Jul 17 02:02:43.781934 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.781757 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" containerName="revoke-keys" Jul 17 02:02:43.784994 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.784973 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" Jul 17 02:02:43.787494 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.787470 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 02:02:43.791261 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.791184 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7"] Jul 17 02:02:43.843558 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.843525 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6355660d-e717-42eb-a62b-c05f1375f3c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7\" (UID: \"6355660d-e717-42eb-a62b-c05f1375f3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" Jul 17 02:02:43.944549 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.944504 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6355660d-e717-42eb-a62b-c05f1375f3c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7\" (UID: \"6355660d-e717-42eb-a62b-c05f1375f3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" Jul 17 02:02:43.945180 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:43.945160 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6355660d-e717-42eb-a62b-c05f1375f3c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7\" (UID: \"6355660d-e717-42eb-a62b-c05f1375f3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" Jul 17 02:02:44.096818 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:44.096731 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" Jul 17 02:02:44.231247 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:44.231221 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7"] Jul 17 02:02:44.232860 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:02:44.232833 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6355660d_e717_42eb_a62b_c05f1375f3c9.slice/crio-fb505adeac8400ec41bd1c73afb9375abbc9130defaa514bfc033836691b8db0 WatchSource:0}: Error finding container fb505adeac8400ec41bd1c73afb9375abbc9130defaa514bfc033836691b8db0: Status 404 returned error can't find the container with id fb505adeac8400ec41bd1c73afb9375abbc9130defaa514bfc033836691b8db0 Jul 17 02:02:45.231063 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:45.230968 2583 generic.go:356] "Generic (PLEG): container finished" podID="6355660d-e717-42eb-a62b-c05f1375f3c9" containerID="0e51b772fe86069e02b4edc0ef7c8cae377f973b3c819519d77e70ed11a6d017" exitCode=6 Jul 17 02:02:45.231063 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:45.231017 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" event={"ID":"6355660d-e717-42eb-a62b-c05f1375f3c9","Type":"ContainerDied","Data":"0e51b772fe86069e02b4edc0ef7c8cae377f973b3c819519d77e70ed11a6d017"} Jul 17 02:02:45.231522 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:45.231065 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" event={"ID":"6355660d-e717-42eb-a62b-c05f1375f3c9","Type":"ContainerStarted","Data":"fb505adeac8400ec41bd1c73afb9375abbc9130defaa514bfc033836691b8db0"} Jul 17 02:02:45.231522 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:45.231293 2583 scope.go:117] "RemoveContainer" containerID="0e51b772fe86069e02b4edc0ef7c8cae377f973b3c819519d77e70ed11a6d017" Jul 17 02:02:46.238241 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:46.238150 2583 generic.go:356] "Generic (PLEG): container finished" podID="6355660d-e717-42eb-a62b-c05f1375f3c9" containerID="5501e95668e3c422d0b1a9135d9f686209e1f98bba163155640c728b26160bd6" exitCode=6 Jul 17 02:02:46.238698 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:46.238234 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" event={"ID":"6355660d-e717-42eb-a62b-c05f1375f3c9","Type":"ContainerDied","Data":"5501e95668e3c422d0b1a9135d9f686209e1f98bba163155640c728b26160bd6"} Jul 17 02:02:46.238698 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:46.238289 2583 scope.go:117] "RemoveContainer" containerID="0e51b772fe86069e02b4edc0ef7c8cae377f973b3c819519d77e70ed11a6d017" Jul 17 02:02:46.238698 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:46.238500 2583 scope.go:117] "RemoveContainer" containerID="5501e95668e3c422d0b1a9135d9f686209e1f98bba163155640c728b26160bd6" Jul 17 02:02:46.238877 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:02:46.238756 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7_odh-ai-gateway-infra(6355660d-e717-42eb-a62b-c05f1375f3c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" podUID="6355660d-e717-42eb-a62b-c05f1375f3c9" Jul 17 02:02:47.245400 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:47.245367 2583 scope.go:117] "RemoveContainer" containerID="5501e95668e3c422d0b1a9135d9f686209e1f98bba163155640c728b26160bd6" Jul 17 02:02:47.245768 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:02:47.245634 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7_odh-ai-gateway-infra(6355660d-e717-42eb-a62b-c05f1375f3c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" podUID="6355660d-e717-42eb-a62b-c05f1375f3c9" Jul 17 02:02:55.406905 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:55.406871 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:02:56.393969 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:56.393932 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl"] Jul 17 02:02:56.394356 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:56.394341 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" containerName="revoke-keys" Jul 17 02:02:56.394407 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:56.394358 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" containerName="revoke-keys" Jul 17 02:02:56.394448 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:56.394415 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8843e040-f0cc-4a2f-b33c-dc2bf5768ce2" containerName="revoke-keys" Jul 17 02:02:56.399020 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:56.399002 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" Jul 17 02:02:56.404065 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:56.404038 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl"] Jul 17 02:02:56.455166 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:56.455132 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9694e6ce-e2d6-4a4f-8080-ed1c15df2d84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl\" (UID: \"9694e6ce-e2d6-4a4f-8080-ed1c15df2d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" Jul 17 02:02:56.556183 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:56.556145 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9694e6ce-e2d6-4a4f-8080-ed1c15df2d84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl\" (UID: \"9694e6ce-e2d6-4a4f-8080-ed1c15df2d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" Jul 17 02:02:56.556735 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:56.556716 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9694e6ce-e2d6-4a4f-8080-ed1c15df2d84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl\" (UID: \"9694e6ce-e2d6-4a4f-8080-ed1c15df2d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" Jul 17 02:02:56.711665 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:56.711568 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" Jul 17 02:02:56.847942 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:56.847861 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl"] Jul 17 02:02:56.850754 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:02:56.850728 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9694e6ce_e2d6_4a4f_8080_ed1c15df2d84.slice/crio-1e657cf26426bbc499a4655535d677c1e13d1388188fb43f906a742df137c799 WatchSource:0}: Error finding container 1e657cf26426bbc499a4655535d677c1e13d1388188fb43f906a742df137c799: Status 404 returned error can't find the container with id 1e657cf26426bbc499a4655535d677c1e13d1388188fb43f906a742df137c799 Jul 17 02:02:57.299924 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:57.299891 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" event={"ID":"9694e6ce-e2d6-4a4f-8080-ed1c15df2d84","Type":"ContainerStarted","Data":"6c60004c53682fbdc263a42e58d7f2b6470533d8ca3406a5303cf9b295476aa2"} Jul 17 02:02:57.300119 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:57.299950 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" event={"ID":"9694e6ce-e2d6-4a4f-8080-ed1c15df2d84","Type":"ContainerStarted","Data":"1e657cf26426bbc499a4655535d677c1e13d1388188fb43f906a742df137c799"} Jul 17 02:02:57.315852 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:57.315799 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" podStartSLOduration=1.31578412 podStartE2EDuration="1.31578412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:02:57.314052168 +0000 UTC m=+2934.709667021" watchObservedRunningTime="2026-07-17 02:02:57.31578412 +0000 UTC m=+2934.711398971" Jul 17 02:02:58.306125 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:58.306060 2583 generic.go:356] "Generic (PLEG): container finished" podID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" containerID="6c60004c53682fbdc263a42e58d7f2b6470533d8ca3406a5303cf9b295476aa2" exitCode=6 Jul 17 02:02:58.306533 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:58.306146 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" event={"ID":"9694e6ce-e2d6-4a4f-8080-ed1c15df2d84","Type":"ContainerDied","Data":"6c60004c53682fbdc263a42e58d7f2b6470533d8ca3406a5303cf9b295476aa2"} Jul 17 02:02:58.306533 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:58.306405 2583 scope.go:117] "RemoveContainer" containerID="6c60004c53682fbdc263a42e58d7f2b6470533d8ca3406a5303cf9b295476aa2" Jul 17 02:02:58.414943 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:58.414904 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b"] Jul 17 02:02:58.419467 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:58.419449 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" Jul 17 02:02:58.425058 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:58.425033 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b"] Jul 17 02:02:58.472717 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:58.472683 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77ccf8b-ac1a-4b26-8134-9684633fbbc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b\" (UID: \"a77ccf8b-ac1a-4b26-8134-9684633fbbc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" Jul 17 02:02:58.573914 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:58.573829 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77ccf8b-ac1a-4b26-8134-9684633fbbc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b\" (UID: \"a77ccf8b-ac1a-4b26-8134-9684633fbbc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" Jul 17 02:02:58.574551 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:58.574530 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77ccf8b-ac1a-4b26-8134-9684633fbbc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b\" (UID: \"a77ccf8b-ac1a-4b26-8134-9684633fbbc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" Jul 17 02:02:58.730974 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:58.730934 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" Jul 17 02:02:58.869066 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:58.868978 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b"] Jul 17 02:02:58.872402 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:02:58.872366 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77ccf8b_ac1a_4b26_8134_9684633fbbc2.slice/crio-64a73ac555980b5fdd41ae0ced644027b0b00061a545a18278cc1a356f052aa6 WatchSource:0}: Error finding container 64a73ac555980b5fdd41ae0ced644027b0b00061a545a18278cc1a356f052aa6: Status 404 returned error can't find the container with id 64a73ac555980b5fdd41ae0ced644027b0b00061a545a18278cc1a356f052aa6 Jul 17 02:02:59.312908 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:59.312879 2583 generic.go:356] "Generic (PLEG): container finished" podID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" containerID="a9448cbd84b1352c13096b356ab4e67fba3b09d1e3cbc0d72480cb843fd063cb" exitCode=6 Jul 17 02:02:59.313394 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:59.312973 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" event={"ID":"9694e6ce-e2d6-4a4f-8080-ed1c15df2d84","Type":"ContainerDied","Data":"a9448cbd84b1352c13096b356ab4e67fba3b09d1e3cbc0d72480cb843fd063cb"} Jul 17 02:02:59.313394 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:59.313044 2583 scope.go:117] "RemoveContainer" containerID="6c60004c53682fbdc263a42e58d7f2b6470533d8ca3406a5303cf9b295476aa2" Jul 17 02:02:59.313394 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:59.313253 2583 scope.go:117] "RemoveContainer" containerID="a9448cbd84b1352c13096b356ab4e67fba3b09d1e3cbc0d72480cb843fd063cb" Jul 17 02:02:59.313524 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:02:59.313501 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl_odh-ai-gateway-infra(9694e6ce-e2d6-4a4f-8080-ed1c15df2d84)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" podUID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" Jul 17 02:02:59.314650 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:59.314616 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" event={"ID":"a77ccf8b-ac1a-4b26-8134-9684633fbbc2","Type":"ContainerStarted","Data":"35c20b74574f7b01428159a90c716b106108d750072d7c2562960d227ee9dc6d"} Jul 17 02:02:59.314856 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:59.314795 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" event={"ID":"a77ccf8b-ac1a-4b26-8134-9684633fbbc2","Type":"ContainerStarted","Data":"64a73ac555980b5fdd41ae0ced644027b0b00061a545a18278cc1a356f052aa6"} Jul 17 02:02:59.344160 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:02:59.344084 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" podStartSLOduration=1.3440692969999999 podStartE2EDuration="1.344069297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:02:59.342165662 +0000 UTC m=+2936.737780513" watchObservedRunningTime="2026-07-17 02:02:59.344069297 +0000 UTC m=+2936.739684147" Jul 17 02:03:00.321012 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:00.320973 2583 generic.go:356] "Generic (PLEG): container finished" podID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" containerID="35c20b74574f7b01428159a90c716b106108d750072d7c2562960d227ee9dc6d" exitCode=6 Jul 17 02:03:00.321498 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:00.321068 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" event={"ID":"a77ccf8b-ac1a-4b26-8134-9684633fbbc2","Type":"ContainerDied","Data":"35c20b74574f7b01428159a90c716b106108d750072d7c2562960d227ee9dc6d"} Jul 17 02:03:00.321498 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:00.321283 2583 scope.go:117] "RemoveContainer" containerID="35c20b74574f7b01428159a90c716b106108d750072d7c2562960d227ee9dc6d" Jul 17 02:03:00.322841 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:00.322821 2583 scope.go:117] "RemoveContainer" containerID="a9448cbd84b1352c13096b356ab4e67fba3b09d1e3cbc0d72480cb843fd063cb" Jul 17 02:03:00.322993 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:03:00.322976 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl_odh-ai-gateway-infra(9694e6ce-e2d6-4a4f-8080-ed1c15df2d84)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" podUID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" Jul 17 02:03:00.355531 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:00.355503 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5"] Jul 17 02:03:00.359964 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:00.359946 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" Jul 17 02:03:00.368463 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:00.368428 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5"] Jul 17 02:03:00.490092 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:00.490056 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f898f72-6e84-43f8-88cf-456c90d8d0be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5\" (UID: \"3f898f72-6e84-43f8-88cf-456c90d8d0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" Jul 17 02:03:00.591089 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:00.590992 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f898f72-6e84-43f8-88cf-456c90d8d0be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5\" (UID: \"3f898f72-6e84-43f8-88cf-456c90d8d0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" Jul 17 02:03:00.591671 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:00.591650 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f898f72-6e84-43f8-88cf-456c90d8d0be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5\" (UID: \"3f898f72-6e84-43f8-88cf-456c90d8d0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" Jul 17 02:03:00.672799 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:00.672758 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" Jul 17 02:03:00.805563 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:00.805534 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5"] Jul 17 02:03:00.807373 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:03:00.807341 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f898f72_6e84_43f8_88cf_456c90d8d0be.slice/crio-71ef0102111ff5928289e6207350ffca3240a6293e386ab435610bb5f8aa7b26 WatchSource:0}: Error finding container 71ef0102111ff5928289e6207350ffca3240a6293e386ab435610bb5f8aa7b26: Status 404 returned error can't find the container with id 71ef0102111ff5928289e6207350ffca3240a6293e386ab435610bb5f8aa7b26 Jul 17 02:03:01.163212 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:01.163179 2583 scope.go:117] "RemoveContainer" containerID="5501e95668e3c422d0b1a9135d9f686209e1f98bba163155640c728b26160bd6" Jul 17 02:03:01.331501 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:01.331468 2583 generic.go:356] "Generic (PLEG): container finished" podID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" containerID="cae5c0dd96a6eefc1dfaa47315111266355854781a478fc35d2c377f86802ced" exitCode=6 Jul 17 02:03:01.331952 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:01.331522 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" event={"ID":"a77ccf8b-ac1a-4b26-8134-9684633fbbc2","Type":"ContainerDied","Data":"cae5c0dd96a6eefc1dfaa47315111266355854781a478fc35d2c377f86802ced"} Jul 17 02:03:01.331952 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:01.331590 2583 scope.go:117] "RemoveContainer" containerID="35c20b74574f7b01428159a90c716b106108d750072d7c2562960d227ee9dc6d" Jul 17 02:03:01.331952 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:01.331763 2583 scope.go:117] "RemoveContainer" containerID="cae5c0dd96a6eefc1dfaa47315111266355854781a478fc35d2c377f86802ced" Jul 17 02:03:01.332500 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:03:01.332012 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b_odh-ai-gateway-infra(a77ccf8b-ac1a-4b26-8134-9684633fbbc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" podUID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" Jul 17 02:03:01.334016 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:01.333981 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" event={"ID":"3f898f72-6e84-43f8-88cf-456c90d8d0be","Type":"ContainerStarted","Data":"d01daca49142494b46c76bf909ea9c3e1e4591e942fa2c20be39865259c6660a"} Jul 17 02:03:01.334164 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:01.334092 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" event={"ID":"3f898f72-6e84-43f8-88cf-456c90d8d0be","Type":"ContainerStarted","Data":"71ef0102111ff5928289e6207350ffca3240a6293e386ab435610bb5f8aa7b26"} Jul 17 02:03:01.362020 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:01.361961 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" podStartSLOduration=1.361943962 podStartE2EDuration="1.361943962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:03:01.360146031 +0000 UTC m=+2938.755760883" watchObservedRunningTime="2026-07-17 02:03:01.361943962 +0000 UTC m=+2938.757558840" Jul 17 02:03:02.341458 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:02.341421 2583 generic.go:356] "Generic (PLEG): container finished" podID="3f898f72-6e84-43f8-88cf-456c90d8d0be" containerID="d01daca49142494b46c76bf909ea9c3e1e4591e942fa2c20be39865259c6660a" exitCode=6 Jul 17 02:03:02.341918 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:02.341510 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" event={"ID":"3f898f72-6e84-43f8-88cf-456c90d8d0be","Type":"ContainerDied","Data":"d01daca49142494b46c76bf909ea9c3e1e4591e942fa2c20be39865259c6660a"} Jul 17 02:03:02.341918 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:02.341701 2583 scope.go:117] "RemoveContainer" containerID="d01daca49142494b46c76bf909ea9c3e1e4591e942fa2c20be39865259c6660a" Jul 17 02:03:02.343395 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:02.343380 2583 scope.go:117] "RemoveContainer" containerID="cae5c0dd96a6eefc1dfaa47315111266355854781a478fc35d2c377f86802ced" Jul 17 02:03:02.343618 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:03:02.343565 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b_odh-ai-gateway-infra(a77ccf8b-ac1a-4b26-8134-9684633fbbc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" podUID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" Jul 17 02:03:02.344946 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:02.344924 2583 generic.go:356] "Generic (PLEG): container finished" podID="6355660d-e717-42eb-a62b-c05f1375f3c9" containerID="6310f332ed5aac1da830b817e63f0c76d9e37fb3da01299983fb41b29b4178f6" exitCode=6 Jul 17 02:03:02.345033 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:02.344955 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" event={"ID":"6355660d-e717-42eb-a62b-c05f1375f3c9","Type":"ContainerDied","Data":"6310f332ed5aac1da830b817e63f0c76d9e37fb3da01299983fb41b29b4178f6"} Jul 17 02:03:02.345033 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:02.344997 2583 scope.go:117] "RemoveContainer" containerID="5501e95668e3c422d0b1a9135d9f686209e1f98bba163155640c728b26160bd6" Jul 17 02:03:02.345315 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:02.345257 2583 scope.go:117] "RemoveContainer" containerID="6310f332ed5aac1da830b817e63f0c76d9e37fb3da01299983fb41b29b4178f6" Jul 17 02:03:02.345430 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:03:02.345412 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7_odh-ai-gateway-infra(6355660d-e717-42eb-a62b-c05f1375f3c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" podUID="6355660d-e717-42eb-a62b-c05f1375f3c9" Jul 17 02:03:03.354324 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:03.354290 2583 generic.go:356] "Generic (PLEG): container finished" podID="3f898f72-6e84-43f8-88cf-456c90d8d0be" containerID="8d5804d1d4f033c57c9ff968cc477b4b99a3a3853b3b0885f8456397ff6c9452" exitCode=6 Jul 17 02:03:03.354776 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:03.354336 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" event={"ID":"3f898f72-6e84-43f8-88cf-456c90d8d0be","Type":"ContainerDied","Data":"8d5804d1d4f033c57c9ff968cc477b4b99a3a3853b3b0885f8456397ff6c9452"} Jul 17 02:03:03.354776 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:03.354393 2583 scope.go:117] "RemoveContainer" containerID="d01daca49142494b46c76bf909ea9c3e1e4591e942fa2c20be39865259c6660a" Jul 17 02:03:03.354776 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:03.354565 2583 scope.go:117] "RemoveContainer" containerID="8d5804d1d4f033c57c9ff968cc477b4b99a3a3853b3b0885f8456397ff6c9452" Jul 17 02:03:03.354930 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:03:03.354834 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5_odh-ai-gateway-infra(3f898f72-6e84-43f8-88cf-456c90d8d0be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" podUID="3f898f72-6e84-43f8-88cf-456c90d8d0be" Jul 17 02:03:03.387415 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:03.387383 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7"] Jul 17 02:03:03.513928 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:03.513900 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" Jul 17 02:03:03.620759 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:03.620667 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6355660d-e717-42eb-a62b-c05f1375f3c9-maas-api-service-ca\") pod \"6355660d-e717-42eb-a62b-c05f1375f3c9\" (UID: \"6355660d-e717-42eb-a62b-c05f1375f3c9\") " Jul 17 02:03:03.621075 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:03.621045 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6355660d-e717-42eb-a62b-c05f1375f3c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6355660d-e717-42eb-a62b-c05f1375f3c9" (UID: "6355660d-e717-42eb-a62b-c05f1375f3c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:03:03.721744 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:03.721706 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6355660d-e717-42eb-a62b-c05f1375f3c9-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:03:04.360691 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:04.360660 2583 scope.go:117] "RemoveContainer" containerID="8d5804d1d4f033c57c9ff968cc477b4b99a3a3853b3b0885f8456397ff6c9452" Jul 17 02:03:04.361183 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:03:04.360907 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5_odh-ai-gateway-infra(3f898f72-6e84-43f8-88cf-456c90d8d0be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" podUID="3f898f72-6e84-43f8-88cf-456c90d8d0be" Jul 17 02:03:04.361781 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:04.361762 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" event={"ID":"6355660d-e717-42eb-a62b-c05f1375f3c9","Type":"ContainerDied","Data":"fb505adeac8400ec41bd1c73afb9375abbc9130defaa514bfc033836691b8db0"} Jul 17 02:03:04.361856 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:04.361782 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7" Jul 17 02:03:04.361856 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:04.361803 2583 scope.go:117] "RemoveContainer" containerID="6310f332ed5aac1da830b817e63f0c76d9e37fb3da01299983fb41b29b4178f6" Jul 17 02:03:04.385086 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:04.385055 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7"] Jul 17 02:03:04.389937 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:04.389907 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-2q9m7"] Jul 17 02:03:05.168566 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:05.168530 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6355660d-e717-42eb-a62b-c05f1375f3c9" path="/var/lib/kubelet/pods/6355660d-e717-42eb-a62b-c05f1375f3c9/volumes" Jul 17 02:03:15.162983 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:15.162951 2583 scope.go:117] "RemoveContainer" containerID="a9448cbd84b1352c13096b356ab4e67fba3b09d1e3cbc0d72480cb843fd063cb" Jul 17 02:03:16.425586 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:16.425552 2583 generic.go:356] "Generic (PLEG): container finished" podID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" containerID="0831364edb023ebd29654fbdcd13b72dc9de8e8eaab7f59025dd7d034e5d393d" exitCode=6 Jul 17 02:03:16.426005 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:16.425634 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" event={"ID":"9694e6ce-e2d6-4a4f-8080-ed1c15df2d84","Type":"ContainerDied","Data":"0831364edb023ebd29654fbdcd13b72dc9de8e8eaab7f59025dd7d034e5d393d"} Jul 17 02:03:16.426005 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:16.425700 2583 scope.go:117] "RemoveContainer" containerID="a9448cbd84b1352c13096b356ab4e67fba3b09d1e3cbc0d72480cb843fd063cb" Jul 17 02:03:16.426124 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:16.426014 2583 scope.go:117] "RemoveContainer" containerID="0831364edb023ebd29654fbdcd13b72dc9de8e8eaab7f59025dd7d034e5d393d" Jul 17 02:03:16.426314 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:03:16.426291 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl_odh-ai-gateway-infra(9694e6ce-e2d6-4a4f-8080-ed1c15df2d84)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" podUID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" Jul 17 02:03:17.452348 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:17.452311 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl"] Jul 17 02:03:17.597982 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:17.597946 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" Jul 17 02:03:17.750436 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:17.750338 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9694e6ce-e2d6-4a4f-8080-ed1c15df2d84-maas-api-service-ca\") pod \"9694e6ce-e2d6-4a4f-8080-ed1c15df2d84\" (UID: \"9694e6ce-e2d6-4a4f-8080-ed1c15df2d84\") " Jul 17 02:03:17.750701 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:17.750672 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9694e6ce-e2d6-4a4f-8080-ed1c15df2d84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" (UID: "9694e6ce-e2d6-4a4f-8080-ed1c15df2d84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:03:17.851838 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:17.851801 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9694e6ce-e2d6-4a4f-8080-ed1c15df2d84-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:03:18.163126 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:18.163080 2583 scope.go:117] "RemoveContainer" containerID="cae5c0dd96a6eefc1dfaa47315111266355854781a478fc35d2c377f86802ced" Jul 17 02:03:18.438134 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:18.438107 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" Jul 17 02:03:18.438302 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:18.438133 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl" event={"ID":"9694e6ce-e2d6-4a4f-8080-ed1c15df2d84","Type":"ContainerDied","Data":"1e657cf26426bbc499a4655535d677c1e13d1388188fb43f906a742df137c799"} Jul 17 02:03:18.438302 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:18.438186 2583 scope.go:117] "RemoveContainer" containerID="0831364edb023ebd29654fbdcd13b72dc9de8e8eaab7f59025dd7d034e5d393d" Jul 17 02:03:18.440191 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:18.440166 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" event={"ID":"a77ccf8b-ac1a-4b26-8134-9684633fbbc2","Type":"ContainerStarted","Data":"2f001993a4ea0f4d204a83744e82a6dcfe968f54a9e3123c4fd03d98810db019"} Jul 17 02:03:18.469754 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:18.469710 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl"] Jul 17 02:03:18.472897 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:18.472870 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-8ljtl"] Jul 17 02:03:19.162541 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:19.162455 2583 scope.go:117] "RemoveContainer" containerID="8d5804d1d4f033c57c9ff968cc477b4b99a3a3853b3b0885f8456397ff6c9452" Jul 17 02:03:19.168127 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:19.168087 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" path="/var/lib/kubelet/pods/9694e6ce-e2d6-4a4f-8080-ed1c15df2d84/volumes" Jul 17 02:03:19.450616 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:19.450529 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" event={"ID":"3f898f72-6e84-43f8-88cf-456c90d8d0be","Type":"ContainerStarted","Data":"73debae75639a42be2be01b048a4984fdb45447b3d46ea0da1b5034efe330989"} Jul 17 02:03:19.452274 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:19.452248 2583 generic.go:356] "Generic (PLEG): container finished" podID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" containerID="2f001993a4ea0f4d204a83744e82a6dcfe968f54a9e3123c4fd03d98810db019" exitCode=6 Jul 17 02:03:19.452395 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:19.452281 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" event={"ID":"a77ccf8b-ac1a-4b26-8134-9684633fbbc2","Type":"ContainerDied","Data":"2f001993a4ea0f4d204a83744e82a6dcfe968f54a9e3123c4fd03d98810db019"} Jul 17 02:03:19.452395 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:19.452322 2583 scope.go:117] "RemoveContainer" containerID="cae5c0dd96a6eefc1dfaa47315111266355854781a478fc35d2c377f86802ced" Jul 17 02:03:19.452635 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:19.452621 2583 scope.go:117] "RemoveContainer" containerID="2f001993a4ea0f4d204a83744e82a6dcfe968f54a9e3123c4fd03d98810db019" Jul 17 02:03:19.452822 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:03:19.452799 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b_odh-ai-gateway-infra(a77ccf8b-ac1a-4b26-8134-9684633fbbc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" podUID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" Jul 17 02:03:19.467192 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:19.467171 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b"] Jul 17 02:03:20.459670 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:20.459633 2583 generic.go:356] "Generic (PLEG): container finished" podID="3f898f72-6e84-43f8-88cf-456c90d8d0be" containerID="73debae75639a42be2be01b048a4984fdb45447b3d46ea0da1b5034efe330989" exitCode=6 Jul 17 02:03:20.460143 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:20.459706 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" event={"ID":"3f898f72-6e84-43f8-88cf-456c90d8d0be","Type":"ContainerDied","Data":"73debae75639a42be2be01b048a4984fdb45447b3d46ea0da1b5034efe330989"} Jul 17 02:03:20.460143 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:20.459766 2583 scope.go:117] "RemoveContainer" containerID="8d5804d1d4f033c57c9ff968cc477b4b99a3a3853b3b0885f8456397ff6c9452" Jul 17 02:03:20.460143 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:20.460041 2583 scope.go:117] "RemoveContainer" containerID="73debae75639a42be2be01b048a4984fdb45447b3d46ea0da1b5034efe330989" Jul 17 02:03:20.460348 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:03:20.460317 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5_odh-ai-gateway-infra(3f898f72-6e84-43f8-88cf-456c90d8d0be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" podUID="3f898f72-6e84-43f8-88cf-456c90d8d0be" Jul 17 02:03:20.483402 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:20.483373 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5"] Jul 17 02:03:20.599284 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:20.599259 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" Jul 17 02:03:20.777684 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:20.777600 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77ccf8b-ac1a-4b26-8134-9684633fbbc2-maas-api-service-ca\") pod \"a77ccf8b-ac1a-4b26-8134-9684633fbbc2\" (UID: \"a77ccf8b-ac1a-4b26-8134-9684633fbbc2\") " Jul 17 02:03:20.777994 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:20.777972 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a77ccf8b-ac1a-4b26-8134-9684633fbbc2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a77ccf8b-ac1a-4b26-8134-9684633fbbc2" (UID: "a77ccf8b-ac1a-4b26-8134-9684633fbbc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:03:20.878561 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:20.878532 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77ccf8b-ac1a-4b26-8134-9684633fbbc2-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:03:21.469077 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:21.469055 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" Jul 17 02:03:21.469423 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:21.469054 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b" event={"ID":"a77ccf8b-ac1a-4b26-8134-9684633fbbc2","Type":"ContainerDied","Data":"64a73ac555980b5fdd41ae0ced644027b0b00061a545a18278cc1a356f052aa6"} Jul 17 02:03:21.469423 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:21.469172 2583 scope.go:117] "RemoveContainer" containerID="2f001993a4ea0f4d204a83744e82a6dcfe968f54a9e3123c4fd03d98810db019" Jul 17 02:03:21.502861 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:21.502819 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b"] Jul 17 02:03:21.506093 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:21.506067 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-zh64b"] Jul 17 02:03:21.603763 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:21.603736 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" Jul 17 02:03:21.790240 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:21.790213 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f898f72-6e84-43f8-88cf-456c90d8d0be-maas-api-service-ca\") pod \"3f898f72-6e84-43f8-88cf-456c90d8d0be\" (UID: \"3f898f72-6e84-43f8-88cf-456c90d8d0be\") " Jul 17 02:03:21.790635 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:21.790616 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f898f72-6e84-43f8-88cf-456c90d8d0be-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f898f72-6e84-43f8-88cf-456c90d8d0be" (UID: "3f898f72-6e84-43f8-88cf-456c90d8d0be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:03:21.891691 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:21.891652 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f898f72-6e84-43f8-88cf-456c90d8d0be-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:03:22.475021 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:22.474992 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" Jul 17 02:03:22.475500 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:22.475037 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5" event={"ID":"3f898f72-6e84-43f8-88cf-456c90d8d0be","Type":"ContainerDied","Data":"71ef0102111ff5928289e6207350ffca3240a6293e386ab435610bb5f8aa7b26"} Jul 17 02:03:22.475500 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:22.475118 2583 scope.go:117] "RemoveContainer" containerID="73debae75639a42be2be01b048a4984fdb45447b3d46ea0da1b5034efe330989" Jul 17 02:03:22.499479 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:22.499444 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5"] Jul 17 02:03:22.505608 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:22.505583 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-7nbp5"] Jul 17 02:03:23.169042 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:23.169008 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f898f72-6e84-43f8-88cf-456c90d8d0be" path="/var/lib/kubelet/pods/3f898f72-6e84-43f8-88cf-456c90d8d0be/volumes" Jul 17 02:03:23.169382 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:23.169368 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" path="/var/lib/kubelet/pods/a77ccf8b-ac1a-4b26-8134-9684633fbbc2/volumes" Jul 17 02:03:43.821221 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821183 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5"] Jul 17 02:03:43.821625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821531 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" containerName="revoke-keys" Jul 17 02:03:43.821625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821542 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" containerName="revoke-keys" Jul 17 02:03:43.821625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821550 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" containerName="revoke-keys" Jul 17 02:03:43.821625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821555 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" containerName="revoke-keys" Jul 17 02:03:43.821625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821569 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" containerName="revoke-keys" Jul 17 02:03:43.821625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821576 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" containerName="revoke-keys" Jul 17 02:03:43.821625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821581 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6355660d-e717-42eb-a62b-c05f1375f3c9" containerName="revoke-keys" Jul 17 02:03:43.821625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821587 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6355660d-e717-42eb-a62b-c05f1375f3c9" containerName="revoke-keys" Jul 17 02:03:43.821625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821596 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f898f72-6e84-43f8-88cf-456c90d8d0be" containerName="revoke-keys" Jul 17 02:03:43.821625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821601 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f898f72-6e84-43f8-88cf-456c90d8d0be" containerName="revoke-keys" Jul 17 02:03:43.821625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821611 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f898f72-6e84-43f8-88cf-456c90d8d0be" containerName="revoke-keys" Jul 17 02:03:43.821625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821616 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f898f72-6e84-43f8-88cf-456c90d8d0be" containerName="revoke-keys" Jul 17 02:03:43.821625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821622 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f898f72-6e84-43f8-88cf-456c90d8d0be" containerName="revoke-keys" Jul 17 02:03:43.821625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821627 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f898f72-6e84-43f8-88cf-456c90d8d0be" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821635 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821640 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821647 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821652 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821659 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6355660d-e717-42eb-a62b-c05f1375f3c9" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821664 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6355660d-e717-42eb-a62b-c05f1375f3c9" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821713 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821721 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6355660d-e717-42eb-a62b-c05f1375f3c9" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821725 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f898f72-6e84-43f8-88cf-456c90d8d0be" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821730 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f898f72-6e84-43f8-88cf-456c90d8d0be" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821735 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821741 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821746 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821750 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a77ccf8b-ac1a-4b26-8134-9684633fbbc2" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821755 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6355660d-e717-42eb-a62b-c05f1375f3c9" containerName="revoke-keys" Jul 17 02:03:43.822051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.821760 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" containerName="revoke-keys" Jul 17 02:03:43.824476 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.824457 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" Jul 17 02:03:43.826898 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.826878 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 02:03:43.831281 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.831256 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5"] Jul 17 02:03:43.878485 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.878445 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/004174a0-a5f8-4c6a-a417-892396124630-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5\" (UID: \"004174a0-a5f8-4c6a-a417-892396124630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" Jul 17 02:03:43.979279 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.979243 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/004174a0-a5f8-4c6a-a417-892396124630-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5\" (UID: \"004174a0-a5f8-4c6a-a417-892396124630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" Jul 17 02:03:43.979988 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:43.979967 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/004174a0-a5f8-4c6a-a417-892396124630-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5\" (UID: \"004174a0-a5f8-4c6a-a417-892396124630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" Jul 17 02:03:44.136273 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:44.136183 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" Jul 17 02:03:44.269061 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:44.269034 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5"] Jul 17 02:03:44.272068 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:03:44.272040 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004174a0_a5f8_4c6a_a417_892396124630.slice/crio-3111c81ea8787502f6711180e929ae1e9deebbee8f147324c8f7b0747af0ef13 WatchSource:0}: Error finding container 3111c81ea8787502f6711180e929ae1e9deebbee8f147324c8f7b0747af0ef13: Status 404 returned error can't find the container with id 3111c81ea8787502f6711180e929ae1e9deebbee8f147324c8f7b0747af0ef13 Jul 17 02:03:44.587307 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:44.587264 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" event={"ID":"004174a0-a5f8-4c6a-a417-892396124630","Type":"ContainerStarted","Data":"c4023d04f37ec06d7c75301ecf419cfeba5ef518318a5edce227cf147d1a8437"} Jul 17 02:03:44.587508 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:44.587325 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" event={"ID":"004174a0-a5f8-4c6a-a417-892396124630","Type":"ContainerStarted","Data":"3111c81ea8787502f6711180e929ae1e9deebbee8f147324c8f7b0747af0ef13"} Jul 17 02:03:44.603155 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:44.603074 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" podStartSLOduration=1.603059445 podStartE2EDuration="1.603059445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:03:44.60159498 +0000 UTC m=+2981.997209832" watchObservedRunningTime="2026-07-17 02:03:44.603059445 +0000 UTC m=+2981.998674296" Jul 17 02:03:45.593795 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:45.593755 2583 generic.go:356] "Generic (PLEG): container finished" podID="004174a0-a5f8-4c6a-a417-892396124630" containerID="c4023d04f37ec06d7c75301ecf419cfeba5ef518318a5edce227cf147d1a8437" exitCode=6 Jul 17 02:03:45.594319 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:45.593807 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" event={"ID":"004174a0-a5f8-4c6a-a417-892396124630","Type":"ContainerDied","Data":"c4023d04f37ec06d7c75301ecf419cfeba5ef518318a5edce227cf147d1a8437"} Jul 17 02:03:45.594319 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:45.594064 2583 scope.go:117] "RemoveContainer" containerID="c4023d04f37ec06d7c75301ecf419cfeba5ef518318a5edce227cf147d1a8437" Jul 17 02:03:46.600125 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:46.600061 2583 generic.go:356] "Generic (PLEG): container finished" podID="004174a0-a5f8-4c6a-a417-892396124630" containerID="e003b5277ff07035e60b7fd55079fd1e3a222a3c15099af7de9f5fdb96be0339" exitCode=6 Jul 17 02:03:46.600554 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:46.600153 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" event={"ID":"004174a0-a5f8-4c6a-a417-892396124630","Type":"ContainerDied","Data":"e003b5277ff07035e60b7fd55079fd1e3a222a3c15099af7de9f5fdb96be0339"} Jul 17 02:03:46.600554 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:46.600210 2583 scope.go:117] "RemoveContainer" containerID="c4023d04f37ec06d7c75301ecf419cfeba5ef518318a5edce227cf147d1a8437" Jul 17 02:03:46.600554 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:46.600384 2583 scope.go:117] "RemoveContainer" containerID="e003b5277ff07035e60b7fd55079fd1e3a222a3c15099af7de9f5fdb96be0339" Jul 17 02:03:46.600701 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:03:46.600646 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5_odh-ai-gateway-infra(004174a0-a5f8-4c6a-a417-892396124630)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" podUID="004174a0-a5f8-4c6a-a417-892396124630" Jul 17 02:03:47.607146 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:47.607120 2583 scope.go:117] "RemoveContainer" containerID="e003b5277ff07035e60b7fd55079fd1e3a222a3c15099af7de9f5fdb96be0339" Jul 17 02:03:47.607527 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:03:47.607312 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5_odh-ai-gateway-infra(004174a0-a5f8-4c6a-a417-892396124630)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" podUID="004174a0-a5f8-4c6a-a417-892396124630" Jul 17 02:03:56.436780 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:56.436739 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm"] Jul 17 02:03:56.437180 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:56.437160 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6355660d-e717-42eb-a62b-c05f1375f3c9" containerName="revoke-keys" Jul 17 02:03:56.437180 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:56.437173 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6355660d-e717-42eb-a62b-c05f1375f3c9" containerName="revoke-keys" Jul 17 02:03:56.437262 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:56.437199 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" containerName="revoke-keys" Jul 17 02:03:56.437262 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:56.437205 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9694e6ce-e2d6-4a4f-8080-ed1c15df2d84" containerName="revoke-keys" Jul 17 02:03:56.437326 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:56.437261 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6355660d-e717-42eb-a62b-c05f1375f3c9" containerName="revoke-keys" Jul 17 02:03:56.437326 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:56.437274 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f898f72-6e84-43f8-88cf-456c90d8d0be" containerName="revoke-keys" Jul 17 02:03:56.441454 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:56.441438 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" Jul 17 02:03:56.450305 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:56.450274 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm"] Jul 17 02:03:56.589309 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:56.589266 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43c3d17-4efa-4b8f-a1e5-44173996f8db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm\" (UID: \"d43c3d17-4efa-4b8f-a1e5-44173996f8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" Jul 17 02:03:56.690330 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:56.690232 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43c3d17-4efa-4b8f-a1e5-44173996f8db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm\" (UID: \"d43c3d17-4efa-4b8f-a1e5-44173996f8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" Jul 17 02:03:56.690877 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:56.690855 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43c3d17-4efa-4b8f-a1e5-44173996f8db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm\" (UID: \"d43c3d17-4efa-4b8f-a1e5-44173996f8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" Jul 17 02:03:56.753039 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:56.753004 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" Jul 17 02:03:56.888862 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:56.888836 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm"] Jul 17 02:03:56.891393 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:03:56.891356 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43c3d17_4efa_4b8f_a1e5_44173996f8db.slice/crio-f1bc9d95c1f8222e99519e37811e022b12ba16c176a23f675e7b33996e416850 WatchSource:0}: Error finding container f1bc9d95c1f8222e99519e37811e022b12ba16c176a23f675e7b33996e416850: Status 404 returned error can't find the container with id f1bc9d95c1f8222e99519e37811e022b12ba16c176a23f675e7b33996e416850 Jul 17 02:03:57.657398 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:57.657358 2583 generic.go:356] "Generic (PLEG): container finished" podID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" containerID="27058e48698b88528b15367da6429de9f15aa747fe052aa6a372493c8ea1d0a1" exitCode=6 Jul 17 02:03:57.657780 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:57.657411 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" event={"ID":"d43c3d17-4efa-4b8f-a1e5-44173996f8db","Type":"ContainerDied","Data":"27058e48698b88528b15367da6429de9f15aa747fe052aa6a372493c8ea1d0a1"} Jul 17 02:03:57.657780 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:57.657464 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" event={"ID":"d43c3d17-4efa-4b8f-a1e5-44173996f8db","Type":"ContainerStarted","Data":"f1bc9d95c1f8222e99519e37811e022b12ba16c176a23f675e7b33996e416850"} Jul 17 02:03:57.657780 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:57.657656 2583 scope.go:117] "RemoveContainer" containerID="27058e48698b88528b15367da6429de9f15aa747fe052aa6a372493c8ea1d0a1" Jul 17 02:03:58.453372 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:58.453333 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp"] Jul 17 02:03:58.456589 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:58.456571 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" Jul 17 02:03:58.464278 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:58.463984 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp"] Jul 17 02:03:58.609245 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:58.609203 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp\" (UID: \"cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" Jul 17 02:03:58.664618 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:58.664579 2583 generic.go:356] "Generic (PLEG): container finished" podID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" containerID="759956d7217048482c093a1b746429290a25a5a0da2b051a692b4c7aabc734ce" exitCode=6 Jul 17 02:03:58.665013 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:58.664639 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" event={"ID":"d43c3d17-4efa-4b8f-a1e5-44173996f8db","Type":"ContainerDied","Data":"759956d7217048482c093a1b746429290a25a5a0da2b051a692b4c7aabc734ce"} Jul 17 02:03:58.665013 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:58.664700 2583 scope.go:117] "RemoveContainer" containerID="27058e48698b88528b15367da6429de9f15aa747fe052aa6a372493c8ea1d0a1" Jul 17 02:03:58.665013 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:58.664972 2583 scope.go:117] "RemoveContainer" containerID="759956d7217048482c093a1b746429290a25a5a0da2b051a692b4c7aabc734ce" Jul 17 02:03:58.665212 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:03:58.665177 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm_odh-ai-gateway-infra(d43c3d17-4efa-4b8f-a1e5-44173996f8db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" podUID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" Jul 17 02:03:58.710353 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:58.710319 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp\" (UID: \"cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" Jul 17 02:03:58.711009 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:58.710988 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp\" (UID: \"cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" Jul 17 02:03:58.768213 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:58.768178 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" Jul 17 02:03:58.905363 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:58.905334 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp"] Jul 17 02:03:58.907433 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:03:58.907403 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc85b11a_b5ed_4c45_a3bd_cfafa9238b4b.slice/crio-4a1988aeb01b892e99e229ee0b35e467eb5a955af450e3aaf74ce945c3faf18a WatchSource:0}: Error finding container 4a1988aeb01b892e99e229ee0b35e467eb5a955af450e3aaf74ce945c3faf18a: Status 404 returned error can't find the container with id 4a1988aeb01b892e99e229ee0b35e467eb5a955af450e3aaf74ce945c3faf18a Jul 17 02:03:59.671812 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:59.671787 2583 scope.go:117] "RemoveContainer" containerID="759956d7217048482c093a1b746429290a25a5a0da2b051a692b4c7aabc734ce" Jul 17 02:03:59.672242 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:03:59.671969 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm_odh-ai-gateway-infra(d43c3d17-4efa-4b8f-a1e5-44173996f8db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" podUID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" Jul 17 02:03:59.673125 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:59.673086 2583 generic.go:356] "Generic (PLEG): container finished" podID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" containerID="2a40c7c2858f272ab6b4c76ef95ddfd5314b4f56beb2ed7236fbd2066520dca1" exitCode=6 Jul 17 02:03:59.673259 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:59.673169 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" event={"ID":"cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b","Type":"ContainerDied","Data":"2a40c7c2858f272ab6b4c76ef95ddfd5314b4f56beb2ed7236fbd2066520dca1"} Jul 17 02:03:59.673259 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:59.673212 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" event={"ID":"cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b","Type":"ContainerStarted","Data":"4a1988aeb01b892e99e229ee0b35e467eb5a955af450e3aaf74ce945c3faf18a"} Jul 17 02:03:59.673377 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:59.673368 2583 scope.go:117] "RemoveContainer" containerID="2a40c7c2858f272ab6b4c76ef95ddfd5314b4f56beb2ed7236fbd2066520dca1" Jul 17 02:03:59.751551 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:59.751517 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t7pwn"] Jul 17 02:03:59.757302 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:59.757268 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" Jul 17 02:03:59.761366 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:59.761340 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 02:03:59.761688 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:59.761382 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-97vrz\"" Jul 17 02:03:59.761896 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:59.761427 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 02:03:59.764283 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:59.764260 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t7pwn"] Jul 17 02:03:59.819536 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:59.819507 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbc8t\" (UniqueName: \"kubernetes.io/projected/f90b9fa5-f369-42be-b18b-653770ff41e4-kube-api-access-kbc8t\") pod \"kuadrant-operator-catalog-t7pwn\" (UID: \"f90b9fa5-f369-42be-b18b-653770ff41e4\") " pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" Jul 17 02:03:59.920041 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:59.920007 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbc8t\" (UniqueName: \"kubernetes.io/projected/f90b9fa5-f369-42be-b18b-653770ff41e4-kube-api-access-kbc8t\") pod \"kuadrant-operator-catalog-t7pwn\" (UID: \"f90b9fa5-f369-42be-b18b-653770ff41e4\") " pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" Jul 17 02:03:59.927784 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:03:59.927758 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbc8t\" (UniqueName: \"kubernetes.io/projected/f90b9fa5-f369-42be-b18b-653770ff41e4-kube-api-access-kbc8t\") pod \"kuadrant-operator-catalog-t7pwn\" (UID: \"f90b9fa5-f369-42be-b18b-653770ff41e4\") " pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" Jul 17 02:04:00.068036 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.067997 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" Jul 17 02:04:00.206726 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.206701 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t7pwn"] Jul 17 02:04:00.209084 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:04:00.209058 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90b9fa5_f369_42be_b18b_653770ff41e4.slice/crio-18e0574bdf41e43b5d0abeb661603abaa68ec7a02757c4333b414ee6a295f9bf WatchSource:0}: Error finding container 18e0574bdf41e43b5d0abeb661603abaa68ec7a02757c4333b414ee6a295f9bf: Status 404 returned error can't find the container with id 18e0574bdf41e43b5d0abeb661603abaa68ec7a02757c4333b414ee6a295f9bf Jul 17 02:04:00.396567 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.396534 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq"] Jul 17 02:04:00.400019 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.399997 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" Jul 17 02:04:00.406939 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.406901 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq"] Jul 17 02:04:00.525795 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.525762 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/501ab11a-1c70-4b4f-bb90-af4e269e76bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq\" (UID: \"501ab11a-1c70-4b4f-bb90-af4e269e76bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" Jul 17 02:04:00.627167 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.627066 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/501ab11a-1c70-4b4f-bb90-af4e269e76bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq\" (UID: \"501ab11a-1c70-4b4f-bb90-af4e269e76bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" Jul 17 02:04:00.627700 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.627680 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/501ab11a-1c70-4b4f-bb90-af4e269e76bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq\" (UID: \"501ab11a-1c70-4b4f-bb90-af4e269e76bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" Jul 17 02:04:00.683315 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.683283 2583 generic.go:356] "Generic (PLEG): container finished" podID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" containerID="b4b89b73cee3b9368655723535344a7237d2a3658c768528cd4975307f73fe70" exitCode=6 Jul 17 02:04:00.683747 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.683369 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" event={"ID":"cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b","Type":"ContainerDied","Data":"b4b89b73cee3b9368655723535344a7237d2a3658c768528cd4975307f73fe70"} Jul 17 02:04:00.683747 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.683435 2583 scope.go:117] "RemoveContainer" containerID="2a40c7c2858f272ab6b4c76ef95ddfd5314b4f56beb2ed7236fbd2066520dca1" Jul 17 02:04:00.683747 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.683738 2583 scope.go:117] "RemoveContainer" containerID="b4b89b73cee3b9368655723535344a7237d2a3658c768528cd4975307f73fe70" Jul 17 02:04:00.684008 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:00.683985 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp_odh-ai-gateway-infra(cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" podUID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" Jul 17 02:04:00.684561 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.684502 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" event={"ID":"f90b9fa5-f369-42be-b18b-653770ff41e4","Type":"ContainerStarted","Data":"18e0574bdf41e43b5d0abeb661603abaa68ec7a02757c4333b414ee6a295f9bf"} Jul 17 02:04:00.711810 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.711791 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" Jul 17 02:04:00.847660 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:00.847633 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq"] Jul 17 02:04:00.849515 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:04:00.849479 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501ab11a_1c70_4b4f_bb90_af4e269e76bb.slice/crio-60e4b616e6e9782a4019dcc4bd0c2c4906d091223bacb24f0e63f663a0502c6d WatchSource:0}: Error finding container 60e4b616e6e9782a4019dcc4bd0c2c4906d091223bacb24f0e63f663a0502c6d: Status 404 returned error can't find the container with id 60e4b616e6e9782a4019dcc4bd0c2c4906d091223bacb24f0e63f663a0502c6d Jul 17 02:04:01.695678 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:01.695598 2583 scope.go:117] "RemoveContainer" containerID="b4b89b73cee3b9368655723535344a7237d2a3658c768528cd4975307f73fe70" Jul 17 02:04:01.696223 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:01.695855 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp_odh-ai-gateway-infra(cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" podUID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" Jul 17 02:04:01.698284 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:01.698255 2583 generic.go:356] "Generic (PLEG): container finished" podID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" containerID="0ddc6785016524248295816a82afea9247dede0c9f65bc2af6cc1efaa587ad03" exitCode=6 Jul 17 02:04:01.698403 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:01.698303 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" event={"ID":"501ab11a-1c70-4b4f-bb90-af4e269e76bb","Type":"ContainerDied","Data":"0ddc6785016524248295816a82afea9247dede0c9f65bc2af6cc1efaa587ad03"} Jul 17 02:04:01.698403 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:01.698341 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" event={"ID":"501ab11a-1c70-4b4f-bb90-af4e269e76bb","Type":"ContainerStarted","Data":"60e4b616e6e9782a4019dcc4bd0c2c4906d091223bacb24f0e63f663a0502c6d"} Jul 17 02:04:01.698609 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:01.698589 2583 scope.go:117] "RemoveContainer" containerID="0ddc6785016524248295816a82afea9247dede0c9f65bc2af6cc1efaa587ad03" Jul 17 02:04:02.162389 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:02.162357 2583 scope.go:117] "RemoveContainer" containerID="e003b5277ff07035e60b7fd55079fd1e3a222a3c15099af7de9f5fdb96be0339" Jul 17 02:04:02.705490 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:02.705437 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" event={"ID":"501ab11a-1c70-4b4f-bb90-af4e269e76bb","Type":"ContainerStarted","Data":"cc37a3a368f557c793cbc8f5cfbb4c732d98c2dbf217db63757cf34e515c47ec"} Jul 17 02:04:02.707218 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:02.707187 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" event={"ID":"f90b9fa5-f369-42be-b18b-653770ff41e4","Type":"ContainerStarted","Data":"688f21059c1f00dfc4bd5103b6c50bfa5a5af677ee312043bd886bc4ede551b5"} Jul 17 02:04:02.721287 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:02.721241 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" podStartSLOduration=2.721227208 podStartE2EDuration="2.721227208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:04:02.719181387 +0000 UTC m=+3000.114796242" watchObservedRunningTime="2026-07-17 02:04:02.721227208 +0000 UTC m=+3000.116842059" Jul 17 02:04:02.735763 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:02.735719 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" podStartSLOduration=1.357170074 podStartE2EDuration="3.735703s" totalImagesPullingTime="2.378532926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:04:02.735029248 +0000 UTC m=+3000.130644102" watchObservedRunningTime="2026-07-17 02:04:02.735703 +0000 UTC m=+3000.131317851" Jul 17 02:04:03.290118 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:03.290046 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 02:04:03.294089 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:03.294062 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 02:04:03.714029 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:03.713994 2583 generic.go:356] "Generic (PLEG): container finished" podID="004174a0-a5f8-4c6a-a417-892396124630" containerID="4275e24826d3e804888b16657e4e7c121f68d4074a31d67265f234f04728a70f" exitCode=6 Jul 17 02:04:03.714480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:03.714064 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" event={"ID":"004174a0-a5f8-4c6a-a417-892396124630","Type":"ContainerDied","Data":"4275e24826d3e804888b16657e4e7c121f68d4074a31d67265f234f04728a70f"} Jul 17 02:04:03.714480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:03.714147 2583 scope.go:117] "RemoveContainer" containerID="e003b5277ff07035e60b7fd55079fd1e3a222a3c15099af7de9f5fdb96be0339" Jul 17 02:04:03.714480 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:03.714435 2583 scope.go:117] "RemoveContainer" containerID="4275e24826d3e804888b16657e4e7c121f68d4074a31d67265f234f04728a70f" Jul 17 02:04:03.714720 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:03.714697 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5_odh-ai-gateway-infra(004174a0-a5f8-4c6a-a417-892396124630)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" podUID="004174a0-a5f8-4c6a-a417-892396124630" Jul 17 02:04:03.716077 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:03.716056 2583 generic.go:356] "Generic (PLEG): container finished" podID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" containerID="cc37a3a368f557c793cbc8f5cfbb4c732d98c2dbf217db63757cf34e515c47ec" exitCode=6 Jul 17 02:04:03.716191 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:03.716140 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" event={"ID":"501ab11a-1c70-4b4f-bb90-af4e269e76bb","Type":"ContainerDied","Data":"cc37a3a368f557c793cbc8f5cfbb4c732d98c2dbf217db63757cf34e515c47ec"} Jul 17 02:04:03.716355 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:03.716341 2583 scope.go:117] "RemoveContainer" containerID="cc37a3a368f557c793cbc8f5cfbb4c732d98c2dbf217db63757cf34e515c47ec" Jul 17 02:04:03.716511 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:03.716494 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq_odh-ai-gateway-infra(501ab11a-1c70-4b4f-bb90-af4e269e76bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" podUID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" Jul 17 02:04:03.727021 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:03.727005 2583 scope.go:117] "RemoveContainer" containerID="0ddc6785016524248295816a82afea9247dede0c9f65bc2af6cc1efaa587ad03" Jul 17 02:04:04.725117 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:04.725070 2583 scope.go:117] "RemoveContainer" containerID="cc37a3a368f557c793cbc8f5cfbb4c732d98c2dbf217db63757cf34e515c47ec" Jul 17 02:04:04.725515 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:04.725280 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq_odh-ai-gateway-infra(501ab11a-1c70-4b4f-bb90-af4e269e76bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" podUID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" Jul 17 02:04:04.738108 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:04.738060 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5"] Jul 17 02:04:04.868035 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:04.868005 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" Jul 17 02:04:04.969370 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:04.969337 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/004174a0-a5f8-4c6a-a417-892396124630-maas-api-service-ca\") pod \"004174a0-a5f8-4c6a-a417-892396124630\" (UID: \"004174a0-a5f8-4c6a-a417-892396124630\") " Jul 17 02:04:04.969736 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:04.969710 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004174a0-a5f8-4c6a-a417-892396124630-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "004174a0-a5f8-4c6a-a417-892396124630" (UID: "004174a0-a5f8-4c6a-a417-892396124630"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:04:05.070091 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:05.070058 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/004174a0-a5f8-4c6a-a417-892396124630-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:04:05.731468 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:05.731442 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" Jul 17 02:04:05.731893 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:05.731438 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5" event={"ID":"004174a0-a5f8-4c6a-a417-892396124630","Type":"ContainerDied","Data":"3111c81ea8787502f6711180e929ae1e9deebbee8f147324c8f7b0747af0ef13"} Jul 17 02:04:05.731893 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:05.731580 2583 scope.go:117] "RemoveContainer" containerID="4275e24826d3e804888b16657e4e7c121f68d4074a31d67265f234f04728a70f" Jul 17 02:04:05.748600 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:05.748575 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5"] Jul 17 02:04:05.754487 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:05.754460 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-5bft5"] Jul 17 02:04:07.168958 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:07.168923 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="004174a0-a5f8-4c6a-a417-892396124630" path="/var/lib/kubelet/pods/004174a0-a5f8-4c6a-a417-892396124630/volumes" Jul 17 02:04:10.068311 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:10.068262 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" Jul 17 02:04:10.068734 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:10.068357 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" Jul 17 02:04:10.091722 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:10.091688 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" Jul 17 02:04:10.162413 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:10.162383 2583 scope.go:117] "RemoveContainer" containerID="759956d7217048482c093a1b746429290a25a5a0da2b051a692b4c7aabc734ce" Jul 17 02:04:10.763181 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:10.763076 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" event={"ID":"d43c3d17-4efa-4b8f-a1e5-44173996f8db","Type":"ContainerStarted","Data":"f8793a2757efcdcbeb888ac84fd8eeaa19e80e09b55c37b3239a8bfb8a6660b5"} Jul 17 02:04:10.781467 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:10.781390 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" podStartSLOduration=14.781375833 podStartE2EDuration="14.781375833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:04:10.780488857 +0000 UTC m=+3008.176103706" watchObservedRunningTime="2026-07-17 02:04:10.781375833 +0000 UTC m=+3008.176990683" Jul 17 02:04:10.786613 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:10.786584 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" Jul 17 02:04:10.824839 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:10.824804 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t7pwn"] Jul 17 02:04:11.769689 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:11.769658 2583 generic.go:356] "Generic (PLEG): container finished" podID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" containerID="f8793a2757efcdcbeb888ac84fd8eeaa19e80e09b55c37b3239a8bfb8a6660b5" exitCode=6 Jul 17 02:04:11.770125 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:11.769730 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" event={"ID":"d43c3d17-4efa-4b8f-a1e5-44173996f8db","Type":"ContainerDied","Data":"f8793a2757efcdcbeb888ac84fd8eeaa19e80e09b55c37b3239a8bfb8a6660b5"} Jul 17 02:04:11.770125 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:11.769790 2583 scope.go:117] "RemoveContainer" containerID="759956d7217048482c093a1b746429290a25a5a0da2b051a692b4c7aabc734ce" Jul 17 02:04:11.770125 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:11.770069 2583 scope.go:117] "RemoveContainer" containerID="f8793a2757efcdcbeb888ac84fd8eeaa19e80e09b55c37b3239a8bfb8a6660b5" Jul 17 02:04:11.770324 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:11.770303 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm_odh-ai-gateway-infra(d43c3d17-4efa-4b8f-a1e5-44173996f8db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" podUID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" Jul 17 02:04:11.792210 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:11.792181 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm"] Jul 17 02:04:12.776191 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:12.776149 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" podUID="f90b9fa5-f369-42be-b18b-653770ff41e4" containerName="registry-server" containerID="cri-o://688f21059c1f00dfc4bd5103b6c50bfa5a5af677ee312043bd886bc4ede551b5" gracePeriod=2 Jul 17 02:04:12.907931 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:12.907907 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" Jul 17 02:04:13.031551 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.031530 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" Jul 17 02:04:13.042166 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.042144 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43c3d17-4efa-4b8f-a1e5-44173996f8db-maas-api-service-ca\") pod \"d43c3d17-4efa-4b8f-a1e5-44173996f8db\" (UID: \"d43c3d17-4efa-4b8f-a1e5-44173996f8db\") " Jul 17 02:04:13.046584 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.042893 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43c3d17-4efa-4b8f-a1e5-44173996f8db-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d43c3d17-4efa-4b8f-a1e5-44173996f8db" (UID: "d43c3d17-4efa-4b8f-a1e5-44173996f8db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:04:13.142785 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.142755 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbc8t\" (UniqueName: \"kubernetes.io/projected/f90b9fa5-f369-42be-b18b-653770ff41e4-kube-api-access-kbc8t\") pod \"f90b9fa5-f369-42be-b18b-653770ff41e4\" (UID: \"f90b9fa5-f369-42be-b18b-653770ff41e4\") " Jul 17 02:04:13.143050 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.143034 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43c3d17-4efa-4b8f-a1e5-44173996f8db-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:04:13.145062 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.145038 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90b9fa5-f369-42be-b18b-653770ff41e4-kube-api-access-kbc8t" (OuterVolumeSpecName: "kube-api-access-kbc8t") pod "f90b9fa5-f369-42be-b18b-653770ff41e4" (UID: "f90b9fa5-f369-42be-b18b-653770ff41e4"). InnerVolumeSpecName "kube-api-access-kbc8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 02:04:13.243814 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.243772 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbc8t\" (UniqueName: \"kubernetes.io/projected/f90b9fa5-f369-42be-b18b-653770ff41e4-kube-api-access-kbc8t\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:04:13.781634 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.781596 2583 generic.go:356] "Generic (PLEG): container finished" podID="f90b9fa5-f369-42be-b18b-653770ff41e4" containerID="688f21059c1f00dfc4bd5103b6c50bfa5a5af677ee312043bd886bc4ede551b5" exitCode=0 Jul 17 02:04:13.782090 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.781659 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" Jul 17 02:04:13.782090 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.781679 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" event={"ID":"f90b9fa5-f369-42be-b18b-653770ff41e4","Type":"ContainerDied","Data":"688f21059c1f00dfc4bd5103b6c50bfa5a5af677ee312043bd886bc4ede551b5"} Jul 17 02:04:13.782090 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.781736 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-t7pwn" event={"ID":"f90b9fa5-f369-42be-b18b-653770ff41e4","Type":"ContainerDied","Data":"18e0574bdf41e43b5d0abeb661603abaa68ec7a02757c4333b414ee6a295f9bf"} Jul 17 02:04:13.782090 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.781763 2583 scope.go:117] "RemoveContainer" containerID="688f21059c1f00dfc4bd5103b6c50bfa5a5af677ee312043bd886bc4ede551b5" Jul 17 02:04:13.783389 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.783211 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" event={"ID":"d43c3d17-4efa-4b8f-a1e5-44173996f8db","Type":"ContainerDied","Data":"f1bc9d95c1f8222e99519e37811e022b12ba16c176a23f675e7b33996e416850"} Jul 17 02:04:13.783389 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.783245 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm" Jul 17 02:04:13.790829 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.790812 2583 scope.go:117] "RemoveContainer" containerID="688f21059c1f00dfc4bd5103b6c50bfa5a5af677ee312043bd886bc4ede551b5" Jul 17 02:04:13.791080 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:13.791061 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688f21059c1f00dfc4bd5103b6c50bfa5a5af677ee312043bd886bc4ede551b5\": container with ID starting with 688f21059c1f00dfc4bd5103b6c50bfa5a5af677ee312043bd886bc4ede551b5 not found: ID does not exist" containerID="688f21059c1f00dfc4bd5103b6c50bfa5a5af677ee312043bd886bc4ede551b5" Jul 17 02:04:13.791176 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.791092 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688f21059c1f00dfc4bd5103b6c50bfa5a5af677ee312043bd886bc4ede551b5"} err="failed to get container status \"688f21059c1f00dfc4bd5103b6c50bfa5a5af677ee312043bd886bc4ede551b5\": rpc error: code = NotFound desc = could not find container \"688f21059c1f00dfc4bd5103b6c50bfa5a5af677ee312043bd886bc4ede551b5\": container with ID starting with 688f21059c1f00dfc4bd5103b6c50bfa5a5af677ee312043bd886bc4ede551b5 not found: ID does not exist" Jul 17 02:04:13.791176 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.791143 2583 scope.go:117] "RemoveContainer" containerID="f8793a2757efcdcbeb888ac84fd8eeaa19e80e09b55c37b3239a8bfb8a6660b5" Jul 17 02:04:13.797710 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.797689 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm"] Jul 17 02:04:13.800756 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.800735 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-sjjnm"] Jul 17 02:04:13.811605 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.811582 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t7pwn"] Jul 17 02:04:13.815654 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:13.815633 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-t7pwn"] Jul 17 02:04:14.163168 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:14.163065 2583 scope.go:117] "RemoveContainer" containerID="b4b89b73cee3b9368655723535344a7237d2a3658c768528cd4975307f73fe70" Jul 17 02:04:14.793395 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:14.793346 2583 generic.go:356] "Generic (PLEG): container finished" podID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" containerID="121d5887573c9d0ca0f1358358ed96a0c7d050fd82c68d5c5aaa434d5f9a5fc9" exitCode=6 Jul 17 02:04:14.793847 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:14.793404 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" event={"ID":"cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b","Type":"ContainerDied","Data":"121d5887573c9d0ca0f1358358ed96a0c7d050fd82c68d5c5aaa434d5f9a5fc9"} Jul 17 02:04:14.793847 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:14.793469 2583 scope.go:117] "RemoveContainer" containerID="b4b89b73cee3b9368655723535344a7237d2a3658c768528cd4975307f73fe70" Jul 17 02:04:14.794016 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:14.793883 2583 scope.go:117] "RemoveContainer" containerID="121d5887573c9d0ca0f1358358ed96a0c7d050fd82c68d5c5aaa434d5f9a5fc9" Jul 17 02:04:14.794221 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:14.794193 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp_odh-ai-gateway-infra(cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" podUID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" Jul 17 02:04:15.168122 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:15.168052 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" path="/var/lib/kubelet/pods/d43c3d17-4efa-4b8f-a1e5-44173996f8db/volumes" Jul 17 02:04:15.168522 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:15.168500 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f90b9fa5-f369-42be-b18b-653770ff41e4" path="/var/lib/kubelet/pods/f90b9fa5-f369-42be-b18b-653770ff41e4/volumes" Jul 17 02:04:15.818310 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:15.818279 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp"] Jul 17 02:04:15.954091 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:15.954070 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" Jul 17 02:04:15.968043 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:15.968017 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b-maas-api-service-ca\") pod \"cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b\" (UID: \"cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b\") " Jul 17 02:04:15.968554 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:15.968463 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" (UID: "cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:04:15.968756 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:15.968734 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:04:16.162562 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:16.162474 2583 scope.go:117] "RemoveContainer" containerID="cc37a3a368f557c793cbc8f5cfbb4c732d98c2dbf217db63757cf34e515c47ec" Jul 17 02:04:16.805840 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:16.805811 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" Jul 17 02:04:16.806037 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:16.805810 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp" event={"ID":"cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b","Type":"ContainerDied","Data":"4a1988aeb01b892e99e229ee0b35e467eb5a955af450e3aaf74ce945c3faf18a"} Jul 17 02:04:16.806037 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:16.805937 2583 scope.go:117] "RemoveContainer" containerID="121d5887573c9d0ca0f1358358ed96a0c7d050fd82c68d5c5aaa434d5f9a5fc9" Jul 17 02:04:16.807820 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:16.807792 2583 generic.go:356] "Generic (PLEG): container finished" podID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" containerID="3cfd7e6cffa3fb2b82da9817371fde9f02fee02f5c395e60d83901264e6fd9e8" exitCode=6 Jul 17 02:04:16.807932 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:16.807851 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" event={"ID":"501ab11a-1c70-4b4f-bb90-af4e269e76bb","Type":"ContainerDied","Data":"3cfd7e6cffa3fb2b82da9817371fde9f02fee02f5c395e60d83901264e6fd9e8"} Jul 17 02:04:16.808214 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:16.808199 2583 scope.go:117] "RemoveContainer" containerID="3cfd7e6cffa3fb2b82da9817371fde9f02fee02f5c395e60d83901264e6fd9e8" Jul 17 02:04:16.808398 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:16.808381 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq_odh-ai-gateway-infra(501ab11a-1c70-4b4f-bb90-af4e269e76bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" podUID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" Jul 17 02:04:16.814810 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:16.814788 2583 scope.go:117] "RemoveContainer" containerID="cc37a3a368f557c793cbc8f5cfbb4c732d98c2dbf217db63757cf34e515c47ec" Jul 17 02:04:16.838254 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:16.838225 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp"] Jul 17 02:04:16.841546 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:16.841523 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-2gcpp"] Jul 17 02:04:17.167713 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:17.167674 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" path="/var/lib/kubelet/pods/cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b/volumes" Jul 17 02:04:17.833082 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:17.833052 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq"] Jul 17 02:04:17.962570 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:17.962546 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" Jul 17 02:04:17.981589 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:17.981565 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/501ab11a-1c70-4b4f-bb90-af4e269e76bb-maas-api-service-ca\") pod \"501ab11a-1c70-4b4f-bb90-af4e269e76bb\" (UID: \"501ab11a-1c70-4b4f-bb90-af4e269e76bb\") " Jul 17 02:04:17.982005 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:17.981972 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/501ab11a-1c70-4b4f-bb90-af4e269e76bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "501ab11a-1c70-4b4f-bb90-af4e269e76bb" (UID: "501ab11a-1c70-4b4f-bb90-af4e269e76bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:04:18.082872 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:18.082830 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/501ab11a-1c70-4b4f-bb90-af4e269e76bb-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:04:18.821006 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:18.820971 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" Jul 17 02:04:18.821212 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:18.820971 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq" event={"ID":"501ab11a-1c70-4b4f-bb90-af4e269e76bb","Type":"ContainerDied","Data":"60e4b616e6e9782a4019dcc4bd0c2c4906d091223bacb24f0e63f663a0502c6d"} Jul 17 02:04:18.821212 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:18.821136 2583 scope.go:117] "RemoveContainer" containerID="3cfd7e6cffa3fb2b82da9817371fde9f02fee02f5c395e60d83901264e6fd9e8" Jul 17 02:04:18.843783 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:18.843754 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq"] Jul 17 02:04:18.848426 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:18.848391 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-9m9fq"] Jul 17 02:04:19.167565 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:19.167487 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" path="/var/lib/kubelet/pods/501ab11a-1c70-4b4f-bb90-af4e269e76bb/volumes" Jul 17 02:04:26.953307 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:26.953268 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:04:36.180759 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:36.180722 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:04:43.876700 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.876667 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6"] Jul 17 02:04:43.877161 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877068 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90b9fa5-f369-42be-b18b-653770ff41e4" containerName="registry-server" Jul 17 02:04:43.877161 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877084 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90b9fa5-f369-42be-b18b-653770ff41e4" containerName="registry-server" Jul 17 02:04:43.877161 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877111 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" containerName="revoke-keys" Jul 17 02:04:43.877161 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877120 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" containerName="revoke-keys" Jul 17 02:04:43.877161 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877130 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" containerName="revoke-keys" Jul 17 02:04:43.877161 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877136 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" containerName="revoke-keys" Jul 17 02:04:43.877161 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877147 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" containerName="revoke-keys" Jul 17 02:04:43.877161 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877152 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" containerName="revoke-keys" Jul 17 02:04:43.877161 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877158 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="004174a0-a5f8-4c6a-a417-892396124630" containerName="revoke-keys" Jul 17 02:04:43.877161 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877166 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="004174a0-a5f8-4c6a-a417-892396124630" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877174 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877179 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877191 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877200 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877210 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877218 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877229 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="004174a0-a5f8-4c6a-a417-892396124630" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877236 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="004174a0-a5f8-4c6a-a417-892396124630" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877242 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877250 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877258 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="004174a0-a5f8-4c6a-a417-892396124630" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877263 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="004174a0-a5f8-4c6a-a417-892396124630" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877354 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f90b9fa5-f369-42be-b18b-653770ff41e4" containerName="registry-server" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877366 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877375 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877384 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877393 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877401 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877412 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d43c3d17-4efa-4b8f-a1e5-44173996f8db" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877420 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="004174a0-a5f8-4c6a-a417-892396124630" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877428 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="004174a0-a5f8-4c6a-a417-892396124630" containerName="revoke-keys" Jul 17 02:04:43.877607 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.877435 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" containerName="revoke-keys" Jul 17 02:04:43.879719 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.879699 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" Jul 17 02:04:43.881860 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.881834 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 02:04:43.886176 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.886151 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6"] Jul 17 02:04:43.995696 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:43.995662 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b5aacb-4b42-43ef-baed-efe59131c4d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6\" (UID: \"84b5aacb-4b42-43ef-baed-efe59131c4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" Jul 17 02:04:44.097176 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:44.097134 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b5aacb-4b42-43ef-baed-efe59131c4d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6\" (UID: \"84b5aacb-4b42-43ef-baed-efe59131c4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" Jul 17 02:04:44.097808 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:44.097782 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b5aacb-4b42-43ef-baed-efe59131c4d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6\" (UID: \"84b5aacb-4b42-43ef-baed-efe59131c4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" Jul 17 02:04:44.192469 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:44.192370 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" Jul 17 02:04:44.326363 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:44.326282 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6"] Jul 17 02:04:44.328804 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:04:44.328777 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b5aacb_4b42_43ef_baed_efe59131c4d3.slice/crio-1f708c93d325ff5aa07ee0349d4381eed3b71b5c82c62f58931da54746bd9d03 WatchSource:0}: Error finding container 1f708c93d325ff5aa07ee0349d4381eed3b71b5c82c62f58931da54746bd9d03: Status 404 returned error can't find the container with id 1f708c93d325ff5aa07ee0349d4381eed3b71b5c82c62f58931da54746bd9d03 Jul 17 02:04:44.956216 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:44.956185 2583 generic.go:356] "Generic (PLEG): container finished" podID="84b5aacb-4b42-43ef-baed-efe59131c4d3" containerID="474933f5872d441d441441da5c4cb62bd30d64886ca8454ff7440f913fa3cf5d" exitCode=6 Jul 17 02:04:44.956617 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:44.956273 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" event={"ID":"84b5aacb-4b42-43ef-baed-efe59131c4d3","Type":"ContainerDied","Data":"474933f5872d441d441441da5c4cb62bd30d64886ca8454ff7440f913fa3cf5d"} Jul 17 02:04:44.956617 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:44.956340 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" event={"ID":"84b5aacb-4b42-43ef-baed-efe59131c4d3","Type":"ContainerStarted","Data":"1f708c93d325ff5aa07ee0349d4381eed3b71b5c82c62f58931da54746bd9d03"} Jul 17 02:04:44.956617 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:44.956473 2583 scope.go:117] "RemoveContainer" containerID="474933f5872d441d441441da5c4cb62bd30d64886ca8454ff7440f913fa3cf5d" Jul 17 02:04:45.964174 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:45.964142 2583 generic.go:356] "Generic (PLEG): container finished" podID="84b5aacb-4b42-43ef-baed-efe59131c4d3" containerID="1ecfaf917eabc99f70b4dc70f2d8e48c844fc85788604eca52623cf42bfca332" exitCode=6 Jul 17 02:04:45.964657 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:45.964190 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" event={"ID":"84b5aacb-4b42-43ef-baed-efe59131c4d3","Type":"ContainerDied","Data":"1ecfaf917eabc99f70b4dc70f2d8e48c844fc85788604eca52623cf42bfca332"} Jul 17 02:04:45.964657 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:45.964237 2583 scope.go:117] "RemoveContainer" containerID="474933f5872d441d441441da5c4cb62bd30d64886ca8454ff7440f913fa3cf5d" Jul 17 02:04:45.964657 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:45.964438 2583 scope.go:117] "RemoveContainer" containerID="1ecfaf917eabc99f70b4dc70f2d8e48c844fc85788604eca52623cf42bfca332" Jul 17 02:04:45.964657 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:45.964633 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6_odh-ai-gateway-infra(84b5aacb-4b42-43ef-baed-efe59131c4d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" podUID="84b5aacb-4b42-43ef-baed-efe59131c4d3" Jul 17 02:04:46.476639 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.476602 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j"] Jul 17 02:04:46.476968 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.476956 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" containerName="revoke-keys" Jul 17 02:04:46.477015 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.476970 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" containerName="revoke-keys" Jul 17 02:04:46.477015 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.476984 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" containerName="revoke-keys" Jul 17 02:04:46.477015 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.476990 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" containerName="revoke-keys" Jul 17 02:04:46.477123 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.477058 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="004174a0-a5f8-4c6a-a417-892396124630" containerName="revoke-keys" Jul 17 02:04:46.477123 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.477070 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc85b11a-b5ed-4c45-a3bd-cfafa9238b4b" containerName="revoke-keys" Jul 17 02:04:46.477123 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.477077 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="501ab11a-1c70-4b4f-bb90-af4e269e76bb" containerName="revoke-keys" Jul 17 02:04:46.479748 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.479729 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" Jul 17 02:04:46.487426 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.487240 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j"] Jul 17 02:04:46.619365 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.619326 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1332e401-3623-4069-8389-d44d9e5f7e18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j\" (UID: \"1332e401-3623-4069-8389-d44d9e5f7e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" Jul 17 02:04:46.720352 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.720299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1332e401-3623-4069-8389-d44d9e5f7e18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j\" (UID: \"1332e401-3623-4069-8389-d44d9e5f7e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" Jul 17 02:04:46.721203 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.721180 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1332e401-3623-4069-8389-d44d9e5f7e18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j\" (UID: \"1332e401-3623-4069-8389-d44d9e5f7e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" Jul 17 02:04:46.792089 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.792048 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" Jul 17 02:04:46.926460 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.926429 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j"] Jul 17 02:04:46.929070 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:04:46.929044 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1332e401_3623_4069_8389_d44d9e5f7e18.slice/crio-b76573b4bba39e286deb370c2c1f54fefb199a0b20990d68c8b9556eb82b8668 WatchSource:0}: Error finding container b76573b4bba39e286deb370c2c1f54fefb199a0b20990d68c8b9556eb82b8668: Status 404 returned error can't find the container with id b76573b4bba39e286deb370c2c1f54fefb199a0b20990d68c8b9556eb82b8668 Jul 17 02:04:46.977821 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.977785 2583 scope.go:117] "RemoveContainer" containerID="1ecfaf917eabc99f70b4dc70f2d8e48c844fc85788604eca52623cf42bfca332" Jul 17 02:04:46.978274 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:46.978015 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6_odh-ai-gateway-infra(84b5aacb-4b42-43ef-baed-efe59131c4d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" podUID="84b5aacb-4b42-43ef-baed-efe59131c4d3" Jul 17 02:04:46.978923 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:46.978896 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" event={"ID":"1332e401-3623-4069-8389-d44d9e5f7e18","Type":"ContainerStarted","Data":"b76573b4bba39e286deb370c2c1f54fefb199a0b20990d68c8b9556eb82b8668"} Jul 17 02:04:47.985930 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:47.985845 2583 generic.go:356] "Generic (PLEG): container finished" podID="1332e401-3623-4069-8389-d44d9e5f7e18" containerID="755f4bc7f56eb6bfa77da5512504c26a36e3d064cbef9967b2fb02578fb6aa7f" exitCode=6 Jul 17 02:04:47.985930 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:47.985920 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" event={"ID":"1332e401-3623-4069-8389-d44d9e5f7e18","Type":"ContainerDied","Data":"755f4bc7f56eb6bfa77da5512504c26a36e3d064cbef9967b2fb02578fb6aa7f"} Jul 17 02:04:47.986341 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:47.986152 2583 scope.go:117] "RemoveContainer" containerID="755f4bc7f56eb6bfa77da5512504c26a36e3d064cbef9967b2fb02578fb6aa7f" Jul 17 02:04:48.488895 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:48.488799 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh"] Jul 17 02:04:48.491235 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:48.491219 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" Jul 17 02:04:48.499867 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:48.499826 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh"] Jul 17 02:04:48.535721 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:48.535688 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6378de8-77ee-40e1-955a-83294b4307ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh\" (UID: \"f6378de8-77ee-40e1-955a-83294b4307ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" Jul 17 02:04:48.636947 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:48.636918 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6378de8-77ee-40e1-955a-83294b4307ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh\" (UID: \"f6378de8-77ee-40e1-955a-83294b4307ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" Jul 17 02:04:48.637523 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:48.637505 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6378de8-77ee-40e1-955a-83294b4307ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh\" (UID: \"f6378de8-77ee-40e1-955a-83294b4307ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" Jul 17 02:04:48.802953 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:48.802907 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" Jul 17 02:04:48.936992 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:48.936969 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh"] Jul 17 02:04:48.938798 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:04:48.938767 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6378de8_77ee_40e1_955a_83294b4307ba.slice/crio-1ebaecbcb7264be72633ccfcdce7d8a5ff4938cceb1e92070e485c53061f598c WatchSource:0}: Error finding container 1ebaecbcb7264be72633ccfcdce7d8a5ff4938cceb1e92070e485c53061f598c: Status 404 returned error can't find the container with id 1ebaecbcb7264be72633ccfcdce7d8a5ff4938cceb1e92070e485c53061f598c Jul 17 02:04:48.993438 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:48.993406 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" event={"ID":"f6378de8-77ee-40e1-955a-83294b4307ba","Type":"ContainerStarted","Data":"1ebaecbcb7264be72633ccfcdce7d8a5ff4938cceb1e92070e485c53061f598c"} Jul 17 02:04:48.995160 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:48.995131 2583 generic.go:356] "Generic (PLEG): container finished" podID="1332e401-3623-4069-8389-d44d9e5f7e18" containerID="f7bb3be2bdd18999017570793c3aad91cc248ec98a329f19762a85781d554e5a" exitCode=6 Jul 17 02:04:48.995258 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:48.995178 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" event={"ID":"1332e401-3623-4069-8389-d44d9e5f7e18","Type":"ContainerDied","Data":"f7bb3be2bdd18999017570793c3aad91cc248ec98a329f19762a85781d554e5a"} Jul 17 02:04:48.995258 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:48.995229 2583 scope.go:117] "RemoveContainer" containerID="755f4bc7f56eb6bfa77da5512504c26a36e3d064cbef9967b2fb02578fb6aa7f" Jul 17 02:04:48.995458 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:48.995437 2583 scope.go:117] "RemoveContainer" containerID="f7bb3be2bdd18999017570793c3aad91cc248ec98a329f19762a85781d554e5a" Jul 17 02:04:48.995697 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:48.995673 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j_odh-ai-gateway-infra(1332e401-3623-4069-8389-d44d9e5f7e18)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" podUID="1332e401-3623-4069-8389-d44d9e5f7e18" Jul 17 02:04:50.001950 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:50.001911 2583 generic.go:356] "Generic (PLEG): container finished" podID="f6378de8-77ee-40e1-955a-83294b4307ba" containerID="18ea768426e056dc92f5e2a3573edfef8897c2a15c27acc4825efe86b51804be" exitCode=6 Jul 17 02:04:50.002435 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:50.001996 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" event={"ID":"f6378de8-77ee-40e1-955a-83294b4307ba","Type":"ContainerDied","Data":"18ea768426e056dc92f5e2a3573edfef8897c2a15c27acc4825efe86b51804be"} Jul 17 02:04:50.002435 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:50.002275 2583 scope.go:117] "RemoveContainer" containerID="18ea768426e056dc92f5e2a3573edfef8897c2a15c27acc4825efe86b51804be" Jul 17 02:04:50.003875 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:50.003859 2583 scope.go:117] "RemoveContainer" containerID="f7bb3be2bdd18999017570793c3aad91cc248ec98a329f19762a85781d554e5a" Jul 17 02:04:50.004038 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:50.004025 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j_odh-ai-gateway-infra(1332e401-3623-4069-8389-d44d9e5f7e18)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" podUID="1332e401-3623-4069-8389-d44d9e5f7e18" Jul 17 02:04:50.439365 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:50.439327 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57"] Jul 17 02:04:50.441805 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:50.441785 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" Jul 17 02:04:50.448581 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:50.448549 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57"] Jul 17 02:04:50.553986 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:50.553951 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b59114-c67b-4b23-a9b1-cedfc7b7bc49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57\" (UID: \"b9b59114-c67b-4b23-a9b1-cedfc7b7bc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" Jul 17 02:04:50.655542 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:50.655456 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b59114-c67b-4b23-a9b1-cedfc7b7bc49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57\" (UID: \"b9b59114-c67b-4b23-a9b1-cedfc7b7bc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" Jul 17 02:04:50.656152 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:50.656127 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b59114-c67b-4b23-a9b1-cedfc7b7bc49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57\" (UID: \"b9b59114-c67b-4b23-a9b1-cedfc7b7bc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" Jul 17 02:04:50.754738 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:50.754704 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" Jul 17 02:04:50.885731 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:50.885704 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57"] Jul 17 02:04:51.010497 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:51.010454 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" event={"ID":"b9b59114-c67b-4b23-a9b1-cedfc7b7bc49","Type":"ContainerStarted","Data":"58fe5c7b1542f68e9fce980374d1b5869355ea0255f5d8d432d9cc48d2764a6e"} Jul 17 02:04:51.012407 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:51.012373 2583 generic.go:356] "Generic (PLEG): container finished" podID="f6378de8-77ee-40e1-955a-83294b4307ba" containerID="a1023b011ef8564f24a292747bbd21ea1691bbed7aed9fcd4abedbd68b86646d" exitCode=6 Jul 17 02:04:51.012541 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:51.012409 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" event={"ID":"f6378de8-77ee-40e1-955a-83294b4307ba","Type":"ContainerDied","Data":"a1023b011ef8564f24a292747bbd21ea1691bbed7aed9fcd4abedbd68b86646d"} Jul 17 02:04:51.012541 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:51.012465 2583 scope.go:117] "RemoveContainer" containerID="18ea768426e056dc92f5e2a3573edfef8897c2a15c27acc4825efe86b51804be" Jul 17 02:04:51.012713 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:51.012695 2583 scope.go:117] "RemoveContainer" containerID="a1023b011ef8564f24a292747bbd21ea1691bbed7aed9fcd4abedbd68b86646d" Jul 17 02:04:51.012985 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:51.012958 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh_odh-ai-gateway-infra(f6378de8-77ee-40e1-955a-83294b4307ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" podUID="f6378de8-77ee-40e1-955a-83294b4307ba" Jul 17 02:04:52.019559 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:52.019529 2583 scope.go:117] "RemoveContainer" containerID="a1023b011ef8564f24a292747bbd21ea1691bbed7aed9fcd4abedbd68b86646d" Jul 17 02:04:52.020021 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:52.019772 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh_odh-ai-gateway-infra(f6378de8-77ee-40e1-955a-83294b4307ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" podUID="f6378de8-77ee-40e1-955a-83294b4307ba" Jul 17 02:04:52.020949 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:52.020926 2583 generic.go:356] "Generic (PLEG): container finished" podID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" containerID="528ae2e67bd39c43c9e6244f9a2286f02d73d7d59b36a7a08d246e7416d93ed7" exitCode=6 Jul 17 02:04:52.021038 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:52.021010 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" event={"ID":"b9b59114-c67b-4b23-a9b1-cedfc7b7bc49","Type":"ContainerDied","Data":"528ae2e67bd39c43c9e6244f9a2286f02d73d7d59b36a7a08d246e7416d93ed7"} Jul 17 02:04:52.021208 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:52.021192 2583 scope.go:117] "RemoveContainer" containerID="528ae2e67bd39c43c9e6244f9a2286f02d73d7d59b36a7a08d246e7416d93ed7" Jul 17 02:04:53.027805 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:53.027695 2583 generic.go:356] "Generic (PLEG): container finished" podID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" containerID="296f64d1407c26ada3b016e3f3025ef7d2038eb06e8bffbc7f27d763281b0839" exitCode=6 Jul 17 02:04:53.027805 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:53.027780 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" event={"ID":"b9b59114-c67b-4b23-a9b1-cedfc7b7bc49","Type":"ContainerDied","Data":"296f64d1407c26ada3b016e3f3025ef7d2038eb06e8bffbc7f27d763281b0839"} Jul 17 02:04:53.028297 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:53.027843 2583 scope.go:117] "RemoveContainer" containerID="528ae2e67bd39c43c9e6244f9a2286f02d73d7d59b36a7a08d246e7416d93ed7" Jul 17 02:04:53.028297 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:53.028032 2583 scope.go:117] "RemoveContainer" containerID="296f64d1407c26ada3b016e3f3025ef7d2038eb06e8bffbc7f27d763281b0839" Jul 17 02:04:53.028401 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:53.028314 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57_odh-ai-gateway-infra(b9b59114-c67b-4b23-a9b1-cedfc7b7bc49)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" podUID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" Jul 17 02:04:53.470499 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:53.470458 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:04:54.036586 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:54.036563 2583 scope.go:117] "RemoveContainer" containerID="296f64d1407c26ada3b016e3f3025ef7d2038eb06e8bffbc7f27d763281b0839" Jul 17 02:04:54.036941 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:04:54.036732 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57_odh-ai-gateway-infra(b9b59114-c67b-4b23-a9b1-cedfc7b7bc49)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" podUID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" Jul 17 02:04:54.373071 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:04:54.372982 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:05:00.163179 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:00.163134 2583 scope.go:117] "RemoveContainer" containerID="1ecfaf917eabc99f70b4dc70f2d8e48c844fc85788604eca52623cf42bfca332" Jul 17 02:05:01.075040 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:01.074959 2583 generic.go:356] "Generic (PLEG): container finished" podID="84b5aacb-4b42-43ef-baed-efe59131c4d3" containerID="c4c41db508fb9f1fbfd030c655f8b9455b546e2cc108a374907ecb1283c31b5c" exitCode=6 Jul 17 02:05:01.075040 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:01.075008 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" event={"ID":"84b5aacb-4b42-43ef-baed-efe59131c4d3","Type":"ContainerDied","Data":"c4c41db508fb9f1fbfd030c655f8b9455b546e2cc108a374907ecb1283c31b5c"} Jul 17 02:05:01.075296 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:01.075061 2583 scope.go:117] "RemoveContainer" containerID="1ecfaf917eabc99f70b4dc70f2d8e48c844fc85788604eca52623cf42bfca332" Jul 17 02:05:01.075395 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:01.075376 2583 scope.go:117] "RemoveContainer" containerID="c4c41db508fb9f1fbfd030c655f8b9455b546e2cc108a374907ecb1283c31b5c" Jul 17 02:05:01.075641 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:01.075613 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6_odh-ai-gateway-infra(84b5aacb-4b42-43ef-baed-efe59131c4d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" podUID="84b5aacb-4b42-43ef-baed-efe59131c4d3" Jul 17 02:05:02.105133 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:02.105080 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6"] Jul 17 02:05:02.239508 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:02.239479 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" Jul 17 02:05:02.358449 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:02.358364 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b5aacb-4b42-43ef-baed-efe59131c4d3-maas-api-service-ca\") pod \"84b5aacb-4b42-43ef-baed-efe59131c4d3\" (UID: \"84b5aacb-4b42-43ef-baed-efe59131c4d3\") " Jul 17 02:05:02.358758 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:02.358731 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b5aacb-4b42-43ef-baed-efe59131c4d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "84b5aacb-4b42-43ef-baed-efe59131c4d3" (UID: "84b5aacb-4b42-43ef-baed-efe59131c4d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:05:02.459195 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:02.459155 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b5aacb-4b42-43ef-baed-efe59131c4d3-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:05:03.095303 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:03.095265 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" event={"ID":"84b5aacb-4b42-43ef-baed-efe59131c4d3","Type":"ContainerDied","Data":"1f708c93d325ff5aa07ee0349d4381eed3b71b5c82c62f58931da54746bd9d03"} Jul 17 02:05:03.095486 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:03.095315 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6" Jul 17 02:05:03.095486 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:03.095328 2583 scope.go:117] "RemoveContainer" containerID="c4c41db508fb9f1fbfd030c655f8b9455b546e2cc108a374907ecb1283c31b5c" Jul 17 02:05:03.118183 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:03.118149 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6"] Jul 17 02:05:03.121821 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:03.121797 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-cz4p6"] Jul 17 02:05:03.166682 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:03.166658 2583 scope.go:117] "RemoveContainer" containerID="f7bb3be2bdd18999017570793c3aad91cc248ec98a329f19762a85781d554e5a" Jul 17 02:05:03.166872 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:03.166842 2583 scope.go:117] "RemoveContainer" containerID="a1023b011ef8564f24a292747bbd21ea1691bbed7aed9fcd4abedbd68b86646d" Jul 17 02:05:03.170227 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:03.170206 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84b5aacb-4b42-43ef-baed-efe59131c4d3" path="/var/lib/kubelet/pods/84b5aacb-4b42-43ef-baed-efe59131c4d3/volumes" Jul 17 02:05:04.102547 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:04.102513 2583 generic.go:356] "Generic (PLEG): container finished" podID="1332e401-3623-4069-8389-d44d9e5f7e18" containerID="39966f2ae25ee4b6dea257f870b84418c32dbf02da241794a1712d286bee7e13" exitCode=6 Jul 17 02:05:04.102547 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:04.102553 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" event={"ID":"1332e401-3623-4069-8389-d44d9e5f7e18","Type":"ContainerDied","Data":"39966f2ae25ee4b6dea257f870b84418c32dbf02da241794a1712d286bee7e13"} Jul 17 02:05:04.102817 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:04.102620 2583 scope.go:117] "RemoveContainer" containerID="f7bb3be2bdd18999017570793c3aad91cc248ec98a329f19762a85781d554e5a" Jul 17 02:05:04.102938 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:04.102920 2583 scope.go:117] "RemoveContainer" containerID="39966f2ae25ee4b6dea257f870b84418c32dbf02da241794a1712d286bee7e13" Jul 17 02:05:04.103206 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:04.103184 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j_odh-ai-gateway-infra(1332e401-3623-4069-8389-d44d9e5f7e18)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" podUID="1332e401-3623-4069-8389-d44d9e5f7e18" Jul 17 02:05:04.104604 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:04.104583 2583 generic.go:356] "Generic (PLEG): container finished" podID="f6378de8-77ee-40e1-955a-83294b4307ba" containerID="fbb50c9518bee0cdef23343115d21c1f18bfe43b0b1adc129aaccd91db828d30" exitCode=6 Jul 17 02:05:04.104714 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:04.104647 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" event={"ID":"f6378de8-77ee-40e1-955a-83294b4307ba","Type":"ContainerDied","Data":"fbb50c9518bee0cdef23343115d21c1f18bfe43b0b1adc129aaccd91db828d30"} Jul 17 02:05:04.104951 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:04.104937 2583 scope.go:117] "RemoveContainer" containerID="fbb50c9518bee0cdef23343115d21c1f18bfe43b0b1adc129aaccd91db828d30" Jul 17 02:05:04.105128 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:04.105092 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh_odh-ai-gateway-infra(f6378de8-77ee-40e1-955a-83294b4307ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" podUID="f6378de8-77ee-40e1-955a-83294b4307ba" Jul 17 02:05:04.112943 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:04.112927 2583 scope.go:117] "RemoveContainer" containerID="a1023b011ef8564f24a292747bbd21ea1691bbed7aed9fcd4abedbd68b86646d" Jul 17 02:05:05.129255 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:05.129216 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j"] Jul 17 02:05:05.142811 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:05.142773 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh"] Jul 17 02:05:05.276407 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:05.276379 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" Jul 17 02:05:05.312379 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:05.312351 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" Jul 17 02:05:05.385251 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:05.385156 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1332e401-3623-4069-8389-d44d9e5f7e18-maas-api-service-ca\") pod \"1332e401-3623-4069-8389-d44d9e5f7e18\" (UID: \"1332e401-3623-4069-8389-d44d9e5f7e18\") " Jul 17 02:05:05.385549 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:05.385522 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1332e401-3623-4069-8389-d44d9e5f7e18-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1332e401-3623-4069-8389-d44d9e5f7e18" (UID: "1332e401-3623-4069-8389-d44d9e5f7e18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:05:05.486332 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:05.486294 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6378de8-77ee-40e1-955a-83294b4307ba-maas-api-service-ca\") pod \"f6378de8-77ee-40e1-955a-83294b4307ba\" (UID: \"f6378de8-77ee-40e1-955a-83294b4307ba\") " Jul 17 02:05:05.486556 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:05.486537 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1332e401-3623-4069-8389-d44d9e5f7e18-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:05:05.486661 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:05.486635 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6378de8-77ee-40e1-955a-83294b4307ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6378de8-77ee-40e1-955a-83294b4307ba" (UID: "f6378de8-77ee-40e1-955a-83294b4307ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:05:05.587371 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:05.587336 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6378de8-77ee-40e1-955a-83294b4307ba-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:05:06.119459 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:06.119425 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" Jul 17 02:05:06.119459 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:06.119451 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j" event={"ID":"1332e401-3623-4069-8389-d44d9e5f7e18","Type":"ContainerDied","Data":"b76573b4bba39e286deb370c2c1f54fefb199a0b20990d68c8b9556eb82b8668"} Jul 17 02:05:06.119703 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:06.119550 2583 scope.go:117] "RemoveContainer" containerID="39966f2ae25ee4b6dea257f870b84418c32dbf02da241794a1712d286bee7e13" Jul 17 02:05:06.123838 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:06.123806 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" event={"ID":"f6378de8-77ee-40e1-955a-83294b4307ba","Type":"ContainerDied","Data":"1ebaecbcb7264be72633ccfcdce7d8a5ff4938cceb1e92070e485c53061f598c"} Jul 17 02:05:06.123838 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:06.123830 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh" Jul 17 02:05:06.131941 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:06.131779 2583 scope.go:117] "RemoveContainer" containerID="fbb50c9518bee0cdef23343115d21c1f18bfe43b0b1adc129aaccd91db828d30" Jul 17 02:05:06.145412 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:06.145387 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j"] Jul 17 02:05:06.150915 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:06.150890 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-n5j7j"] Jul 17 02:05:06.160858 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:06.160826 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh"] Jul 17 02:05:06.162685 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:06.162502 2583 scope.go:117] "RemoveContainer" containerID="296f64d1407c26ada3b016e3f3025ef7d2038eb06e8bffbc7f27d763281b0839" Jul 17 02:05:06.164421 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:06.164399 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cnzxh"] Jul 17 02:05:07.132709 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:07.132625 2583 generic.go:356] "Generic (PLEG): container finished" podID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" containerID="3afff19237750a4a09c571c6a8d6ee5292632c37200ffcffbd88837f416bd4c7" exitCode=6 Jul 17 02:05:07.132709 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:07.132675 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" event={"ID":"b9b59114-c67b-4b23-a9b1-cedfc7b7bc49","Type":"ContainerDied","Data":"3afff19237750a4a09c571c6a8d6ee5292632c37200ffcffbd88837f416bd4c7"} Jul 17 02:05:07.133188 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:07.132722 2583 scope.go:117] "RemoveContainer" containerID="296f64d1407c26ada3b016e3f3025ef7d2038eb06e8bffbc7f27d763281b0839" Jul 17 02:05:07.133188 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:07.133009 2583 scope.go:117] "RemoveContainer" containerID="3afff19237750a4a09c571c6a8d6ee5292632c37200ffcffbd88837f416bd4c7" Jul 17 02:05:07.133309 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:07.133286 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57_odh-ai-gateway-infra(b9b59114-c67b-4b23-a9b1-cedfc7b7bc49)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" podUID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" Jul 17 02:05:07.167975 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:07.167938 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1332e401-3623-4069-8389-d44d9e5f7e18" path="/var/lib/kubelet/pods/1332e401-3623-4069-8389-d44d9e5f7e18/volumes" Jul 17 02:05:07.168310 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:07.168297 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6378de8-77ee-40e1-955a-83294b4307ba" path="/var/lib/kubelet/pods/f6378de8-77ee-40e1-955a-83294b4307ba/volumes" Jul 17 02:05:08.158152 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:08.158117 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57"] Jul 17 02:05:08.298469 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:08.298447 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" Jul 17 02:05:08.412814 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:08.412732 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b59114-c67b-4b23-a9b1-cedfc7b7bc49-maas-api-service-ca\") pod \"b9b59114-c67b-4b23-a9b1-cedfc7b7bc49\" (UID: \"b9b59114-c67b-4b23-a9b1-cedfc7b7bc49\") " Jul 17 02:05:08.413132 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:08.413089 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b59114-c67b-4b23-a9b1-cedfc7b7bc49-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" (UID: "b9b59114-c67b-4b23-a9b1-cedfc7b7bc49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:05:08.513843 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:08.513806 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b59114-c67b-4b23-a9b1-cedfc7b7bc49-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:05:09.146089 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:09.146057 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" Jul 17 02:05:09.146308 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:09.146062 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57" event={"ID":"b9b59114-c67b-4b23-a9b1-cedfc7b7bc49","Type":"ContainerDied","Data":"58fe5c7b1542f68e9fce980374d1b5869355ea0255f5d8d432d9cc48d2764a6e"} Jul 17 02:05:09.146308 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:09.146169 2583 scope.go:117] "RemoveContainer" containerID="3afff19237750a4a09c571c6a8d6ee5292632c37200ffcffbd88837f416bd4c7" Jul 17 02:05:09.167890 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:09.167863 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57"] Jul 17 02:05:09.173575 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:09.173550 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-cfr57"] Jul 17 02:05:11.168114 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:11.168070 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" path="/var/lib/kubelet/pods/b9b59114-c67b-4b23-a9b1-cedfc7b7bc49/volumes" Jul 17 02:05:14.513564 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:14.513521 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:05:14.563548 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:14.563518 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:05:16.197424 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:16.197390 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:05:33.908798 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.908769 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf"] Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909124 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909135 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909145 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1332e401-3623-4069-8389-d44d9e5f7e18" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909151 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1332e401-3623-4069-8389-d44d9e5f7e18" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909158 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84b5aacb-4b42-43ef-baed-efe59131c4d3" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909163 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="84b5aacb-4b42-43ef-baed-efe59131c4d3" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909169 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6378de8-77ee-40e1-955a-83294b4307ba" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909174 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6378de8-77ee-40e1-955a-83294b4307ba" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909182 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6378de8-77ee-40e1-955a-83294b4307ba" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909188 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6378de8-77ee-40e1-955a-83294b4307ba" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909195 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909200 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909209 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84b5aacb-4b42-43ef-baed-efe59131c4d3" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909214 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="84b5aacb-4b42-43ef-baed-efe59131c4d3" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909223 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84b5aacb-4b42-43ef-baed-efe59131c4d3" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909228 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="84b5aacb-4b42-43ef-baed-efe59131c4d3" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909234 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1332e401-3623-4069-8389-d44d9e5f7e18" containerName="revoke-keys" Jul 17 02:05:33.909242 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909240 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1332e401-3623-4069-8389-d44d9e5f7e18" containerName="revoke-keys" Jul 17 02:05:33.909827 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909289 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" containerName="revoke-keys" Jul 17 02:05:33.909827 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909295 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="84b5aacb-4b42-43ef-baed-efe59131c4d3" containerName="revoke-keys" Jul 17 02:05:33.909827 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909301 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1332e401-3623-4069-8389-d44d9e5f7e18" containerName="revoke-keys" Jul 17 02:05:33.909827 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909307 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6378de8-77ee-40e1-955a-83294b4307ba" containerName="revoke-keys" Jul 17 02:05:33.909827 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909312 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="84b5aacb-4b42-43ef-baed-efe59131c4d3" containerName="revoke-keys" Jul 17 02:05:33.909827 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909317 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" containerName="revoke-keys" Jul 17 02:05:33.909827 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909322 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="84b5aacb-4b42-43ef-baed-efe59131c4d3" containerName="revoke-keys" Jul 17 02:05:33.909827 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909329 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1332e401-3623-4069-8389-d44d9e5f7e18" containerName="revoke-keys" Jul 17 02:05:33.909827 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.909333 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6378de8-77ee-40e1-955a-83294b4307ba" containerName="revoke-keys" Jul 17 02:05:33.911379 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.911363 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" Jul 17 02:05:33.913600 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.913572 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 02:05:33.918954 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:33.918573 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf"] Jul 17 02:05:34.024992 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:34.024946 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b803bc67-2daa-4dad-99a7-944d4d6a1015-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf\" (UID: \"b803bc67-2daa-4dad-99a7-944d4d6a1015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" Jul 17 02:05:34.125470 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:34.125429 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b803bc67-2daa-4dad-99a7-944d4d6a1015-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf\" (UID: \"b803bc67-2daa-4dad-99a7-944d4d6a1015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" Jul 17 02:05:34.126055 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:34.126036 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b803bc67-2daa-4dad-99a7-944d4d6a1015-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf\" (UID: \"b803bc67-2daa-4dad-99a7-944d4d6a1015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" Jul 17 02:05:34.222685 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:34.222603 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" Jul 17 02:05:34.361541 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:34.361518 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf"] Jul 17 02:05:34.363814 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:05:34.363788 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb803bc67_2daa_4dad_99a7_944d4d6a1015.slice/crio-e32068518b3f14da6a25afca6b552ae62473bb2a97012b236e01591a4c2f2b26 WatchSource:0}: Error finding container e32068518b3f14da6a25afca6b552ae62473bb2a97012b236e01591a4c2f2b26: Status 404 returned error can't find the container with id e32068518b3f14da6a25afca6b552ae62473bb2a97012b236e01591a4c2f2b26 Jul 17 02:05:35.278723 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:35.278632 2583 generic.go:356] "Generic (PLEG): container finished" podID="b803bc67-2daa-4dad-99a7-944d4d6a1015" containerID="d6780132673177ea4ef3e2a3aef4fb0b2f7035808779a3179899cb9764633987" exitCode=6 Jul 17 02:05:35.279204 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:35.278721 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" event={"ID":"b803bc67-2daa-4dad-99a7-944d4d6a1015","Type":"ContainerDied","Data":"d6780132673177ea4ef3e2a3aef4fb0b2f7035808779a3179899cb9764633987"} Jul 17 02:05:35.279204 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:35.278766 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" event={"ID":"b803bc67-2daa-4dad-99a7-944d4d6a1015","Type":"ContainerStarted","Data":"e32068518b3f14da6a25afca6b552ae62473bb2a97012b236e01591a4c2f2b26"} Jul 17 02:05:35.279204 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:35.278870 2583 scope.go:117] "RemoveContainer" containerID="d6780132673177ea4ef3e2a3aef4fb0b2f7035808779a3179899cb9764633987" Jul 17 02:05:36.285437 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.285402 2583 generic.go:356] "Generic (PLEG): container finished" podID="b803bc67-2daa-4dad-99a7-944d4d6a1015" containerID="5a68f977821d279376634bf2ef837ee0a8ac20df98a1335d08c2f449682c653d" exitCode=6 Jul 17 02:05:36.285932 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.285486 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" event={"ID":"b803bc67-2daa-4dad-99a7-944d4d6a1015","Type":"ContainerDied","Data":"5a68f977821d279376634bf2ef837ee0a8ac20df98a1335d08c2f449682c653d"} Jul 17 02:05:36.285932 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.285548 2583 scope.go:117] "RemoveContainer" containerID="d6780132673177ea4ef3e2a3aef4fb0b2f7035808779a3179899cb9764633987" Jul 17 02:05:36.285932 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.285753 2583 scope.go:117] "RemoveContainer" containerID="5a68f977821d279376634bf2ef837ee0a8ac20df98a1335d08c2f449682c653d" Jul 17 02:05:36.286089 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:36.286014 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf_odh-ai-gateway-infra(b803bc67-2daa-4dad-99a7-944d4d6a1015)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" podUID="b803bc67-2daa-4dad-99a7-944d4d6a1015" Jul 17 02:05:36.518158 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.518115 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw"] Jul 17 02:05:36.518546 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.518527 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6378de8-77ee-40e1-955a-83294b4307ba" containerName="revoke-keys" Jul 17 02:05:36.518634 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.518550 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6378de8-77ee-40e1-955a-83294b4307ba" containerName="revoke-keys" Jul 17 02:05:36.518634 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.518562 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" containerName="revoke-keys" Jul 17 02:05:36.518634 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.518569 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" containerName="revoke-keys" Jul 17 02:05:36.518634 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.518595 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1332e401-3623-4069-8389-d44d9e5f7e18" containerName="revoke-keys" Jul 17 02:05:36.518634 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.518605 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1332e401-3623-4069-8389-d44d9e5f7e18" containerName="revoke-keys" Jul 17 02:05:36.521577 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.519174 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b59114-c67b-4b23-a9b1-cedfc7b7bc49" containerName="revoke-keys" Jul 17 02:05:36.521577 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.519224 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1332e401-3623-4069-8389-d44d9e5f7e18" containerName="revoke-keys" Jul 17 02:05:36.521577 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.519237 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6378de8-77ee-40e1-955a-83294b4307ba" containerName="revoke-keys" Jul 17 02:05:36.526605 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.526580 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" Jul 17 02:05:36.528435 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.528409 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw"] Jul 17 02:05:36.645279 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.645168 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8905e90-fa77-46a0-88d7-77deb7150009-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw\" (UID: \"e8905e90-fa77-46a0-88d7-77deb7150009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" Jul 17 02:05:36.746185 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.746146 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8905e90-fa77-46a0-88d7-77deb7150009-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw\" (UID: \"e8905e90-fa77-46a0-88d7-77deb7150009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" Jul 17 02:05:36.746976 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.746951 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8905e90-fa77-46a0-88d7-77deb7150009-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw\" (UID: \"e8905e90-fa77-46a0-88d7-77deb7150009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" Jul 17 02:05:36.837659 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.837618 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" Jul 17 02:05:36.975652 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:36.975628 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw"] Jul 17 02:05:36.977929 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:05:36.977905 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8905e90_fa77_46a0_88d7_77deb7150009.slice/crio-9bc16edcd81a30e4054a92aadbb537f4953933d1b1cb432fb6baa094b92d724f WatchSource:0}: Error finding container 9bc16edcd81a30e4054a92aadbb537f4953933d1b1cb432fb6baa094b92d724f: Status 404 returned error can't find the container with id 9bc16edcd81a30e4054a92aadbb537f4953933d1b1cb432fb6baa094b92d724f Jul 17 02:05:37.293021 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:37.292993 2583 scope.go:117] "RemoveContainer" containerID="5a68f977821d279376634bf2ef837ee0a8ac20df98a1335d08c2f449682c653d" Jul 17 02:05:37.293472 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:37.293291 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf_odh-ai-gateway-infra(b803bc67-2daa-4dad-99a7-944d4d6a1015)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" podUID="b803bc67-2daa-4dad-99a7-944d4d6a1015" Jul 17 02:05:37.294273 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:37.294249 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" event={"ID":"e8905e90-fa77-46a0-88d7-77deb7150009","Type":"ContainerStarted","Data":"06d760984597b21f03c42df03ccd5ba57ecffbeae779f1827e1324663d2a5e0a"} Jul 17 02:05:37.294392 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:37.294295 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" event={"ID":"e8905e90-fa77-46a0-88d7-77deb7150009","Type":"ContainerStarted","Data":"9bc16edcd81a30e4054a92aadbb537f4953933d1b1cb432fb6baa094b92d724f"} Jul 17 02:05:37.323305 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:37.323250 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" podStartSLOduration=1.3232318969999999 podStartE2EDuration="1.323231897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:05:37.322130223 +0000 UTC m=+3094.717745077" watchObservedRunningTime="2026-07-17 02:05:37.323231897 +0000 UTC m=+3094.718846744" Jul 17 02:05:38.301587 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:38.301554 2583 generic.go:356] "Generic (PLEG): container finished" podID="e8905e90-fa77-46a0-88d7-77deb7150009" containerID="06d760984597b21f03c42df03ccd5ba57ecffbeae779f1827e1324663d2a5e0a" exitCode=6 Jul 17 02:05:38.301990 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:38.301647 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" event={"ID":"e8905e90-fa77-46a0-88d7-77deb7150009","Type":"ContainerDied","Data":"06d760984597b21f03c42df03ccd5ba57ecffbeae779f1827e1324663d2a5e0a"} Jul 17 02:05:38.301990 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:38.301792 2583 scope.go:117] "RemoveContainer" containerID="06d760984597b21f03c42df03ccd5ba57ecffbeae779f1827e1324663d2a5e0a" Jul 17 02:05:38.538217 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:38.538182 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s"] Jul 17 02:05:38.541691 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:38.541675 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" Jul 17 02:05:38.548170 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:38.548144 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s"] Jul 17 02:05:38.660435 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:38.660394 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb625b5-f71f-4b23-8aef-52df1ecc3576-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s\" (UID: \"9fb625b5-f71f-4b23-8aef-52df1ecc3576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" Jul 17 02:05:38.761258 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:38.761222 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb625b5-f71f-4b23-8aef-52df1ecc3576-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s\" (UID: \"9fb625b5-f71f-4b23-8aef-52df1ecc3576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" Jul 17 02:05:38.761860 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:38.761840 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb625b5-f71f-4b23-8aef-52df1ecc3576-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s\" (UID: \"9fb625b5-f71f-4b23-8aef-52df1ecc3576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" Jul 17 02:05:38.853259 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:38.853220 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" Jul 17 02:05:38.988830 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:38.988801 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s"] Jul 17 02:05:38.990524 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:05:38.990500 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb625b5_f71f_4b23_8aef_52df1ecc3576.slice/crio-8f5140441c51e34d75b8fdf184c44113326b54d87cd37f9462fd05b8afd198fa WatchSource:0}: Error finding container 8f5140441c51e34d75b8fdf184c44113326b54d87cd37f9462fd05b8afd198fa: Status 404 returned error can't find the container with id 8f5140441c51e34d75b8fdf184c44113326b54d87cd37f9462fd05b8afd198fa Jul 17 02:05:39.308423 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:39.308386 2583 generic.go:356] "Generic (PLEG): container finished" podID="e8905e90-fa77-46a0-88d7-77deb7150009" containerID="24958bf5557470a6add4211645112690c32605e40cf5521accdf37e3a1d16faf" exitCode=6 Jul 17 02:05:39.308872 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:39.308431 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" event={"ID":"e8905e90-fa77-46a0-88d7-77deb7150009","Type":"ContainerDied","Data":"24958bf5557470a6add4211645112690c32605e40cf5521accdf37e3a1d16faf"} Jul 17 02:05:39.308872 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:39.308501 2583 scope.go:117] "RemoveContainer" containerID="06d760984597b21f03c42df03ccd5ba57ecffbeae779f1827e1324663d2a5e0a" Jul 17 02:05:39.308872 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:39.308681 2583 scope.go:117] "RemoveContainer" containerID="24958bf5557470a6add4211645112690c32605e40cf5521accdf37e3a1d16faf" Jul 17 02:05:39.309006 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:39.308948 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw_odh-ai-gateway-infra(e8905e90-fa77-46a0-88d7-77deb7150009)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" podUID="e8905e90-fa77-46a0-88d7-77deb7150009" Jul 17 02:05:39.309876 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:39.309851 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" event={"ID":"9fb625b5-f71f-4b23-8aef-52df1ecc3576","Type":"ContainerStarted","Data":"8f5140441c51e34d75b8fdf184c44113326b54d87cd37f9462fd05b8afd198fa"} Jul 17 02:05:40.316815 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:40.316782 2583 scope.go:117] "RemoveContainer" containerID="24958bf5557470a6add4211645112690c32605e40cf5521accdf37e3a1d16faf" Jul 17 02:05:40.317267 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:40.317001 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw_odh-ai-gateway-infra(e8905e90-fa77-46a0-88d7-77deb7150009)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" podUID="e8905e90-fa77-46a0-88d7-77deb7150009" Jul 17 02:05:40.318006 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:40.317979 2583 generic.go:356] "Generic (PLEG): container finished" podID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" containerID="ea27ad1150933885f493123c0ff08a59c19f4b678b90e7f8b16e50f305042aa2" exitCode=6 Jul 17 02:05:40.318119 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:40.318034 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" event={"ID":"9fb625b5-f71f-4b23-8aef-52df1ecc3576","Type":"ContainerDied","Data":"ea27ad1150933885f493123c0ff08a59c19f4b678b90e7f8b16e50f305042aa2"} Jul 17 02:05:40.318246 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:40.318192 2583 scope.go:117] "RemoveContainer" containerID="ea27ad1150933885f493123c0ff08a59c19f4b678b90e7f8b16e50f305042aa2" Jul 17 02:05:40.481074 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:40.481040 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5"] Jul 17 02:05:40.484458 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:40.484440 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" Jul 17 02:05:40.491748 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:40.491720 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5"] Jul 17 02:05:40.579494 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:40.579386 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ee980c-2849-445f-aaa2-8620633bffaf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5\" (UID: \"32ee980c-2849-445f-aaa2-8620633bffaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" Jul 17 02:05:40.680662 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:40.680626 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ee980c-2849-445f-aaa2-8620633bffaf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5\" (UID: \"32ee980c-2849-445f-aaa2-8620633bffaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" Jul 17 02:05:40.681295 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:40.681273 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ee980c-2849-445f-aaa2-8620633bffaf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5\" (UID: \"32ee980c-2849-445f-aaa2-8620633bffaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" Jul 17 02:05:40.795595 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:40.795562 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" Jul 17 02:05:40.928315 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:40.928277 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5"] Jul 17 02:05:40.929872 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:05:40.929844 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ee980c_2849_445f_aaa2_8620633bffaf.slice/crio-125b4a5f40482454cfef7a75e685d6e1e18a6218e377492a9451706aa127185e WatchSource:0}: Error finding container 125b4a5f40482454cfef7a75e685d6e1e18a6218e377492a9451706aa127185e: Status 404 returned error can't find the container with id 125b4a5f40482454cfef7a75e685d6e1e18a6218e377492a9451706aa127185e Jul 17 02:05:41.325306 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:41.325267 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" event={"ID":"32ee980c-2849-445f-aaa2-8620633bffaf","Type":"ContainerStarted","Data":"c8507d42779aaf925999a4cadb74e531dd03f1d0b1162880e2b7864e16f5d6dc"} Jul 17 02:05:41.325797 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:41.325333 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" event={"ID":"32ee980c-2849-445f-aaa2-8620633bffaf","Type":"ContainerStarted","Data":"125b4a5f40482454cfef7a75e685d6e1e18a6218e377492a9451706aa127185e"} Jul 17 02:05:41.327178 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:41.327153 2583 generic.go:356] "Generic (PLEG): container finished" podID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" containerID="be128bb60d3095d1ab946efaa6ee775029599b4fc737e6f022f021aecce20645" exitCode=6 Jul 17 02:05:41.327291 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:41.327196 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" event={"ID":"9fb625b5-f71f-4b23-8aef-52df1ecc3576","Type":"ContainerDied","Data":"be128bb60d3095d1ab946efaa6ee775029599b4fc737e6f022f021aecce20645"} Jul 17 02:05:41.327291 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:41.327233 2583 scope.go:117] "RemoveContainer" containerID="ea27ad1150933885f493123c0ff08a59c19f4b678b90e7f8b16e50f305042aa2" Jul 17 02:05:41.327412 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:41.327397 2583 scope.go:117] "RemoveContainer" containerID="be128bb60d3095d1ab946efaa6ee775029599b4fc737e6f022f021aecce20645" Jul 17 02:05:41.327602 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:41.327579 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s_odh-ai-gateway-infra(9fb625b5-f71f-4b23-8aef-52df1ecc3576)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" podUID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" Jul 17 02:05:41.342047 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:41.341999 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" podStartSLOduration=1.34198191 podStartE2EDuration="1.34198191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:05:41.339582167 +0000 UTC m=+3098.735197019" watchObservedRunningTime="2026-07-17 02:05:41.34198191 +0000 UTC m=+3098.737596761" Jul 17 02:05:41.707603 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:41.707569 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:05:42.334738 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:42.334703 2583 generic.go:356] "Generic (PLEG): container finished" podID="32ee980c-2849-445f-aaa2-8620633bffaf" containerID="c8507d42779aaf925999a4cadb74e531dd03f1d0b1162880e2b7864e16f5d6dc" exitCode=6 Jul 17 02:05:42.335198 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:42.334743 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" event={"ID":"32ee980c-2849-445f-aaa2-8620633bffaf","Type":"ContainerDied","Data":"c8507d42779aaf925999a4cadb74e531dd03f1d0b1162880e2b7864e16f5d6dc"} Jul 17 02:05:42.335198 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:42.334995 2583 scope.go:117] "RemoveContainer" containerID="c8507d42779aaf925999a4cadb74e531dd03f1d0b1162880e2b7864e16f5d6dc" Jul 17 02:05:42.336794 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:42.336773 2583 scope.go:117] "RemoveContainer" containerID="be128bb60d3095d1ab946efaa6ee775029599b4fc737e6f022f021aecce20645" Jul 17 02:05:42.337017 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:42.336997 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s_odh-ai-gateway-infra(9fb625b5-f71f-4b23-8aef-52df1ecc3576)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" podUID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" Jul 17 02:05:43.347342 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:43.347304 2583 generic.go:356] "Generic (PLEG): container finished" podID="32ee980c-2849-445f-aaa2-8620633bffaf" containerID="8a2bcc84c36261bc59e45d73e1aa30f73d3699d55c70d218e228ae96faf9376d" exitCode=6 Jul 17 02:05:43.347797 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:43.347390 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" event={"ID":"32ee980c-2849-445f-aaa2-8620633bffaf","Type":"ContainerDied","Data":"8a2bcc84c36261bc59e45d73e1aa30f73d3699d55c70d218e228ae96faf9376d"} Jul 17 02:05:43.347797 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:43.347455 2583 scope.go:117] "RemoveContainer" containerID="c8507d42779aaf925999a4cadb74e531dd03f1d0b1162880e2b7864e16f5d6dc" Jul 17 02:05:43.347797 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:43.347655 2583 scope.go:117] "RemoveContainer" containerID="8a2bcc84c36261bc59e45d73e1aa30f73d3699d55c70d218e228ae96faf9376d" Jul 17 02:05:43.348009 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:43.347855 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5_odh-ai-gateway-infra(32ee980c-2849-445f-aaa2-8620633bffaf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" podUID="32ee980c-2849-445f-aaa2-8620633bffaf" Jul 17 02:05:44.354840 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:44.354814 2583 scope.go:117] "RemoveContainer" containerID="8a2bcc84c36261bc59e45d73e1aa30f73d3699d55c70d218e228ae96faf9376d" Jul 17 02:05:44.355302 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:44.355034 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5_odh-ai-gateway-infra(32ee980c-2849-445f-aaa2-8620633bffaf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" podUID="32ee980c-2849-445f-aaa2-8620633bffaf" Jul 17 02:05:49.163143 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:49.163082 2583 scope.go:117] "RemoveContainer" containerID="5a68f977821d279376634bf2ef837ee0a8ac20df98a1335d08c2f449682c653d" Jul 17 02:05:49.382201 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:49.382167 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" event={"ID":"b803bc67-2daa-4dad-99a7-944d4d6a1015","Type":"ContainerStarted","Data":"06cba0a5eacc5515dc48c6e70dfe84cccbab53006d08ca01c92ec9c79578dd64"} Jul 17 02:05:49.399590 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:49.399537 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" podStartSLOduration=16.399520718 podStartE2EDuration="16.399520718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:05:49.39795352 +0000 UTC m=+3106.793568373" watchObservedRunningTime="2026-07-17 02:05:49.399520718 +0000 UTC m=+3106.795135569" Jul 17 02:05:50.390608 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:50.390575 2583 generic.go:356] "Generic (PLEG): container finished" podID="b803bc67-2daa-4dad-99a7-944d4d6a1015" containerID="06cba0a5eacc5515dc48c6e70dfe84cccbab53006d08ca01c92ec9c79578dd64" exitCode=6 Jul 17 02:05:50.391060 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:50.390643 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" event={"ID":"b803bc67-2daa-4dad-99a7-944d4d6a1015","Type":"ContainerDied","Data":"06cba0a5eacc5515dc48c6e70dfe84cccbab53006d08ca01c92ec9c79578dd64"} Jul 17 02:05:50.391060 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:50.390706 2583 scope.go:117] "RemoveContainer" containerID="5a68f977821d279376634bf2ef837ee0a8ac20df98a1335d08c2f449682c653d" Jul 17 02:05:50.391060 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:50.390990 2583 scope.go:117] "RemoveContainer" containerID="06cba0a5eacc5515dc48c6e70dfe84cccbab53006d08ca01c92ec9c79578dd64" Jul 17 02:05:50.391234 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:50.391214 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf_odh-ai-gateway-infra(b803bc67-2daa-4dad-99a7-944d4d6a1015)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" podUID="b803bc67-2daa-4dad-99a7-944d4d6a1015" Jul 17 02:05:50.413268 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:50.413239 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf"] Jul 17 02:05:51.162268 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:51.162233 2583 scope.go:117] "RemoveContainer" containerID="24958bf5557470a6add4211645112690c32605e40cf5521accdf37e3a1d16faf" Jul 17 02:05:51.532495 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:51.532469 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" Jul 17 02:05:51.681931 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:51.681837 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b803bc67-2daa-4dad-99a7-944d4d6a1015-maas-api-service-ca\") pod \"b803bc67-2daa-4dad-99a7-944d4d6a1015\" (UID: \"b803bc67-2daa-4dad-99a7-944d4d6a1015\") " Jul 17 02:05:51.682296 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:51.682261 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b803bc67-2daa-4dad-99a7-944d4d6a1015-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b803bc67-2daa-4dad-99a7-944d4d6a1015" (UID: "b803bc67-2daa-4dad-99a7-944d4d6a1015"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:05:51.783015 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:51.782980 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b803bc67-2daa-4dad-99a7-944d4d6a1015-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:05:52.404054 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:52.404018 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" Jul 17 02:05:52.404281 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:52.404063 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf" event={"ID":"b803bc67-2daa-4dad-99a7-944d4d6a1015","Type":"ContainerDied","Data":"e32068518b3f14da6a25afca6b552ae62473bb2a97012b236e01591a4c2f2b26"} Jul 17 02:05:52.404281 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:52.404164 2583 scope.go:117] "RemoveContainer" containerID="06cba0a5eacc5515dc48c6e70dfe84cccbab53006d08ca01c92ec9c79578dd64" Jul 17 02:05:52.406050 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:52.406027 2583 generic.go:356] "Generic (PLEG): container finished" podID="e8905e90-fa77-46a0-88d7-77deb7150009" containerID="191a79bac60bcad43b683087a42a0f457cb2309751a55e7f6879001d49365419" exitCode=6 Jul 17 02:05:52.406215 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:52.406062 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" event={"ID":"e8905e90-fa77-46a0-88d7-77deb7150009","Type":"ContainerDied","Data":"191a79bac60bcad43b683087a42a0f457cb2309751a55e7f6879001d49365419"} Jul 17 02:05:52.406380 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:52.406366 2583 scope.go:117] "RemoveContainer" containerID="191a79bac60bcad43b683087a42a0f457cb2309751a55e7f6879001d49365419" Jul 17 02:05:52.406571 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:52.406553 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw_odh-ai-gateway-infra(e8905e90-fa77-46a0-88d7-77deb7150009)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" podUID="e8905e90-fa77-46a0-88d7-77deb7150009" Jul 17 02:05:52.414361 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:52.414342 2583 scope.go:117] "RemoveContainer" containerID="24958bf5557470a6add4211645112690c32605e40cf5521accdf37e3a1d16faf" Jul 17 02:05:52.439224 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:52.439194 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf"] Jul 17 02:05:52.443804 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:52.443781 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-sh7vf"] Jul 17 02:05:53.168176 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:53.168132 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b803bc67-2daa-4dad-99a7-944d4d6a1015" path="/var/lib/kubelet/pods/b803bc67-2daa-4dad-99a7-944d4d6a1015/volumes" Jul 17 02:05:53.433634 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:53.433545 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw"] Jul 17 02:05:53.567240 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:53.567216 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" Jul 17 02:05:53.701022 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:53.700927 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8905e90-fa77-46a0-88d7-77deb7150009-maas-api-service-ca\") pod \"e8905e90-fa77-46a0-88d7-77deb7150009\" (UID: \"e8905e90-fa77-46a0-88d7-77deb7150009\") " Jul 17 02:05:53.701380 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:53.701351 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8905e90-fa77-46a0-88d7-77deb7150009-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8905e90-fa77-46a0-88d7-77deb7150009" (UID: "e8905e90-fa77-46a0-88d7-77deb7150009"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:05:53.802012 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:53.801973 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8905e90-fa77-46a0-88d7-77deb7150009-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:05:54.419847 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:54.419819 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" Jul 17 02:05:54.420379 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:54.419860 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw" event={"ID":"e8905e90-fa77-46a0-88d7-77deb7150009","Type":"ContainerDied","Data":"9bc16edcd81a30e4054a92aadbb537f4953933d1b1cb432fb6baa094b92d724f"} Jul 17 02:05:54.420379 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:54.419921 2583 scope.go:117] "RemoveContainer" containerID="191a79bac60bcad43b683087a42a0f457cb2309751a55e7f6879001d49365419" Jul 17 02:05:54.440702 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:54.440665 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw"] Jul 17 02:05:54.445925 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:54.445885 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-rl8rw"] Jul 17 02:05:55.162526 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:55.162493 2583 scope.go:117] "RemoveContainer" containerID="8a2bcc84c36261bc59e45d73e1aa30f73d3699d55c70d218e228ae96faf9376d" Jul 17 02:05:55.168207 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:55.168180 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8905e90-fa77-46a0-88d7-77deb7150009" path="/var/lib/kubelet/pods/e8905e90-fa77-46a0-88d7-77deb7150009/volumes" Jul 17 02:05:56.433910 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:56.433877 2583 generic.go:356] "Generic (PLEG): container finished" podID="32ee980c-2849-445f-aaa2-8620633bffaf" containerID="a384ea59c8646d70813bd8a5a8cda358de38611990b4e83b9c9c9a02068a1753" exitCode=6 Jul 17 02:05:56.434359 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:56.433953 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" event={"ID":"32ee980c-2849-445f-aaa2-8620633bffaf","Type":"ContainerDied","Data":"a384ea59c8646d70813bd8a5a8cda358de38611990b4e83b9c9c9a02068a1753"} Jul 17 02:05:56.434359 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:56.434012 2583 scope.go:117] "RemoveContainer" containerID="8a2bcc84c36261bc59e45d73e1aa30f73d3699d55c70d218e228ae96faf9376d" Jul 17 02:05:56.434359 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:56.434340 2583 scope.go:117] "RemoveContainer" containerID="a384ea59c8646d70813bd8a5a8cda358de38611990b4e83b9c9c9a02068a1753" Jul 17 02:05:56.434554 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:56.434535 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5_odh-ai-gateway-infra(32ee980c-2849-445f-aaa2-8620633bffaf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" podUID="32ee980c-2849-445f-aaa2-8620633bffaf" Jul 17 02:05:57.162477 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:57.162440 2583 scope.go:117] "RemoveContainer" containerID="be128bb60d3095d1ab946efaa6ee775029599b4fc737e6f022f021aecce20645" Jul 17 02:05:57.442676 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:57.442595 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" event={"ID":"9fb625b5-f71f-4b23-8aef-52df1ecc3576","Type":"ContainerStarted","Data":"9dbd8b622486319e3fd851d4c30c2969414ff1f0cb1956cca757f0d0a3f3dbe6"} Jul 17 02:05:57.457278 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:57.457244 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5"] Jul 17 02:05:57.458241 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:57.457991 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" podStartSLOduration=19.457978778 podStartE2EDuration="19.457978778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:05:57.457587695 +0000 UTC m=+3114.853202548" watchObservedRunningTime="2026-07-17 02:05:57.457978778 +0000 UTC m=+3114.853593641" Jul 17 02:05:57.581192 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:57.581169 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" Jul 17 02:05:57.740935 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:57.740841 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ee980c-2849-445f-aaa2-8620633bffaf-maas-api-service-ca\") pod \"32ee980c-2849-445f-aaa2-8620633bffaf\" (UID: \"32ee980c-2849-445f-aaa2-8620633bffaf\") " Jul 17 02:05:57.741276 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:57.741254 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ee980c-2849-445f-aaa2-8620633bffaf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "32ee980c-2849-445f-aaa2-8620633bffaf" (UID: "32ee980c-2849-445f-aaa2-8620633bffaf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:05:57.842526 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:57.842490 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ee980c-2849-445f-aaa2-8620633bffaf-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:05:58.448615 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:58.448588 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" Jul 17 02:05:58.448615 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:58.448595 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5" event={"ID":"32ee980c-2849-445f-aaa2-8620633bffaf","Type":"ContainerDied","Data":"125b4a5f40482454cfef7a75e685d6e1e18a6218e377492a9451706aa127185e"} Jul 17 02:05:58.449158 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:58.448663 2583 scope.go:117] "RemoveContainer" containerID="a384ea59c8646d70813bd8a5a8cda358de38611990b4e83b9c9c9a02068a1753" Jul 17 02:05:58.450757 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:58.450733 2583 generic.go:356] "Generic (PLEG): container finished" podID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" containerID="9dbd8b622486319e3fd851d4c30c2969414ff1f0cb1956cca757f0d0a3f3dbe6" exitCode=6 Jul 17 02:05:58.450881 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:58.450805 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" event={"ID":"9fb625b5-f71f-4b23-8aef-52df1ecc3576","Type":"ContainerDied","Data":"9dbd8b622486319e3fd851d4c30c2969414ff1f0cb1956cca757f0d0a3f3dbe6"} Jul 17 02:05:58.451174 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:58.451155 2583 scope.go:117] "RemoveContainer" containerID="9dbd8b622486319e3fd851d4c30c2969414ff1f0cb1956cca757f0d0a3f3dbe6" Jul 17 02:05:58.451400 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:05:58.451378 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s_odh-ai-gateway-infra(9fb625b5-f71f-4b23-8aef-52df1ecc3576)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" podUID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" Jul 17 02:05:58.457748 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:58.457729 2583 scope.go:117] "RemoveContainer" containerID="be128bb60d3095d1ab946efaa6ee775029599b4fc737e6f022f021aecce20645" Jul 17 02:05:58.465466 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:58.465442 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s"] Jul 17 02:05:58.489022 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:58.488993 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5"] Jul 17 02:05:58.496276 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:58.496253 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-bx5q5"] Jul 17 02:05:59.167851 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:59.167820 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ee980c-2849-445f-aaa2-8620633bffaf" path="/var/lib/kubelet/pods/32ee980c-2849-445f-aaa2-8620633bffaf/volumes" Jul 17 02:05:59.589595 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:59.589568 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" Jul 17 02:05:59.758735 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:59.758631 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb625b5-f71f-4b23-8aef-52df1ecc3576-maas-api-service-ca\") pod \"9fb625b5-f71f-4b23-8aef-52df1ecc3576\" (UID: \"9fb625b5-f71f-4b23-8aef-52df1ecc3576\") " Jul 17 02:05:59.759038 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:59.759013 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb625b5-f71f-4b23-8aef-52df1ecc3576-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9fb625b5-f71f-4b23-8aef-52df1ecc3576" (UID: "9fb625b5-f71f-4b23-8aef-52df1ecc3576"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:05:59.859832 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:05:59.859793 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb625b5-f71f-4b23-8aef-52df1ecc3576-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:06:00.466173 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:00.466141 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" Jul 17 02:06:00.466173 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:00.466166 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s" event={"ID":"9fb625b5-f71f-4b23-8aef-52df1ecc3576","Type":"ContainerDied","Data":"8f5140441c51e34d75b8fdf184c44113326b54d87cd37f9462fd05b8afd198fa"} Jul 17 02:06:00.466492 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:00.466237 2583 scope.go:117] "RemoveContainer" containerID="9dbd8b622486319e3fd851d4c30c2969414ff1f0cb1956cca757f0d0a3f3dbe6" Jul 17 02:06:00.490360 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:00.490332 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s"] Jul 17 02:06:00.492227 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:00.492206 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-rk52s"] Jul 17 02:06:01.167831 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:01.167800 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" path="/var/lib/kubelet/pods/9fb625b5-f71f-4b23-8aef-52df1ecc3576/volumes" Jul 17 02:06:03.615235 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:03.615205 2583 scope.go:117] "RemoveContainer" containerID="c709fbaa6755d7e8aa738314a5f9dff82ba6c5a6152db9168792bd9f5caa9c46" Jul 17 02:06:20.268245 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:20.268207 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:06:21.171276 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:21.171238 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:06:23.944635 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.944599 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7"] Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.944924 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.944935 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.944952 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ee980c-2849-445f-aaa2-8620633bffaf" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.944957 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ee980c-2849-445f-aaa2-8620633bffaf" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.944968 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ee980c-2849-445f-aaa2-8620633bffaf" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.944975 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ee980c-2849-445f-aaa2-8620633bffaf" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.944982 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b803bc67-2daa-4dad-99a7-944d4d6a1015" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.944987 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b803bc67-2daa-4dad-99a7-944d4d6a1015" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.944994 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8905e90-fa77-46a0-88d7-77deb7150009" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.944999 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8905e90-fa77-46a0-88d7-77deb7150009" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945006 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8905e90-fa77-46a0-88d7-77deb7150009" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945011 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8905e90-fa77-46a0-88d7-77deb7150009" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945018 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8905e90-fa77-46a0-88d7-77deb7150009" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945023 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8905e90-fa77-46a0-88d7-77deb7150009" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945030 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b803bc67-2daa-4dad-99a7-944d4d6a1015" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945035 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b803bc67-2daa-4dad-99a7-944d4d6a1015" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945041 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ee980c-2849-445f-aaa2-8620633bffaf" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945046 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ee980c-2849-445f-aaa2-8620633bffaf" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945053 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b803bc67-2daa-4dad-99a7-944d4d6a1015" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945057 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b803bc67-2daa-4dad-99a7-944d4d6a1015" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945064 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" containerName="revoke-keys" Jul 17 02:06:23.945076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945069 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" containerName="revoke-keys" Jul 17 02:06:23.945749 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945142 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" containerName="revoke-keys" Jul 17 02:06:23.945749 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945150 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ee980c-2849-445f-aaa2-8620633bffaf" containerName="revoke-keys" Jul 17 02:06:23.945749 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945157 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b803bc67-2daa-4dad-99a7-944d4d6a1015" containerName="revoke-keys" Jul 17 02:06:23.945749 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945162 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" containerName="revoke-keys" Jul 17 02:06:23.945749 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945168 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b803bc67-2daa-4dad-99a7-944d4d6a1015" containerName="revoke-keys" Jul 17 02:06:23.945749 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945174 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8905e90-fa77-46a0-88d7-77deb7150009" containerName="revoke-keys" Jul 17 02:06:23.945749 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945179 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ee980c-2849-445f-aaa2-8620633bffaf" containerName="revoke-keys" Jul 17 02:06:23.945749 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945184 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8905e90-fa77-46a0-88d7-77deb7150009" containerName="revoke-keys" Jul 17 02:06:23.945749 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.945188 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8905e90-fa77-46a0-88d7-77deb7150009" containerName="revoke-keys" Jul 17 02:06:23.948178 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.948160 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" Jul 17 02:06:23.950489 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.950469 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 02:06:23.955133 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:23.955081 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7"] Jul 17 02:06:24.070933 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:24.070888 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3479e111-51bb-4a05-85cf-611cddadd7d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7\" (UID: \"3479e111-51bb-4a05-85cf-611cddadd7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" Jul 17 02:06:24.171623 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:24.171581 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3479e111-51bb-4a05-85cf-611cddadd7d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7\" (UID: \"3479e111-51bb-4a05-85cf-611cddadd7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" Jul 17 02:06:24.172212 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:24.172192 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3479e111-51bb-4a05-85cf-611cddadd7d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7\" (UID: \"3479e111-51bb-4a05-85cf-611cddadd7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" Jul 17 02:06:24.260206 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:24.260117 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" Jul 17 02:06:24.396111 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:24.396072 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7"] Jul 17 02:06:24.398813 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:06:24.398771 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3479e111_51bb_4a05_85cf_611cddadd7d4.slice/crio-66cb51c4d37520e5978a1061f71f610851cb831f3b3c4291dc278ea66fc3187f WatchSource:0}: Error finding container 66cb51c4d37520e5978a1061f71f610851cb831f3b3c4291dc278ea66fc3187f: Status 404 returned error can't find the container with id 66cb51c4d37520e5978a1061f71f610851cb831f3b3c4291dc278ea66fc3187f Jul 17 02:06:24.580039 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:24.580004 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" event={"ID":"3479e111-51bb-4a05-85cf-611cddadd7d4","Type":"ContainerStarted","Data":"66cb51c4d37520e5978a1061f71f610851cb831f3b3c4291dc278ea66fc3187f"} Jul 17 02:06:25.586710 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:25.586669 2583 generic.go:356] "Generic (PLEG): container finished" podID="3479e111-51bb-4a05-85cf-611cddadd7d4" containerID="8124b106e154f95f2c255528eeb4407d34f86d8fb4c78e0fe44a2bc55dcb6f8e" exitCode=6 Jul 17 02:06:25.587253 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:25.586712 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" event={"ID":"3479e111-51bb-4a05-85cf-611cddadd7d4","Type":"ContainerDied","Data":"8124b106e154f95f2c255528eeb4407d34f86d8fb4c78e0fe44a2bc55dcb6f8e"} Jul 17 02:06:25.587253 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:25.586962 2583 scope.go:117] "RemoveContainer" containerID="8124b106e154f95f2c255528eeb4407d34f86d8fb4c78e0fe44a2bc55dcb6f8e" Jul 17 02:06:26.555531 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.555494 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c"] Jul 17 02:06:26.555852 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.555840 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" containerName="revoke-keys" Jul 17 02:06:26.555897 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.555853 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" containerName="revoke-keys" Jul 17 02:06:26.555938 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.555905 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ee980c-2849-445f-aaa2-8620633bffaf" containerName="revoke-keys" Jul 17 02:06:26.555938 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.555915 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb625b5-f71f-4b23-8aef-52df1ecc3576" containerName="revoke-keys" Jul 17 02:06:26.555938 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.555922 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b803bc67-2daa-4dad-99a7-944d4d6a1015" containerName="revoke-keys" Jul 17 02:06:26.558912 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.558896 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" Jul 17 02:06:26.566084 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.565932 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c"] Jul 17 02:06:26.593234 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.593196 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c\" (UID: \"85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" Jul 17 02:06:26.593795 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.593775 2583 generic.go:356] "Generic (PLEG): container finished" podID="3479e111-51bb-4a05-85cf-611cddadd7d4" containerID="ac11c4393bff95f09fbd871770dba12a9d37f86845bca5536ee16739bbae417f" exitCode=6 Jul 17 02:06:26.593864 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.593850 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" event={"ID":"3479e111-51bb-4a05-85cf-611cddadd7d4","Type":"ContainerDied","Data":"ac11c4393bff95f09fbd871770dba12a9d37f86845bca5536ee16739bbae417f"} Jul 17 02:06:26.593904 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.593895 2583 scope.go:117] "RemoveContainer" containerID="8124b106e154f95f2c255528eeb4407d34f86d8fb4c78e0fe44a2bc55dcb6f8e" Jul 17 02:06:26.594060 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.594037 2583 scope.go:117] "RemoveContainer" containerID="ac11c4393bff95f09fbd871770dba12a9d37f86845bca5536ee16739bbae417f" Jul 17 02:06:26.594295 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:06:26.594274 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7_odh-ai-gateway-infra(3479e111-51bb-4a05-85cf-611cddadd7d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" podUID="3479e111-51bb-4a05-85cf-611cddadd7d4" Jul 17 02:06:26.694136 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.694089 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c\" (UID: \"85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" Jul 17 02:06:26.694731 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.694712 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c\" (UID: \"85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" Jul 17 02:06:26.870671 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:26.870582 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" Jul 17 02:06:27.004753 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:27.004720 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c"] Jul 17 02:06:27.007837 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:06:27.007807 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85fdc7bc_84ed_4dee_bec5_a7d0e3fd05cb.slice/crio-355c08c220ab5b99bb8b6396f2a4285675e04103d559afc877918a127babb320 WatchSource:0}: Error finding container 355c08c220ab5b99bb8b6396f2a4285675e04103d559afc877918a127babb320: Status 404 returned error can't find the container with id 355c08c220ab5b99bb8b6396f2a4285675e04103d559afc877918a127babb320 Jul 17 02:06:27.600562 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:27.600529 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" event={"ID":"85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb","Type":"ContainerStarted","Data":"e47eb64d91baf3b685db3c0e6036504076c7d8f2ebfe922724f9317e731b91c4"} Jul 17 02:06:27.601022 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:27.600589 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" event={"ID":"85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb","Type":"ContainerStarted","Data":"355c08c220ab5b99bb8b6396f2a4285675e04103d559afc877918a127babb320"} Jul 17 02:06:27.602311 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:27.602290 2583 scope.go:117] "RemoveContainer" containerID="ac11c4393bff95f09fbd871770dba12a9d37f86845bca5536ee16739bbae417f" Jul 17 02:06:27.602473 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:06:27.602449 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7_odh-ai-gateway-infra(3479e111-51bb-4a05-85cf-611cddadd7d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" podUID="3479e111-51bb-4a05-85cf-611cddadd7d4" Jul 17 02:06:27.615701 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:27.615640 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" podStartSLOduration=1.615624207 podStartE2EDuration="1.615624207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:06:27.614543149 +0000 UTC m=+3145.010158194" watchObservedRunningTime="2026-07-17 02:06:27.615624207 +0000 UTC m=+3145.011239058" Jul 17 02:06:28.072446 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.069556 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb"] Jul 17 02:06:28.080406 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.080353 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.082615 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.082584 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb"] Jul 17 02:06:28.084512 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.084481 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fd4ae7\"/\"rate-test-model-fd4ae7-kserve-self-signed-certs\"" Jul 17 02:06:28.084752 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.084732 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fd4ae7\"/\"openshift-service-ca.crt\"" Jul 17 02:06:28.085143 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.085090 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fd4ae7\"/\"kube-root-ca.crt\"" Jul 17 02:06:28.085298 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.085155 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fd4ae7\"/\"default-dockercfg-6bf8t\"" Jul 17 02:06:28.087898 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.087871 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:06:28.207746 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.207702 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nswf\" (UniqueName: \"kubernetes.io/projected/94d53218-d25a-4ddd-8716-919962c20aea-kube-api-access-8nswf\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.207746 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.207744 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-home\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.208022 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.207806 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-dshm\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.208022 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.207849 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-model-cache\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.208022 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.207922 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d53218-d25a-4ddd-8716-919962c20aea-tls-certs\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.208226 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.208034 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-tmp-dir\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.309041 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.309000 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-tmp-dir\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.309261 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.309068 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nswf\" (UniqueName: \"kubernetes.io/projected/94d53218-d25a-4ddd-8716-919962c20aea-kube-api-access-8nswf\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.309261 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.309118 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-home\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.309261 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.309165 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-dshm\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.309261 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.309187 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-model-cache\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.309261 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.309229 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d53218-d25a-4ddd-8716-919962c20aea-tls-certs\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.309526 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.309498 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-tmp-dir\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.309587 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.309528 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-home\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.309587 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.309564 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-model-cache\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.311621 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.311603 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-dshm\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.311788 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.311772 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d53218-d25a-4ddd-8716-919962c20aea-tls-certs\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.318228 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.318202 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nswf\" (UniqueName: \"kubernetes.io/projected/94d53218-d25a-4ddd-8716-919962c20aea-kube-api-access-8nswf\") pod \"rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.392652 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.392565 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:28.527619 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.527594 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb"] Jul 17 02:06:28.529544 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:06:28.529517 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94d53218_d25a_4ddd_8716_919962c20aea.slice/crio-80ac65cab8f85f256f054d3ebb3f7535ffcdc70b838a829926825059dd522059 WatchSource:0}: Error finding container 80ac65cab8f85f256f054d3ebb3f7535ffcdc70b838a829926825059dd522059: Status 404 returned error can't find the container with id 80ac65cab8f85f256f054d3ebb3f7535ffcdc70b838a829926825059dd522059 Jul 17 02:06:28.531536 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.531517 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 02:06:28.608396 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.608362 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" event={"ID":"94d53218-d25a-4ddd-8716-919962c20aea","Type":"ContainerStarted","Data":"80ac65cab8f85f256f054d3ebb3f7535ffcdc70b838a829926825059dd522059"} Jul 17 02:06:28.609957 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.609932 2583 generic.go:356] "Generic (PLEG): container finished" podID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" containerID="e47eb64d91baf3b685db3c0e6036504076c7d8f2ebfe922724f9317e731b91c4" exitCode=6 Jul 17 02:06:28.610063 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.609968 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" event={"ID":"85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb","Type":"ContainerDied","Data":"e47eb64d91baf3b685db3c0e6036504076c7d8f2ebfe922724f9317e731b91c4"} Jul 17 02:06:28.610204 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:28.610191 2583 scope.go:117] "RemoveContainer" containerID="e47eb64d91baf3b685db3c0e6036504076c7d8f2ebfe922724f9317e731b91c4" Jul 17 02:06:29.617205 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:29.617157 2583 generic.go:356] "Generic (PLEG): container finished" podID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" containerID="746c1975fcbad2855b8a6efb43b8e62bc5dfda72f03e81fea3ed8cb4213e1895" exitCode=6 Jul 17 02:06:29.617205 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:29.617198 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" event={"ID":"85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb","Type":"ContainerDied","Data":"746c1975fcbad2855b8a6efb43b8e62bc5dfda72f03e81fea3ed8cb4213e1895"} Jul 17 02:06:29.617712 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:29.617270 2583 scope.go:117] "RemoveContainer" containerID="e47eb64d91baf3b685db3c0e6036504076c7d8f2ebfe922724f9317e731b91c4" Jul 17 02:06:29.617712 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:29.617454 2583 scope.go:117] "RemoveContainer" containerID="746c1975fcbad2855b8a6efb43b8e62bc5dfda72f03e81fea3ed8cb4213e1895" Jul 17 02:06:29.617813 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:06:29.617722 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c_odh-ai-gateway-infra(85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" podUID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" Jul 17 02:06:29.619200 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:29.619176 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" event={"ID":"94d53218-d25a-4ddd-8716-919962c20aea","Type":"ContainerStarted","Data":"8bc5efc4ff5ea8a10210632d53b8cd0d097cce0c5d57772ac07c4454ae56a38c"} Jul 17 02:06:29.647460 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:29.647401 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" podStartSLOduration=0.642570632 podStartE2EDuration="1.647386283s" totalImagesPullingTime="1.004815651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:06:29.645205816 +0000 UTC m=+3147.040820667" watchObservedRunningTime="2026-07-17 02:06:29.647386283 +0000 UTC m=+3147.043001133" Jul 17 02:06:30.526313 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:30.526274 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv"] Jul 17 02:06:30.531069 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:30.531048 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" Jul 17 02:06:30.535968 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:30.535934 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv"] Jul 17 02:06:30.625742 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:30.625710 2583 scope.go:117] "RemoveContainer" containerID="746c1975fcbad2855b8a6efb43b8e62bc5dfda72f03e81fea3ed8cb4213e1895" Jul 17 02:06:30.626127 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:06:30.625903 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c_odh-ai-gateway-infra(85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" podUID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" Jul 17 02:06:30.630143 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:30.630116 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7448be5a-a6b1-460b-9fbb-9b6414b985b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv\" (UID: \"7448be5a-a6b1-460b-9fbb-9b6414b985b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" Jul 17 02:06:30.731176 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:30.731144 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7448be5a-a6b1-460b-9fbb-9b6414b985b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv\" (UID: \"7448be5a-a6b1-460b-9fbb-9b6414b985b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" Jul 17 02:06:30.731829 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:30.731808 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7448be5a-a6b1-460b-9fbb-9b6414b985b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv\" (UID: \"7448be5a-a6b1-460b-9fbb-9b6414b985b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" Jul 17 02:06:30.845417 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:30.845363 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" Jul 17 02:06:30.975041 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:30.975015 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv"] Jul 17 02:06:30.977374 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:06:30.977341 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7448be5a_a6b1_460b_9fbb_9b6414b985b3.slice/crio-9a7c9b916eb4d075de9049aab2744dbda56ddab5d768282d169d8063f0ce8e2b WatchSource:0}: Error finding container 9a7c9b916eb4d075de9049aab2744dbda56ddab5d768282d169d8063f0ce8e2b: Status 404 returned error can't find the container with id 9a7c9b916eb4d075de9049aab2744dbda56ddab5d768282d169d8063f0ce8e2b Jul 17 02:06:31.632523 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:31.632495 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" event={"ID":"7448be5a-a6b1-460b-9fbb-9b6414b985b3","Type":"ContainerStarted","Data":"3619f242f2cd2ef25c9d4046a28a48a4f90dc8a138278c89235188376c060396"} Jul 17 02:06:31.632821 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:31.632547 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" event={"ID":"7448be5a-a6b1-460b-9fbb-9b6414b985b3","Type":"ContainerStarted","Data":"9a7c9b916eb4d075de9049aab2744dbda56ddab5d768282d169d8063f0ce8e2b"} Jul 17 02:06:31.632821 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:31.632635 2583 scope.go:117] "RemoveContainer" containerID="3619f242f2cd2ef25c9d4046a28a48a4f90dc8a138278c89235188376c060396" Jul 17 02:06:32.638896 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:32.638854 2583 generic.go:356] "Generic (PLEG): container finished" podID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" containerID="3619f242f2cd2ef25c9d4046a28a48a4f90dc8a138278c89235188376c060396" exitCode=6 Jul 17 02:06:32.638896 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:32.638888 2583 generic.go:356] "Generic (PLEG): container finished" podID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" containerID="cfd07b11b2eafd211bd5df8e4e84f5dab0086682a44478ae65eb8b763a4328d0" exitCode=6 Jul 17 02:06:32.639395 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:32.638936 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" event={"ID":"7448be5a-a6b1-460b-9fbb-9b6414b985b3","Type":"ContainerDied","Data":"3619f242f2cd2ef25c9d4046a28a48a4f90dc8a138278c89235188376c060396"} Jul 17 02:06:32.639395 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:32.639016 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" event={"ID":"7448be5a-a6b1-460b-9fbb-9b6414b985b3","Type":"ContainerDied","Data":"cfd07b11b2eafd211bd5df8e4e84f5dab0086682a44478ae65eb8b763a4328d0"} Jul 17 02:06:32.639395 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:32.639039 2583 scope.go:117] "RemoveContainer" containerID="3619f242f2cd2ef25c9d4046a28a48a4f90dc8a138278c89235188376c060396" Jul 17 02:06:32.639395 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:32.639234 2583 scope.go:117] "RemoveContainer" containerID="cfd07b11b2eafd211bd5df8e4e84f5dab0086682a44478ae65eb8b763a4328d0" Jul 17 02:06:32.639558 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:06:32.639488 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv_odh-ai-gateway-infra(7448be5a-a6b1-460b-9fbb-9b6414b985b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" podUID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" Jul 17 02:06:32.649900 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:32.649879 2583 scope.go:117] "RemoveContainer" containerID="3619f242f2cd2ef25c9d4046a28a48a4f90dc8a138278c89235188376c060396" Jul 17 02:06:32.650188 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:06:32.650168 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3619f242f2cd2ef25c9d4046a28a48a4f90dc8a138278c89235188376c060396\": container with ID starting with 3619f242f2cd2ef25c9d4046a28a48a4f90dc8a138278c89235188376c060396 not found: ID does not exist" containerID="3619f242f2cd2ef25c9d4046a28a48a4f90dc8a138278c89235188376c060396" Jul 17 02:06:32.650261 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:32.650202 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3619f242f2cd2ef25c9d4046a28a48a4f90dc8a138278c89235188376c060396"} err="failed to get container status \"3619f242f2cd2ef25c9d4046a28a48a4f90dc8a138278c89235188376c060396\": rpc error: code = NotFound desc = could not find container \"3619f242f2cd2ef25c9d4046a28a48a4f90dc8a138278c89235188376c060396\": container with ID starting with 3619f242f2cd2ef25c9d4046a28a48a4f90dc8a138278c89235188376c060396 not found: ID does not exist" Jul 17 02:06:33.645764 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:33.645731 2583 scope.go:117] "RemoveContainer" containerID="cfd07b11b2eafd211bd5df8e4e84f5dab0086682a44478ae65eb8b763a4328d0" Jul 17 02:06:33.646289 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:06:33.645960 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv_odh-ai-gateway-infra(7448be5a-a6b1-460b-9fbb-9b6414b985b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" podUID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" Jul 17 02:06:38.393632 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:38.393595 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:38.394139 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:38.393652 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:38.406316 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:38.406293 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:38.679726 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:38.679642 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:06:41.162770 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:41.162727 2583 scope.go:117] "RemoveContainer" containerID="ac11c4393bff95f09fbd871770dba12a9d37f86845bca5536ee16739bbae417f" Jul 17 02:06:41.691891 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:41.691811 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" event={"ID":"3479e111-51bb-4a05-85cf-611cddadd7d4","Type":"ContainerStarted","Data":"21e7d65a1d64e3528a45917dfee8ca28513b1da212bce36f2ce2156772c4a700"} Jul 17 02:06:41.710483 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:41.710429 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" podStartSLOduration=18.710414408 podStartE2EDuration="18.710414408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:06:41.708303645 +0000 UTC m=+3159.103918507" watchObservedRunningTime="2026-07-17 02:06:41.710414408 +0000 UTC m=+3159.106029260" Jul 17 02:06:42.698952 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:42.698913 2583 generic.go:356] "Generic (PLEG): container finished" podID="3479e111-51bb-4a05-85cf-611cddadd7d4" containerID="21e7d65a1d64e3528a45917dfee8ca28513b1da212bce36f2ce2156772c4a700" exitCode=6 Jul 17 02:06:42.699448 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:42.698988 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" event={"ID":"3479e111-51bb-4a05-85cf-611cddadd7d4","Type":"ContainerDied","Data":"21e7d65a1d64e3528a45917dfee8ca28513b1da212bce36f2ce2156772c4a700"} Jul 17 02:06:42.699448 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:42.699054 2583 scope.go:117] "RemoveContainer" containerID="ac11c4393bff95f09fbd871770dba12a9d37f86845bca5536ee16739bbae417f" Jul 17 02:06:42.699448 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:42.699430 2583 scope.go:117] "RemoveContainer" containerID="21e7d65a1d64e3528a45917dfee8ca28513b1da212bce36f2ce2156772c4a700" Jul 17 02:06:42.699691 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:06:42.699666 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7_odh-ai-gateway-infra(3479e111-51bb-4a05-85cf-611cddadd7d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" podUID="3479e111-51bb-4a05-85cf-611cddadd7d4" Jul 17 02:06:42.723124 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:42.723066 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7"] Jul 17 02:06:43.840536 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:43.840511 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" Jul 17 02:06:43.951364 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:43.951320 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3479e111-51bb-4a05-85cf-611cddadd7d4-maas-api-service-ca\") pod \"3479e111-51bb-4a05-85cf-611cddadd7d4\" (UID: \"3479e111-51bb-4a05-85cf-611cddadd7d4\") " Jul 17 02:06:43.951685 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:43.951665 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3479e111-51bb-4a05-85cf-611cddadd7d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3479e111-51bb-4a05-85cf-611cddadd7d4" (UID: "3479e111-51bb-4a05-85cf-611cddadd7d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:06:44.052913 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:44.052872 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3479e111-51bb-4a05-85cf-611cddadd7d4-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:06:44.712220 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:44.712190 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" Jul 17 02:06:44.712220 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:44.712210 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7" event={"ID":"3479e111-51bb-4a05-85cf-611cddadd7d4","Type":"ContainerDied","Data":"66cb51c4d37520e5978a1061f71f610851cb831f3b3c4291dc278ea66fc3187f"} Jul 17 02:06:44.712479 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:44.712266 2583 scope.go:117] "RemoveContainer" containerID="21e7d65a1d64e3528a45917dfee8ca28513b1da212bce36f2ce2156772c4a700" Jul 17 02:06:44.732488 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:44.732451 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7"] Jul 17 02:06:44.740385 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:44.740344 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-gfpp7"] Jul 17 02:06:45.167684 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:45.167650 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3479e111-51bb-4a05-85cf-611cddadd7d4" path="/var/lib/kubelet/pods/3479e111-51bb-4a05-85cf-611cddadd7d4/volumes" Jul 17 02:06:46.162714 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:46.162685 2583 scope.go:117] "RemoveContainer" containerID="746c1975fcbad2855b8a6efb43b8e62bc5dfda72f03e81fea3ed8cb4213e1895" Jul 17 02:06:46.724824 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:46.724746 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" event={"ID":"85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb","Type":"ContainerStarted","Data":"33f4254776a02e9005bb4ed4c06302b735fc5027211df8edd82ef2344185afd2"} Jul 17 02:06:47.162793 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:47.162766 2583 scope.go:117] "RemoveContainer" containerID="cfd07b11b2eafd211bd5df8e4e84f5dab0086682a44478ae65eb8b763a4328d0" Jul 17 02:06:47.732159 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:47.732132 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" event={"ID":"7448be5a-a6b1-460b-9fbb-9b6414b985b3","Type":"ContainerStarted","Data":"7199ebf4560545ed8346439608869a31e62fe9077ce76024e911b8cf996e513b"} Jul 17 02:06:47.732524 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:47.732455 2583 scope.go:117] "RemoveContainer" containerID="7199ebf4560545ed8346439608869a31e62fe9077ce76024e911b8cf996e513b" Jul 17 02:06:47.732661 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:06:47.732642 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv_odh-ai-gateway-infra(7448be5a-a6b1-460b-9fbb-9b6414b985b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" podUID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" Jul 17 02:06:47.733964 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:47.733939 2583 generic.go:356] "Generic (PLEG): container finished" podID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" containerID="33f4254776a02e9005bb4ed4c06302b735fc5027211df8edd82ef2344185afd2" exitCode=6 Jul 17 02:06:47.734059 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:47.734009 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" event={"ID":"85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb","Type":"ContainerDied","Data":"33f4254776a02e9005bb4ed4c06302b735fc5027211df8edd82ef2344185afd2"} Jul 17 02:06:47.734119 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:47.734064 2583 scope.go:117] "RemoveContainer" containerID="746c1975fcbad2855b8a6efb43b8e62bc5dfda72f03e81fea3ed8cb4213e1895" Jul 17 02:06:47.734322 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:47.734305 2583 scope.go:117] "RemoveContainer" containerID="33f4254776a02e9005bb4ed4c06302b735fc5027211df8edd82ef2344185afd2" Jul 17 02:06:47.734539 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:06:47.734515 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c_odh-ai-gateway-infra(85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" podUID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" Jul 17 02:06:47.748972 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:47.748948 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c"] Jul 17 02:06:48.740622 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:48.740587 2583 generic.go:356] "Generic (PLEG): container finished" podID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" containerID="7199ebf4560545ed8346439608869a31e62fe9077ce76024e911b8cf996e513b" exitCode=6 Jul 17 02:06:48.741130 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:48.740665 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" event={"ID":"7448be5a-a6b1-460b-9fbb-9b6414b985b3","Type":"ContainerDied","Data":"7199ebf4560545ed8346439608869a31e62fe9077ce76024e911b8cf996e513b"} Jul 17 02:06:48.741130 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:48.740738 2583 scope.go:117] "RemoveContainer" containerID="cfd07b11b2eafd211bd5df8e4e84f5dab0086682a44478ae65eb8b763a4328d0" Jul 17 02:06:48.741130 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:48.741081 2583 scope.go:117] "RemoveContainer" containerID="7199ebf4560545ed8346439608869a31e62fe9077ce76024e911b8cf996e513b" Jul 17 02:06:48.741371 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:06:48.741346 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv_odh-ai-gateway-infra(7448be5a-a6b1-460b-9fbb-9b6414b985b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" podUID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" Jul 17 02:06:48.761051 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:48.761026 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv"] Jul 17 02:06:48.880833 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:48.880803 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" Jul 17 02:06:48.995372 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:48.995267 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb-maas-api-service-ca\") pod \"85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb\" (UID: \"85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb\") " Jul 17 02:06:48.995629 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:48.995605 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" (UID: "85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:06:49.096268 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:49.096232 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:06:49.749854 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:49.749830 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" event={"ID":"85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb","Type":"ContainerDied","Data":"355c08c220ab5b99bb8b6396f2a4285675e04103d559afc877918a127babb320"} Jul 17 02:06:49.750207 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:49.749886 2583 scope.go:117] "RemoveContainer" containerID="33f4254776a02e9005bb4ed4c06302b735fc5027211df8edd82ef2344185afd2" Jul 17 02:06:49.750207 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:49.749903 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c" Jul 17 02:06:49.775604 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:49.775569 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c"] Jul 17 02:06:49.780726 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:49.780685 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-9cr4c"] Jul 17 02:06:49.889077 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:49.889050 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" Jul 17 02:06:50.003806 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:50.003710 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7448be5a-a6b1-460b-9fbb-9b6414b985b3-maas-api-service-ca\") pod \"7448be5a-a6b1-460b-9fbb-9b6414b985b3\" (UID: \"7448be5a-a6b1-460b-9fbb-9b6414b985b3\") " Jul 17 02:06:50.004110 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:50.004065 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7448be5a-a6b1-460b-9fbb-9b6414b985b3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7448be5a-a6b1-460b-9fbb-9b6414b985b3" (UID: "7448be5a-a6b1-460b-9fbb-9b6414b985b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:06:50.005307 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:50.005278 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:06:50.105348 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:50.105313 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7448be5a-a6b1-460b-9fbb-9b6414b985b3-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:06:50.756424 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:50.756392 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" Jul 17 02:06:50.756424 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:50.756410 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv" event={"ID":"7448be5a-a6b1-460b-9fbb-9b6414b985b3","Type":"ContainerDied","Data":"9a7c9b916eb4d075de9049aab2744dbda56ddab5d768282d169d8063f0ce8e2b"} Jul 17 02:06:50.756982 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:50.756477 2583 scope.go:117] "RemoveContainer" containerID="7199ebf4560545ed8346439608869a31e62fe9077ce76024e911b8cf996e513b" Jul 17 02:06:50.776358 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:50.776320 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv"] Jul 17 02:06:50.781116 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:50.781076 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-29vkv"] Jul 17 02:06:51.167993 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:51.167957 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" path="/var/lib/kubelet/pods/7448be5a-a6b1-460b-9fbb-9b6414b985b3/volumes" Jul 17 02:06:51.168332 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:06:51.168318 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" path="/var/lib/kubelet/pods/85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb/volumes" Jul 17 02:07:11.812425 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:11.812382 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:07:11.913349 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:11.913311 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:07:21.272899 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:21.272869 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:07:22.401298 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:22.401262 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:07:23.984819 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.984785 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w"] Jul 17 02:07:23.985214 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985165 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" containerName="revoke-keys" Jul 17 02:07:23.985214 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985180 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" containerName="revoke-keys" Jul 17 02:07:23.985214 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985187 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" containerName="revoke-keys" Jul 17 02:07:23.985214 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985193 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" containerName="revoke-keys" Jul 17 02:07:23.985214 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985200 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3479e111-51bb-4a05-85cf-611cddadd7d4" containerName="revoke-keys" Jul 17 02:07:23.985214 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985208 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3479e111-51bb-4a05-85cf-611cddadd7d4" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985222 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985227 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985235 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985239 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985246 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3479e111-51bb-4a05-85cf-611cddadd7d4" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985251 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3479e111-51bb-4a05-85cf-611cddadd7d4" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985256 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985261 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985270 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985275 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985332 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3479e111-51bb-4a05-85cf-611cddadd7d4" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985339 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985346 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985351 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985358 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3479e111-51bb-4a05-85cf-611cddadd7d4" containerName="revoke-keys" Jul 17 02:07:23.985404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.985366 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" containerName="revoke-keys" Jul 17 02:07:23.988368 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.988351 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" Jul 17 02:07:23.990592 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.990567 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 02:07:23.994977 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:23.994952 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w"] Jul 17 02:07:24.117715 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:24.117667 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb48ec2-3baf-4163-9785-cdd8581512fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w\" (UID: \"beb48ec2-3baf-4163-9785-cdd8581512fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" Jul 17 02:07:24.218289 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:24.218249 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb48ec2-3baf-4163-9785-cdd8581512fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w\" (UID: \"beb48ec2-3baf-4163-9785-cdd8581512fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" Jul 17 02:07:24.218893 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:24.218868 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb48ec2-3baf-4163-9785-cdd8581512fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w\" (UID: \"beb48ec2-3baf-4163-9785-cdd8581512fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" Jul 17 02:07:24.300361 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:24.300332 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" Jul 17 02:07:24.433844 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:24.433678 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w"] Jul 17 02:07:24.436235 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:07:24.436205 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb48ec2_3baf_4163_9785_cdd8581512fe.slice/crio-9202b763932dc0bffe92e2ccc58501b549aada22b5c9328ecedd934082f90e97 WatchSource:0}: Error finding container 9202b763932dc0bffe92e2ccc58501b549aada22b5c9328ecedd934082f90e97: Status 404 returned error can't find the container with id 9202b763932dc0bffe92e2ccc58501b549aada22b5c9328ecedd934082f90e97 Jul 17 02:07:24.925967 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:24.925867 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" event={"ID":"beb48ec2-3baf-4163-9785-cdd8581512fe","Type":"ContainerStarted","Data":"5cd6255990426e2e699e555b89411948b2185fef7a0e9c4ccd55ff1ec962a59a"} Jul 17 02:07:24.925967 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:24.925926 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" event={"ID":"beb48ec2-3baf-4163-9785-cdd8581512fe","Type":"ContainerStarted","Data":"9202b763932dc0bffe92e2ccc58501b549aada22b5c9328ecedd934082f90e97"} Jul 17 02:07:24.942957 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:24.942902 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" podStartSLOduration=1.942885538 podStartE2EDuration="1.942885538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:07:24.941669004 +0000 UTC m=+3202.337283856" watchObservedRunningTime="2026-07-17 02:07:24.942885538 +0000 UTC m=+3202.338500388" Jul 17 02:07:25.932154 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:25.932118 2583 generic.go:356] "Generic (PLEG): container finished" podID="beb48ec2-3baf-4163-9785-cdd8581512fe" containerID="5cd6255990426e2e699e555b89411948b2185fef7a0e9c4ccd55ff1ec962a59a" exitCode=6 Jul 17 02:07:25.932557 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:25.932174 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" event={"ID":"beb48ec2-3baf-4163-9785-cdd8581512fe","Type":"ContainerDied","Data":"5cd6255990426e2e699e555b89411948b2185fef7a0e9c4ccd55ff1ec962a59a"} Jul 17 02:07:25.932557 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:25.932341 2583 scope.go:117] "RemoveContainer" containerID="5cd6255990426e2e699e555b89411948b2185fef7a0e9c4ccd55ff1ec962a59a" Jul 17 02:07:26.601710 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.601668 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr"] Jul 17 02:07:26.602117 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.602076 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3479e111-51bb-4a05-85cf-611cddadd7d4" containerName="revoke-keys" Jul 17 02:07:26.602117 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.602114 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3479e111-51bb-4a05-85cf-611cddadd7d4" containerName="revoke-keys" Jul 17 02:07:26.602266 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.602247 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7448be5a-a6b1-460b-9fbb-9b6414b985b3" containerName="revoke-keys" Jul 17 02:07:26.602266 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.602262 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="85fdc7bc-84ed-4dee-bec5-a7d0e3fd05cb" containerName="revoke-keys" Jul 17 02:07:26.602367 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.602272 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3479e111-51bb-4a05-85cf-611cddadd7d4" containerName="revoke-keys" Jul 17 02:07:26.605339 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.605319 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" Jul 17 02:07:26.611995 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.611969 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr"] Jul 17 02:07:26.742545 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.742513 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr\" (UID: \"dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" Jul 17 02:07:26.843850 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.843810 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr\" (UID: \"dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" Jul 17 02:07:26.844518 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.844491 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr\" (UID: \"dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" Jul 17 02:07:26.917634 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.917544 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" Jul 17 02:07:26.939848 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.939813 2583 generic.go:356] "Generic (PLEG): container finished" podID="beb48ec2-3baf-4163-9785-cdd8581512fe" containerID="a061d0f99376492f7f1b5f981780e678ad96c6f1d053062c4d632f939fb161e7" exitCode=6 Jul 17 02:07:26.940339 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.939858 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" event={"ID":"beb48ec2-3baf-4163-9785-cdd8581512fe","Type":"ContainerDied","Data":"a061d0f99376492f7f1b5f981780e678ad96c6f1d053062c4d632f939fb161e7"} Jul 17 02:07:26.940339 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.939946 2583 scope.go:117] "RemoveContainer" containerID="5cd6255990426e2e699e555b89411948b2185fef7a0e9c4ccd55ff1ec962a59a" Jul 17 02:07:26.940497 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:26.940452 2583 scope.go:117] "RemoveContainer" containerID="a061d0f99376492f7f1b5f981780e678ad96c6f1d053062c4d632f939fb161e7" Jul 17 02:07:26.940760 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:07:26.940733 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w_odh-ai-gateway-infra(beb48ec2-3baf-4163-9785-cdd8581512fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" podUID="beb48ec2-3baf-4163-9785-cdd8581512fe" Jul 17 02:07:27.054383 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:27.054270 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr"] Jul 17 02:07:27.056400 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:07:27.056375 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfbd3044_2652_4dd7_bf5a_b4a0dd3c29c5.slice/crio-a9e53bf85d5fb984fb4349063e6b6db56b0629b10487cb7c76fa4794b04b4ddd WatchSource:0}: Error finding container a9e53bf85d5fb984fb4349063e6b6db56b0629b10487cb7c76fa4794b04b4ddd: Status 404 returned error can't find the container with id a9e53bf85d5fb984fb4349063e6b6db56b0629b10487cb7c76fa4794b04b4ddd Jul 17 02:07:27.953034 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:27.952934 2583 generic.go:356] "Generic (PLEG): container finished" podID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" containerID="d744bcb7008eec568146adcefd7aaa490fd2a82778ff8b821a47c309b25bddb1" exitCode=6 Jul 17 02:07:27.953503 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:27.953022 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" event={"ID":"dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5","Type":"ContainerDied","Data":"d744bcb7008eec568146adcefd7aaa490fd2a82778ff8b821a47c309b25bddb1"} Jul 17 02:07:27.953503 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:27.953123 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" event={"ID":"dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5","Type":"ContainerStarted","Data":"a9e53bf85d5fb984fb4349063e6b6db56b0629b10487cb7c76fa4794b04b4ddd"} Jul 17 02:07:27.953503 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:27.953236 2583 scope.go:117] "RemoveContainer" containerID="d744bcb7008eec568146adcefd7aaa490fd2a82778ff8b821a47c309b25bddb1" Jul 17 02:07:27.954844 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:27.954823 2583 scope.go:117] "RemoveContainer" containerID="a061d0f99376492f7f1b5f981780e678ad96c6f1d053062c4d632f939fb161e7" Jul 17 02:07:27.955017 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:07:27.954997 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w_odh-ai-gateway-infra(beb48ec2-3baf-4163-9785-cdd8581512fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" podUID="beb48ec2-3baf-4163-9785-cdd8581512fe" Jul 17 02:07:28.961380 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:28.961340 2583 generic.go:356] "Generic (PLEG): container finished" podID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" containerID="a98eaa40939410bdd15dd488828da2bb952605a57c4fdcd8558a942519e35c9b" exitCode=6 Jul 17 02:07:28.961840 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:28.961400 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" event={"ID":"dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5","Type":"ContainerDied","Data":"a98eaa40939410bdd15dd488828da2bb952605a57c4fdcd8558a942519e35c9b"} Jul 17 02:07:28.961840 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:28.961461 2583 scope.go:117] "RemoveContainer" containerID="d744bcb7008eec568146adcefd7aaa490fd2a82778ff8b821a47c309b25bddb1" Jul 17 02:07:28.961840 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:28.961605 2583 scope.go:117] "RemoveContainer" containerID="a98eaa40939410bdd15dd488828da2bb952605a57c4fdcd8558a942519e35c9b" Jul 17 02:07:28.961972 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:07:28.961838 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr_odh-ai-gateway-infra(dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" podUID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" Jul 17 02:07:29.306349 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:29.306321 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:07:29.407263 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:29.407226 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:07:29.968147 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:29.968116 2583 scope.go:117] "RemoveContainer" containerID="a98eaa40939410bdd15dd488828da2bb952605a57c4fdcd8558a942519e35c9b" Jul 17 02:07:29.968566 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:07:29.968317 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr_odh-ai-gateway-infra(dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" podUID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" Jul 17 02:07:30.569526 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:30.569488 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn"] Jul 17 02:07:30.574313 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:30.574294 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" Jul 17 02:07:30.579347 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:30.579319 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn"] Jul 17 02:07:30.679576 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:30.679530 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e60265-c257-4be1-b30e-d1e1e2827bed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn\" (UID: \"67e60265-c257-4be1-b30e-d1e1e2827bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" Jul 17 02:07:30.780826 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:30.780776 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e60265-c257-4be1-b30e-d1e1e2827bed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn\" (UID: \"67e60265-c257-4be1-b30e-d1e1e2827bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" Jul 17 02:07:30.781462 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:30.781441 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e60265-c257-4be1-b30e-d1e1e2827bed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn\" (UID: \"67e60265-c257-4be1-b30e-d1e1e2827bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" Jul 17 02:07:30.886595 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:30.886508 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" Jul 17 02:07:31.023234 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:31.023198 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn"] Jul 17 02:07:31.025312 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:07:31.025289 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e60265_c257_4be1_b30e_d1e1e2827bed.slice/crio-03db5d444f2830186a15723414a73d583302299faebb02dfc1f817e6b1b20085 WatchSource:0}: Error finding container 03db5d444f2830186a15723414a73d583302299faebb02dfc1f817e6b1b20085: Status 404 returned error can't find the container with id 03db5d444f2830186a15723414a73d583302299faebb02dfc1f817e6b1b20085 Jul 17 02:07:31.981864 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:31.981780 2583 generic.go:356] "Generic (PLEG): container finished" podID="67e60265-c257-4be1-b30e-d1e1e2827bed" containerID="776fb00b7530570f9789c974e50bf33be57f2c47719860aad35acbc7ddc8e1ab" exitCode=6 Jul 17 02:07:31.982035 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:31.981866 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" event={"ID":"67e60265-c257-4be1-b30e-d1e1e2827bed","Type":"ContainerDied","Data":"776fb00b7530570f9789c974e50bf33be57f2c47719860aad35acbc7ddc8e1ab"} Jul 17 02:07:31.982035 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:31.981921 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" event={"ID":"67e60265-c257-4be1-b30e-d1e1e2827bed","Type":"ContainerStarted","Data":"03db5d444f2830186a15723414a73d583302299faebb02dfc1f817e6b1b20085"} Jul 17 02:07:31.982035 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:31.982030 2583 scope.go:117] "RemoveContainer" containerID="776fb00b7530570f9789c974e50bf33be57f2c47719860aad35acbc7ddc8e1ab" Jul 17 02:07:32.989642 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:32.989550 2583 generic.go:356] "Generic (PLEG): container finished" podID="67e60265-c257-4be1-b30e-d1e1e2827bed" containerID="62646a7f639b4ff0c4ede0b9fc1943e5d4980bae5a4bf3937b859f7b9c1c4d9c" exitCode=6 Jul 17 02:07:32.989642 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:32.989620 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" event={"ID":"67e60265-c257-4be1-b30e-d1e1e2827bed","Type":"ContainerDied","Data":"62646a7f639b4ff0c4ede0b9fc1943e5d4980bae5a4bf3937b859f7b9c1c4d9c"} Jul 17 02:07:32.990171 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:32.989697 2583 scope.go:117] "RemoveContainer" containerID="776fb00b7530570f9789c974e50bf33be57f2c47719860aad35acbc7ddc8e1ab" Jul 17 02:07:32.990171 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:32.989970 2583 scope.go:117] "RemoveContainer" containerID="62646a7f639b4ff0c4ede0b9fc1943e5d4980bae5a4bf3937b859f7b9c1c4d9c" Jul 17 02:07:32.990288 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:07:32.990262 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn_odh-ai-gateway-infra(67e60265-c257-4be1-b30e-d1e1e2827bed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" podUID="67e60265-c257-4be1-b30e-d1e1e2827bed" Jul 17 02:07:33.945687 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:33.945648 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb"] Jul 17 02:07:33.962437 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:33.962407 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:07:33.996248 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:33.996224 2583 scope.go:117] "RemoveContainer" containerID="62646a7f639b4ff0c4ede0b9fc1943e5d4980bae5a4bf3937b859f7b9c1c4d9c" Jul 17 02:07:33.996597 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:07:33.996407 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn_odh-ai-gateway-infra(67e60265-c257-4be1-b30e-d1e1e2827bed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" podUID="67e60265-c257-4be1-b30e-d1e1e2827bed" Jul 17 02:07:34.776405 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:34.776370 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:07:38.623550 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:38.623511 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4"] Jul 17 02:07:38.628013 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:38.627995 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" Jul 17 02:07:38.633664 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:38.633633 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4"] Jul 17 02:07:38.756111 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:38.756076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7d5228-604a-4e61-ad38-20800f55c7b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4\" (UID: \"ef7d5228-604a-4e61-ad38-20800f55c7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" Jul 17 02:07:38.810393 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:38.810362 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:07:38.857282 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:38.857238 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7d5228-604a-4e61-ad38-20800f55c7b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4\" (UID: \"ef7d5228-604a-4e61-ad38-20800f55c7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" Jul 17 02:07:38.857875 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:38.857853 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7d5228-604a-4e61-ad38-20800f55c7b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4\" (UID: \"ef7d5228-604a-4e61-ad38-20800f55c7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" Jul 17 02:07:38.911584 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:38.911487 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:07:38.939872 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:38.939852 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" Jul 17 02:07:39.097991 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:39.097964 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4"] Jul 17 02:07:39.099659 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:07:39.099631 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7d5228_604a_4e61_ad38_20800f55c7b1.slice/crio-1b67e6e8e1e5bce64cc6ff3bdcd69e78165d8789ea7e03f80f52cdb576efedc6 WatchSource:0}: Error finding container 1b67e6e8e1e5bce64cc6ff3bdcd69e78165d8789ea7e03f80f52cdb576efedc6: Status 404 returned error can't find the container with id 1b67e6e8e1e5bce64cc6ff3bdcd69e78165d8789ea7e03f80f52cdb576efedc6 Jul 17 02:07:39.162403 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:39.162348 2583 scope.go:117] "RemoveContainer" containerID="a061d0f99376492f7f1b5f981780e678ad96c6f1d053062c4d632f939fb161e7" Jul 17 02:07:40.033739 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:40.033661 2583 generic.go:356] "Generic (PLEG): container finished" podID="beb48ec2-3baf-4163-9785-cdd8581512fe" containerID="f4ccfaecf2c984f7d615dd29a612df7613255be4ed6e70b1f5332301d6676d5e" exitCode=6 Jul 17 02:07:40.034204 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:40.033768 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" event={"ID":"beb48ec2-3baf-4163-9785-cdd8581512fe","Type":"ContainerDied","Data":"f4ccfaecf2c984f7d615dd29a612df7613255be4ed6e70b1f5332301d6676d5e"} Jul 17 02:07:40.034204 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:40.033869 2583 scope.go:117] "RemoveContainer" containerID="a061d0f99376492f7f1b5f981780e678ad96c6f1d053062c4d632f939fb161e7" Jul 17 02:07:40.034340 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:40.034202 2583 scope.go:117] "RemoveContainer" containerID="f4ccfaecf2c984f7d615dd29a612df7613255be4ed6e70b1f5332301d6676d5e" Jul 17 02:07:40.034469 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:07:40.034448 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w_odh-ai-gateway-infra(beb48ec2-3baf-4163-9785-cdd8581512fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" podUID="beb48ec2-3baf-4163-9785-cdd8581512fe" Jul 17 02:07:40.035477 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:40.035455 2583 generic.go:356] "Generic (PLEG): container finished" podID="ef7d5228-604a-4e61-ad38-20800f55c7b1" containerID="2bb045fba0bcdc36e148746da762222a07ea34c495a1901d59c6d29d77e9ad65" exitCode=6 Jul 17 02:07:40.035579 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:40.035506 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" event={"ID":"ef7d5228-604a-4e61-ad38-20800f55c7b1","Type":"ContainerDied","Data":"2bb045fba0bcdc36e148746da762222a07ea34c495a1901d59c6d29d77e9ad65"} Jul 17 02:07:40.035579 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:40.035542 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" event={"ID":"ef7d5228-604a-4e61-ad38-20800f55c7b1","Type":"ContainerStarted","Data":"1b67e6e8e1e5bce64cc6ff3bdcd69e78165d8789ea7e03f80f52cdb576efedc6"} Jul 17 02:07:40.035687 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:40.035671 2583 scope.go:117] "RemoveContainer" containerID="2bb045fba0bcdc36e148746da762222a07ea34c495a1901d59c6d29d77e9ad65" Jul 17 02:07:41.045300 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:41.045271 2583 generic.go:356] "Generic (PLEG): container finished" podID="ef7d5228-604a-4e61-ad38-20800f55c7b1" containerID="fb3a43e52f63e21e8970edbf2365def356842fc1f6fcb7ac328568c9a9783eb6" exitCode=6 Jul 17 02:07:41.045762 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:41.045353 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" event={"ID":"ef7d5228-604a-4e61-ad38-20800f55c7b1","Type":"ContainerDied","Data":"fb3a43e52f63e21e8970edbf2365def356842fc1f6fcb7ac328568c9a9783eb6"} Jul 17 02:07:41.045762 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:41.045421 2583 scope.go:117] "RemoveContainer" containerID="2bb045fba0bcdc36e148746da762222a07ea34c495a1901d59c6d29d77e9ad65" Jul 17 02:07:41.045762 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:41.045660 2583 scope.go:117] "RemoveContainer" containerID="fb3a43e52f63e21e8970edbf2365def356842fc1f6fcb7ac328568c9a9783eb6" Jul 17 02:07:41.045922 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:07:41.045897 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4_odh-ai-gateway-infra(ef7d5228-604a-4e61-ad38-20800f55c7b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" podUID="ef7d5228-604a-4e61-ad38-20800f55c7b1" Jul 17 02:07:41.058589 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:41.058567 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w"] Jul 17 02:07:41.163038 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:41.163015 2583 scope.go:117] "RemoveContainer" containerID="a98eaa40939410bdd15dd488828da2bb952605a57c4fdcd8558a942519e35c9b" Jul 17 02:07:41.212424 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:41.212402 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" Jul 17 02:07:41.378011 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:41.377980 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb48ec2-3baf-4163-9785-cdd8581512fe-maas-api-service-ca\") pod \"beb48ec2-3baf-4163-9785-cdd8581512fe\" (UID: \"beb48ec2-3baf-4163-9785-cdd8581512fe\") " Jul 17 02:07:41.378361 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:41.378338 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb48ec2-3baf-4163-9785-cdd8581512fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "beb48ec2-3baf-4163-9785-cdd8581512fe" (UID: "beb48ec2-3baf-4163-9785-cdd8581512fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:07:41.478750 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:41.478712 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb48ec2-3baf-4163-9785-cdd8581512fe-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:07:42.051501 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:42.051469 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" Jul 17 02:07:42.051501 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:42.051494 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w" event={"ID":"beb48ec2-3baf-4163-9785-cdd8581512fe","Type":"ContainerDied","Data":"9202b763932dc0bffe92e2ccc58501b549aada22b5c9328ecedd934082f90e97"} Jul 17 02:07:42.051993 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:42.051567 2583 scope.go:117] "RemoveContainer" containerID="f4ccfaecf2c984f7d615dd29a612df7613255be4ed6e70b1f5332301d6676d5e" Jul 17 02:07:42.053634 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:42.053608 2583 generic.go:356] "Generic (PLEG): container finished" podID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" containerID="e858eebe18ee195dd4e841bf98a75054e6ab14517308261053cfdcf7daa3fe04" exitCode=6 Jul 17 02:07:42.053724 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:42.053674 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" event={"ID":"dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5","Type":"ContainerDied","Data":"e858eebe18ee195dd4e841bf98a75054e6ab14517308261053cfdcf7daa3fe04"} Jul 17 02:07:42.053977 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:42.053959 2583 scope.go:117] "RemoveContainer" containerID="e858eebe18ee195dd4e841bf98a75054e6ab14517308261053cfdcf7daa3fe04" Jul 17 02:07:42.054443 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:07:42.054255 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr_odh-ai-gateway-infra(dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" podUID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" Jul 17 02:07:42.055618 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:42.055560 2583 scope.go:117] "RemoveContainer" containerID="fb3a43e52f63e21e8970edbf2365def356842fc1f6fcb7ac328568c9a9783eb6" Jul 17 02:07:42.055749 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:07:42.055731 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4_odh-ai-gateway-infra(ef7d5228-604a-4e61-ad38-20800f55c7b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" podUID="ef7d5228-604a-4e61-ad38-20800f55c7b1" Jul 17 02:07:42.060412 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:42.060363 2583 scope.go:117] "RemoveContainer" containerID="a98eaa40939410bdd15dd488828da2bb952605a57c4fdcd8558a942519e35c9b" Jul 17 02:07:42.134449 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:42.134416 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w"] Jul 17 02:07:42.152146 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:42.152120 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-f7j8w"] Jul 17 02:07:43.101117 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:43.101064 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr"] Jul 17 02:07:43.171272 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:43.171236 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beb48ec2-3baf-4163-9785-cdd8581512fe" path="/var/lib/kubelet/pods/beb48ec2-3baf-4163-9785-cdd8581512fe/volumes" Jul 17 02:07:43.239741 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:43.239716 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" Jul 17 02:07:43.394781 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:43.394677 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5-maas-api-service-ca\") pod \"dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5\" (UID: \"dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5\") " Jul 17 02:07:43.395057 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:43.395035 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" (UID: "dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:07:43.496004 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:43.495966 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:07:44.070054 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:44.070027 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" Jul 17 02:07:44.070306 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:44.070065 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr" event={"ID":"dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5","Type":"ContainerDied","Data":"a9e53bf85d5fb984fb4349063e6b6db56b0629b10487cb7c76fa4794b04b4ddd"} Jul 17 02:07:44.070306 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:44.070159 2583 scope.go:117] "RemoveContainer" containerID="e858eebe18ee195dd4e841bf98a75054e6ab14517308261053cfdcf7daa3fe04" Jul 17 02:07:44.091629 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:44.091597 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr"] Jul 17 02:07:44.099055 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:44.099025 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-qr8rr"] Jul 17 02:07:45.162499 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:45.162466 2583 scope.go:117] "RemoveContainer" containerID="62646a7f639b4ff0c4ede0b9fc1943e5d4980bae5a4bf3937b859f7b9c1c4d9c" Jul 17 02:07:45.168041 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:45.168016 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" path="/var/lib/kubelet/pods/dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5/volumes" Jul 17 02:07:46.082626 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:46.082535 2583 generic.go:356] "Generic (PLEG): container finished" podID="67e60265-c257-4be1-b30e-d1e1e2827bed" containerID="ea79bbd7bf09c3901fdb2c1296a9a83369f8d38aec63fea80dd016e499d62d60" exitCode=6 Jul 17 02:07:46.082626 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:46.082613 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" event={"ID":"67e60265-c257-4be1-b30e-d1e1e2827bed","Type":"ContainerDied","Data":"ea79bbd7bf09c3901fdb2c1296a9a83369f8d38aec63fea80dd016e499d62d60"} Jul 17 02:07:46.082872 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:46.082687 2583 scope.go:117] "RemoveContainer" containerID="62646a7f639b4ff0c4ede0b9fc1943e5d4980bae5a4bf3937b859f7b9c1c4d9c" Jul 17 02:07:46.083033 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:46.083016 2583 scope.go:117] "RemoveContainer" containerID="ea79bbd7bf09c3901fdb2c1296a9a83369f8d38aec63fea80dd016e499d62d60" Jul 17 02:07:46.083287 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:07:46.083268 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn_odh-ai-gateway-infra(67e60265-c257-4be1-b30e-d1e1e2827bed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" podUID="67e60265-c257-4be1-b30e-d1e1e2827bed" Jul 17 02:07:47.108321 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:47.108286 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn"] Jul 17 02:07:47.246668 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:47.246637 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" Jul 17 02:07:47.322760 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:47.322726 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e60265-c257-4be1-b30e-d1e1e2827bed-maas-api-service-ca\") pod \"67e60265-c257-4be1-b30e-d1e1e2827bed\" (UID: \"67e60265-c257-4be1-b30e-d1e1e2827bed\") " Jul 17 02:07:47.323178 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:47.323149 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e60265-c257-4be1-b30e-d1e1e2827bed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67e60265-c257-4be1-b30e-d1e1e2827bed" (UID: "67e60265-c257-4be1-b30e-d1e1e2827bed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:07:47.423917 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:47.423832 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e60265-c257-4be1-b30e-d1e1e2827bed-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:07:48.095193 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:48.095157 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" Jul 17 02:07:48.095193 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:48.095177 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn" event={"ID":"67e60265-c257-4be1-b30e-d1e1e2827bed","Type":"ContainerDied","Data":"03db5d444f2830186a15723414a73d583302299faebb02dfc1f817e6b1b20085"} Jul 17 02:07:48.095421 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:48.095232 2583 scope.go:117] "RemoveContainer" containerID="ea79bbd7bf09c3901fdb2c1296a9a83369f8d38aec63fea80dd016e499d62d60" Jul 17 02:07:48.119331 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:48.119298 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn"] Jul 17 02:07:48.124872 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:48.124835 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-qpdcn"] Jul 17 02:07:48.967813 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:48.967769 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" podUID="94d53218-d25a-4ddd-8716-919962c20aea" containerName="main" containerID="cri-o://8bc5efc4ff5ea8a10210632d53b8cd0d097cce0c5d57772ac07c4454ae56a38c" gracePeriod=45 Jul 17 02:07:49.108417 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.108385 2583 generic.go:356] "Generic (PLEG): container finished" podID="94d53218-d25a-4ddd-8716-919962c20aea" containerID="8bc5efc4ff5ea8a10210632d53b8cd0d097cce0c5d57772ac07c4454ae56a38c" exitCode=0 Jul 17 02:07:49.108568 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.108467 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" event={"ID":"94d53218-d25a-4ddd-8716-919962c20aea","Type":"ContainerDied","Data":"8bc5efc4ff5ea8a10210632d53b8cd0d097cce0c5d57772ac07c4454ae56a38c"} Jul 17 02:07:49.169849 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.169814 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e60265-c257-4be1-b30e-d1e1e2827bed" path="/var/lib/kubelet/pods/67e60265-c257-4be1-b30e-d1e1e2827bed/volumes" Jul 17 02:07:49.216721 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.216694 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:07:49.336941 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.336906 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-model-cache\") pod \"94d53218-d25a-4ddd-8716-919962c20aea\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " Jul 17 02:07:49.336941 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.336946 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d53218-d25a-4ddd-8716-919962c20aea-tls-certs\") pod \"94d53218-d25a-4ddd-8716-919962c20aea\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " Jul 17 02:07:49.337229 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.336982 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nswf\" (UniqueName: \"kubernetes.io/projected/94d53218-d25a-4ddd-8716-919962c20aea-kube-api-access-8nswf\") pod \"94d53218-d25a-4ddd-8716-919962c20aea\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " Jul 17 02:07:49.337229 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.337036 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-home\") pod \"94d53218-d25a-4ddd-8716-919962c20aea\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " Jul 17 02:07:49.337229 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.337077 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-tmp-dir\") pod \"94d53218-d25a-4ddd-8716-919962c20aea\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " Jul 17 02:07:49.337229 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.337168 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-dshm\") pod \"94d53218-d25a-4ddd-8716-919962c20aea\" (UID: \"94d53218-d25a-4ddd-8716-919962c20aea\") " Jul 17 02:07:49.337457 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.337297 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-model-cache" (OuterVolumeSpecName: "model-cache") pod "94d53218-d25a-4ddd-8716-919962c20aea" (UID: "94d53218-d25a-4ddd-8716-919962c20aea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 02:07:49.337457 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.337363 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-home" (OuterVolumeSpecName: "home") pod "94d53218-d25a-4ddd-8716-919962c20aea" (UID: "94d53218-d25a-4ddd-8716-919962c20aea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 02:07:49.337457 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.337413 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-model-cache\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:07:49.337457 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.337434 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94d53218-d25a-4ddd-8716-919962c20aea" (UID: "94d53218-d25a-4ddd-8716-919962c20aea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 02:07:49.339595 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.339572 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-dshm" (OuterVolumeSpecName: "dshm") pod "94d53218-d25a-4ddd-8716-919962c20aea" (UID: "94d53218-d25a-4ddd-8716-919962c20aea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 02:07:49.339686 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.339595 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d53218-d25a-4ddd-8716-919962c20aea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94d53218-d25a-4ddd-8716-919962c20aea" (UID: "94d53218-d25a-4ddd-8716-919962c20aea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 02:07:49.339686 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.339598 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d53218-d25a-4ddd-8716-919962c20aea-kube-api-access-8nswf" (OuterVolumeSpecName: "kube-api-access-8nswf") pod "94d53218-d25a-4ddd-8716-919962c20aea" (UID: "94d53218-d25a-4ddd-8716-919962c20aea"). InnerVolumeSpecName "kube-api-access-8nswf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 02:07:49.437818 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.437777 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-home\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:07:49.437818 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.437814 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-tmp-dir\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:07:49.437818 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.437822 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d53218-d25a-4ddd-8716-919962c20aea-dshm\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:07:49.438048 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.437831 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d53218-d25a-4ddd-8716-919962c20aea-tls-certs\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:07:49.438048 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:49.437841 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nswf\" (UniqueName: \"kubernetes.io/projected/94d53218-d25a-4ddd-8716-919962c20aea-kube-api-access-8nswf\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:07:50.116177 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:50.116140 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" event={"ID":"94d53218-d25a-4ddd-8716-919962c20aea","Type":"ContainerDied","Data":"80ac65cab8f85f256f054d3ebb3f7535ffcdc70b838a829926825059dd522059"} Jul 17 02:07:50.116374 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:50.116200 2583 scope.go:117] "RemoveContainer" containerID="8bc5efc4ff5ea8a10210632d53b8cd0d097cce0c5d57772ac07c4454ae56a38c" Jul 17 02:07:50.116374 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:50.116230 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb" Jul 17 02:07:50.137846 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:50.137815 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb"] Jul 17 02:07:50.139532 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:50.139510 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fd4ae7/rate-test-model-fd4ae7-kserve-79b5879b7f-9cjnb"] Jul 17 02:07:51.167740 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:51.167702 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94d53218-d25a-4ddd-8716-919962c20aea" path="/var/lib/kubelet/pods/94d53218-d25a-4ddd-8716-919962c20aea/volumes" Jul 17 02:07:53.470488 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470411 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b"] Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470737 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb48ec2-3baf-4163-9785-cdd8581512fe" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470749 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb48ec2-3baf-4163-9785-cdd8581512fe" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470761 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470767 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470782 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e60265-c257-4be1-b30e-d1e1e2827bed" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470788 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e60265-c257-4be1-b30e-d1e1e2827bed" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470797 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e60265-c257-4be1-b30e-d1e1e2827bed" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470803 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e60265-c257-4be1-b30e-d1e1e2827bed" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470809 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e60265-c257-4be1-b30e-d1e1e2827bed" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470815 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e60265-c257-4be1-b30e-d1e1e2827bed" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470825 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb48ec2-3baf-4163-9785-cdd8581512fe" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470830 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb48ec2-3baf-4163-9785-cdd8581512fe" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470835 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94d53218-d25a-4ddd-8716-919962c20aea" containerName="main" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470840 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="94d53218-d25a-4ddd-8716-919962c20aea" containerName="main" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470849 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470853 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470861 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" containerName="revoke-keys" Jul 17 02:07:53.470870 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470866 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" containerName="revoke-keys" Jul 17 02:07:53.471435 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470921 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb48ec2-3baf-4163-9785-cdd8581512fe" containerName="revoke-keys" Jul 17 02:07:53.471435 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470927 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e60265-c257-4be1-b30e-d1e1e2827bed" containerName="revoke-keys" Jul 17 02:07:53.471435 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470932 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" containerName="revoke-keys" Jul 17 02:07:53.471435 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470938 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="94d53218-d25a-4ddd-8716-919962c20aea" containerName="main" Jul 17 02:07:53.471435 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470945 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" containerName="revoke-keys" Jul 17 02:07:53.471435 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470951 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb48ec2-3baf-4163-9785-cdd8581512fe" containerName="revoke-keys" Jul 17 02:07:53.471435 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470956 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfbd3044-2652-4dd7-bf5a-b4a0dd3c29c5" containerName="revoke-keys" Jul 17 02:07:53.471435 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470962 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb48ec2-3baf-4163-9785-cdd8581512fe" containerName="revoke-keys" Jul 17 02:07:53.471435 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.470968 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e60265-c257-4be1-b30e-d1e1e2827bed" containerName="revoke-keys" Jul 17 02:07:53.475212 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.475192 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b" Jul 17 02:07:53.480872 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.480844 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b"] Jul 17 02:07:53.573850 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.573812 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6318e69c-61c9-413b-83bd-8588898b114b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b\" (UID: \"6318e69c-61c9-413b-83bd-8588898b114b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b" Jul 17 02:07:53.675172 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.675128 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6318e69c-61c9-413b-83bd-8588898b114b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b\" (UID: \"6318e69c-61c9-413b-83bd-8588898b114b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b" Jul 17 02:07:53.675780 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.675758 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6318e69c-61c9-413b-83bd-8588898b114b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b\" (UID: \"6318e69c-61c9-413b-83bd-8588898b114b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b" Jul 17 02:07:53.786974 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.786943 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b" Jul 17 02:07:53.921235 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:53.921210 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b"] Jul 17 02:07:53.923780 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:07:53.923741 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6318e69c_61c9_413b_83bd_8588898b114b.slice/crio-08217b7de1b5006b8670e7fef536dcc52586f40f32155ccfe7e1cd2a6abfc2dc WatchSource:0}: Error finding container 08217b7de1b5006b8670e7fef536dcc52586f40f32155ccfe7e1cd2a6abfc2dc: Status 404 returned error can't find the container with id 08217b7de1b5006b8670e7fef536dcc52586f40f32155ccfe7e1cd2a6abfc2dc Jul 17 02:07:54.140761 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:54.140721 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b" event={"ID":"6318e69c-61c9-413b-83bd-8588898b114b","Type":"ContainerStarted","Data":"c5f6a06f51150e4a3f65dcd885412eb33b426769bc0fe92fca85d0a6e0b41441"} Jul 17 02:07:54.140912 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:54.140793 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b" event={"ID":"6318e69c-61c9-413b-83bd-8588898b114b","Type":"ContainerStarted","Data":"08217b7de1b5006b8670e7fef536dcc52586f40f32155ccfe7e1cd2a6abfc2dc"} Jul 17 02:07:54.154610 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:54.154557 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b" podStartSLOduration=1.154540971 podStartE2EDuration="1.154540971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:07:54.153308111 +0000 UTC m=+3231.548922961" watchObservedRunningTime="2026-07-17 02:07:54.154540971 +0000 UTC m=+3231.550155836" Jul 17 02:07:54.873778 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:54.873746 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:07:54.975940 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:54.975902 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:07:55.147763 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:55.147668 2583 generic.go:356] "Generic (PLEG): container finished" podID="6318e69c-61c9-413b-83bd-8588898b114b" containerID="c5f6a06f51150e4a3f65dcd885412eb33b426769bc0fe92fca85d0a6e0b41441" exitCode=0 Jul 17 02:07:55.147763 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:55.147757 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b" event={"ID":"6318e69c-61c9-413b-83bd-8588898b114b","Type":"ContainerDied","Data":"c5f6a06f51150e4a3f65dcd885412eb33b426769bc0fe92fca85d0a6e0b41441"} Jul 17 02:07:55.162489 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:55.162459 2583 scope.go:117] "RemoveContainer" containerID="fb3a43e52f63e21e8970edbf2365def356842fc1f6fcb7ac328568c9a9783eb6" Jul 17 02:07:56.154790 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:56.154747 2583 generic.go:356] "Generic (PLEG): container finished" podID="ef7d5228-604a-4e61-ad38-20800f55c7b1" containerID="22fbcbfa58833350c651f8ae3969fe1b1d9346894a09629c885bac808f76ce90" exitCode=6 Jul 17 02:07:56.155214 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:56.154820 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" event={"ID":"ef7d5228-604a-4e61-ad38-20800f55c7b1","Type":"ContainerDied","Data":"22fbcbfa58833350c651f8ae3969fe1b1d9346894a09629c885bac808f76ce90"} Jul 17 02:07:56.155214 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:56.154887 2583 scope.go:117] "RemoveContainer" containerID="fb3a43e52f63e21e8970edbf2365def356842fc1f6fcb7ac328568c9a9783eb6" Jul 17 02:07:56.155411 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:56.155240 2583 scope.go:117] "RemoveContainer" containerID="22fbcbfa58833350c651f8ae3969fe1b1d9346894a09629c885bac808f76ce90" Jul 17 02:07:56.155496 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:07:56.155474 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4_odh-ai-gateway-infra(ef7d5228-604a-4e61-ad38-20800f55c7b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" podUID="ef7d5228-604a-4e61-ad38-20800f55c7b1" Jul 17 02:07:56.294910 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:56.294883 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b" Jul 17 02:07:56.399175 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:56.399141 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6318e69c-61c9-413b-83bd-8588898b114b-maas-api-service-ca\") pod \"6318e69c-61c9-413b-83bd-8588898b114b\" (UID: \"6318e69c-61c9-413b-83bd-8588898b114b\") " Jul 17 02:07:56.399470 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:56.399445 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6318e69c-61c9-413b-83bd-8588898b114b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6318e69c-61c9-413b-83bd-8588898b114b" (UID: "6318e69c-61c9-413b-83bd-8588898b114b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:07:56.500132 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:56.500029 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6318e69c-61c9-413b-83bd-8588898b114b-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:07:57.160526 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:57.160495 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b" Jul 17 02:07:57.160526 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:57.160510 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b" event={"ID":"6318e69c-61c9-413b-83bd-8588898b114b","Type":"ContainerDied","Data":"08217b7de1b5006b8670e7fef536dcc52586f40f32155ccfe7e1cd2a6abfc2dc"} Jul 17 02:07:57.161074 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:57.160559 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08217b7de1b5006b8670e7fef536dcc52586f40f32155ccfe7e1cd2a6abfc2dc" Jul 17 02:07:57.179772 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:57.179742 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4"] Jul 17 02:07:57.302024 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:57.302001 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" Jul 17 02:07:57.408872 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:57.408839 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7d5228-604a-4e61-ad38-20800f55c7b1-maas-api-service-ca\") pod \"ef7d5228-604a-4e61-ad38-20800f55c7b1\" (UID: \"ef7d5228-604a-4e61-ad38-20800f55c7b1\") " Jul 17 02:07:57.409211 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:57.409187 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7d5228-604a-4e61-ad38-20800f55c7b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef7d5228-604a-4e61-ad38-20800f55c7b1" (UID: "ef7d5228-604a-4e61-ad38-20800f55c7b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:07:57.509491 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:57.509413 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7d5228-604a-4e61-ad38-20800f55c7b1-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:07:58.168045 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:58.168016 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" Jul 17 02:07:58.168538 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:58.168067 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4" event={"ID":"ef7d5228-604a-4e61-ad38-20800f55c7b1","Type":"ContainerDied","Data":"1b67e6e8e1e5bce64cc6ff3bdcd69e78165d8789ea7e03f80f52cdb576efedc6"} Jul 17 02:07:58.168538 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:58.168143 2583 scope.go:117] "RemoveContainer" containerID="22fbcbfa58833350c651f8ae3969fe1b1d9346894a09629c885bac808f76ce90" Jul 17 02:07:58.195159 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:58.194578 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4"] Jul 17 02:07:58.199455 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:58.199432 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-nbhl4"] Jul 17 02:07:59.167792 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:07:59.167754 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef7d5228-604a-4e61-ad38-20800f55c7b1" path="/var/lib/kubelet/pods/ef7d5228-604a-4e61-ad38-20800f55c7b1/volumes" Jul 17 02:08:03.854339 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:03.854302 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k"] Jul 17 02:08:03.854732 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:03.854582 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" podUID="b25794c4-a21c-4449-9feb-4c3682e818d0" containerName="maas-api" containerID="cri-o://2097882d52911338420bcb1590228ad05a91500da893d4bed4b0285800ed4b8e" gracePeriod=30 Jul 17 02:08:03.883779 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:03.883745 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:08:04.199569 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:04.199538 2583 generic.go:356] "Generic (PLEG): container finished" podID="b25794c4-a21c-4449-9feb-4c3682e818d0" containerID="2097882d52911338420bcb1590228ad05a91500da893d4bed4b0285800ed4b8e" exitCode=0 Jul 17 02:08:04.199711 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:04.199612 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" event={"ID":"b25794c4-a21c-4449-9feb-4c3682e818d0","Type":"ContainerDied","Data":"2097882d52911338420bcb1590228ad05a91500da893d4bed4b0285800ed4b8e"} Jul 17 02:08:04.308487 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:04.308463 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:08:04.367468 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:04.367428 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25794c4-a21c-4449-9feb-4c3682e818d0-maas-api-tls\") pod \"b25794c4-a21c-4449-9feb-4c3682e818d0\" (UID: \"b25794c4-a21c-4449-9feb-4c3682e818d0\") " Jul 17 02:08:04.367674 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:04.367482 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25794c4-a21c-4449-9feb-4c3682e818d0-openshift-service-ca\") pod \"b25794c4-a21c-4449-9feb-4c3682e818d0\" (UID: \"b25794c4-a21c-4449-9feb-4c3682e818d0\") " Jul 17 02:08:04.367674 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:04.367605 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgnvk\" (UniqueName: \"kubernetes.io/projected/b25794c4-a21c-4449-9feb-4c3682e818d0-kube-api-access-jgnvk\") pod \"b25794c4-a21c-4449-9feb-4c3682e818d0\" (UID: \"b25794c4-a21c-4449-9feb-4c3682e818d0\") " Jul 17 02:08:04.367893 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:04.367866 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25794c4-a21c-4449-9feb-4c3682e818d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b25794c4-a21c-4449-9feb-4c3682e818d0" (UID: "b25794c4-a21c-4449-9feb-4c3682e818d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:08:04.369799 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:04.369778 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25794c4-a21c-4449-9feb-4c3682e818d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b25794c4-a21c-4449-9feb-4c3682e818d0" (UID: "b25794c4-a21c-4449-9feb-4c3682e818d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 02:08:04.369857 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:04.369790 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25794c4-a21c-4449-9feb-4c3682e818d0-kube-api-access-jgnvk" (OuterVolumeSpecName: "kube-api-access-jgnvk") pod "b25794c4-a21c-4449-9feb-4c3682e818d0" (UID: "b25794c4-a21c-4449-9feb-4c3682e818d0"). InnerVolumeSpecName "kube-api-access-jgnvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 02:08:04.468750 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:04.468659 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25794c4-a21c-4449-9feb-4c3682e818d0-maas-api-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:08:04.468750 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:04.468690 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25794c4-a21c-4449-9feb-4c3682e818d0-openshift-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:08:04.468750 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:04.468703 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgnvk\" (UniqueName: \"kubernetes.io/projected/b25794c4-a21c-4449-9feb-4c3682e818d0-kube-api-access-jgnvk\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:08:05.206147 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:05.206118 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" Jul 17 02:08:05.206147 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:05.206130 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k" event={"ID":"b25794c4-a21c-4449-9feb-4c3682e818d0","Type":"ContainerDied","Data":"9744ed62321911db1b6e78f5ab4aed8b5ae20601504ddf84431e6d6f57735e6b"} Jul 17 02:08:05.206604 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:05.206196 2583 scope.go:117] "RemoveContainer" containerID="2097882d52911338420bcb1590228ad05a91500da893d4bed4b0285800ed4b8e" Jul 17 02:08:05.222016 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:05.221976 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k"] Jul 17 02:08:05.225998 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:05.225968 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd4ae7-bc8567568-sm56k"] Jul 17 02:08:07.167689 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:07.167656 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b25794c4-a21c-4449-9feb-4c3682e818d0" path="/var/lib/kubelet/pods/b25794c4-a21c-4449-9feb-4c3682e818d0/volumes" Jul 17 02:08:08.566907 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:08.566871 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:08:14.029386 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029348 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp"] Jul 17 02:08:14.029772 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029699 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7d5228-604a-4e61-ad38-20800f55c7b1" containerName="revoke-keys" Jul 17 02:08:14.029772 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029713 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7d5228-604a-4e61-ad38-20800f55c7b1" containerName="revoke-keys" Jul 17 02:08:14.029772 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029731 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb48ec2-3baf-4163-9785-cdd8581512fe" containerName="revoke-keys" Jul 17 02:08:14.029772 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029738 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb48ec2-3baf-4163-9785-cdd8581512fe" containerName="revoke-keys" Jul 17 02:08:14.029772 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029744 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7d5228-604a-4e61-ad38-20800f55c7b1" containerName="revoke-keys" Jul 17 02:08:14.029772 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029751 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7d5228-604a-4e61-ad38-20800f55c7b1" containerName="revoke-keys" Jul 17 02:08:14.029772 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029760 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6318e69c-61c9-413b-83bd-8588898b114b" containerName="revoke-keys" Jul 17 02:08:14.029772 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029765 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6318e69c-61c9-413b-83bd-8588898b114b" containerName="revoke-keys" Jul 17 02:08:14.030023 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029780 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25794c4-a21c-4449-9feb-4c3682e818d0" containerName="maas-api" Jul 17 02:08:14.030023 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029789 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25794c4-a21c-4449-9feb-4c3682e818d0" containerName="maas-api" Jul 17 02:08:14.030023 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029843 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25794c4-a21c-4449-9feb-4c3682e818d0" containerName="maas-api" Jul 17 02:08:14.030023 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029875 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6318e69c-61c9-413b-83bd-8588898b114b" containerName="revoke-keys" Jul 17 02:08:14.030023 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029883 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7d5228-604a-4e61-ad38-20800f55c7b1" containerName="revoke-keys" Jul 17 02:08:14.030023 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029889 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e60265-c257-4be1-b30e-d1e1e2827bed" containerName="revoke-keys" Jul 17 02:08:14.030023 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.029894 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7d5228-604a-4e61-ad38-20800f55c7b1" containerName="revoke-keys" Jul 17 02:08:14.032939 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.032920 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" Jul 17 02:08:14.035182 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.035164 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 02:08:14.040404 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.040277 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp"] Jul 17 02:08:14.152284 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.152251 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdb30c4-34c8-41f5-a050-763a0be5954f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp\" (UID: \"3cdb30c4-34c8-41f5-a050-763a0be5954f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" Jul 17 02:08:14.252775 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.252746 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdb30c4-34c8-41f5-a050-763a0be5954f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp\" (UID: \"3cdb30c4-34c8-41f5-a050-763a0be5954f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" Jul 17 02:08:14.253405 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.253386 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdb30c4-34c8-41f5-a050-763a0be5954f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp\" (UID: \"3cdb30c4-34c8-41f5-a050-763a0be5954f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" Jul 17 02:08:14.344118 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.344069 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" Jul 17 02:08:14.473167 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.473141 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp"] Jul 17 02:08:14.475565 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:08:14.475536 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdb30c4_34c8_41f5_a050_763a0be5954f.slice/crio-314e017b4e0ab9252c255f52fa9c5d533c94ff0ca160a5346c83c1e208f01e1a WatchSource:0}: Error finding container 314e017b4e0ab9252c255f52fa9c5d533c94ff0ca160a5346c83c1e208f01e1a: Status 404 returned error can't find the container with id 314e017b4e0ab9252c255f52fa9c5d533c94ff0ca160a5346c83c1e208f01e1a Jul 17 02:08:14.562127 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.562080 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:08:14.617300 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:14.617217 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:08:15.261254 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:15.261215 2583 generic.go:356] "Generic (PLEG): container finished" podID="3cdb30c4-34c8-41f5-a050-763a0be5954f" containerID="b00281dab331e7cb75fbaa5e50c812d2aefc6e3b6de6f109e6fa9cae0eb9815a" exitCode=6 Jul 17 02:08:15.261645 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:15.261251 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" event={"ID":"3cdb30c4-34c8-41f5-a050-763a0be5954f","Type":"ContainerDied","Data":"b00281dab331e7cb75fbaa5e50c812d2aefc6e3b6de6f109e6fa9cae0eb9815a"} Jul 17 02:08:15.261645 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:15.261313 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" event={"ID":"3cdb30c4-34c8-41f5-a050-763a0be5954f","Type":"ContainerStarted","Data":"314e017b4e0ab9252c255f52fa9c5d533c94ff0ca160a5346c83c1e208f01e1a"} Jul 17 02:08:15.261645 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:15.261493 2583 scope.go:117] "RemoveContainer" containerID="b00281dab331e7cb75fbaa5e50c812d2aefc6e3b6de6f109e6fa9cae0eb9815a" Jul 17 02:08:16.268373 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:16.268290 2583 generic.go:356] "Generic (PLEG): container finished" podID="3cdb30c4-34c8-41f5-a050-763a0be5954f" containerID="5634cdfb3f327b39d7b2f9879f19a093af91bdf2733a11129308150651fde081" exitCode=6 Jul 17 02:08:16.268373 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:16.268330 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" event={"ID":"3cdb30c4-34c8-41f5-a050-763a0be5954f","Type":"ContainerDied","Data":"5634cdfb3f327b39d7b2f9879f19a093af91bdf2733a11129308150651fde081"} Jul 17 02:08:16.268842 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:16.268378 2583 scope.go:117] "RemoveContainer" containerID="b00281dab331e7cb75fbaa5e50c812d2aefc6e3b6de6f109e6fa9cae0eb9815a" Jul 17 02:08:16.268842 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:16.268528 2583 scope.go:117] "RemoveContainer" containerID="5634cdfb3f327b39d7b2f9879f19a093af91bdf2733a11129308150651fde081" Jul 17 02:08:16.268842 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:08:16.268750 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp_odh-ai-gateway-infra(3cdb30c4-34c8-41f5-a050-763a0be5954f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" podUID="3cdb30c4-34c8-41f5-a050-763a0be5954f" Jul 17 02:08:16.642540 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:16.642504 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb"] Jul 17 02:08:16.642854 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:16.642841 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7d5228-604a-4e61-ad38-20800f55c7b1" containerName="revoke-keys" Jul 17 02:08:16.642903 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:16.642857 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7d5228-604a-4e61-ad38-20800f55c7b1" containerName="revoke-keys" Jul 17 02:08:16.642943 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:16.642914 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7d5228-604a-4e61-ad38-20800f55c7b1" containerName="revoke-keys" Jul 17 02:08:16.646927 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:16.646912 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" Jul 17 02:08:16.652529 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:16.652502 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb"] Jul 17 02:08:16.772652 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:16.772614 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78733375-7e33-424e-9401-3457a93fbcb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb\" (UID: \"78733375-7e33-424e-9401-3457a93fbcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" Jul 17 02:08:16.873539 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:16.873502 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78733375-7e33-424e-9401-3457a93fbcb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb\" (UID: \"78733375-7e33-424e-9401-3457a93fbcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" Jul 17 02:08:16.874094 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:16.874074 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78733375-7e33-424e-9401-3457a93fbcb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb\" (UID: \"78733375-7e33-424e-9401-3457a93fbcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" Jul 17 02:08:16.960056 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:16.959740 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" Jul 17 02:08:17.095842 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:17.095744 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb"] Jul 17 02:08:17.099724 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:08:17.099696 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78733375_7e33_424e_9401_3457a93fbcb9.slice/crio-7c4f430be5f6da54d62d8450b4eb79a2c96f1acacc9fb4d3133681575aff4612 WatchSource:0}: Error finding container 7c4f430be5f6da54d62d8450b4eb79a2c96f1acacc9fb4d3133681575aff4612: Status 404 returned error can't find the container with id 7c4f430be5f6da54d62d8450b4eb79a2c96f1acacc9fb4d3133681575aff4612 Jul 17 02:08:17.275159 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:17.275031 2583 scope.go:117] "RemoveContainer" containerID="5634cdfb3f327b39d7b2f9879f19a093af91bdf2733a11129308150651fde081" Jul 17 02:08:17.275604 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:08:17.275322 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp_odh-ai-gateway-infra(3cdb30c4-34c8-41f5-a050-763a0be5954f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" podUID="3cdb30c4-34c8-41f5-a050-763a0be5954f" Jul 17 02:08:17.276149 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:17.276126 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" event={"ID":"78733375-7e33-424e-9401-3457a93fbcb9","Type":"ContainerStarted","Data":"7c4f430be5f6da54d62d8450b4eb79a2c96f1acacc9fb4d3133681575aff4612"} Jul 17 02:08:18.282411 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:18.282381 2583 generic.go:356] "Generic (PLEG): container finished" podID="78733375-7e33-424e-9401-3457a93fbcb9" containerID="3c4ebd04428769abd83fc849b2ac9ee0d3a2993f1b9aa0ff696bca4609dbf7b4" exitCode=6 Jul 17 02:08:18.282803 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:18.282472 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" event={"ID":"78733375-7e33-424e-9401-3457a93fbcb9","Type":"ContainerDied","Data":"3c4ebd04428769abd83fc849b2ac9ee0d3a2993f1b9aa0ff696bca4609dbf7b4"} Jul 17 02:08:18.282803 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:18.282616 2583 scope.go:117] "RemoveContainer" containerID="3c4ebd04428769abd83fc849b2ac9ee0d3a2993f1b9aa0ff696bca4609dbf7b4" Jul 17 02:08:19.272426 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:19.272342 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:08:19.289008 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:19.288980 2583 generic.go:356] "Generic (PLEG): container finished" podID="78733375-7e33-424e-9401-3457a93fbcb9" containerID="0d3530fc61d26e1c4af39ddd967debf01514c2395b6576966da92389ffde5258" exitCode=6 Jul 17 02:08:19.289508 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:19.289044 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" event={"ID":"78733375-7e33-424e-9401-3457a93fbcb9","Type":"ContainerDied","Data":"0d3530fc61d26e1c4af39ddd967debf01514c2395b6576966da92389ffde5258"} Jul 17 02:08:19.289508 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:19.289091 2583 scope.go:117] "RemoveContainer" containerID="3c4ebd04428769abd83fc849b2ac9ee0d3a2993f1b9aa0ff696bca4609dbf7b4" Jul 17 02:08:19.289508 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:19.289292 2583 scope.go:117] "RemoveContainer" containerID="0d3530fc61d26e1c4af39ddd967debf01514c2395b6576966da92389ffde5258" Jul 17 02:08:19.289508 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:08:19.289500 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb_odh-ai-gateway-infra(78733375-7e33-424e-9401-3457a93fbcb9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" podUID="78733375-7e33-424e-9401-3457a93fbcb9" Jul 17 02:08:19.496755 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:19.496724 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:08:19.622248 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:19.622212 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr"] Jul 17 02:08:19.626859 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:19.626836 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr" Jul 17 02:08:19.632372 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:19.632351 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr"] Jul 17 02:08:19.698624 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:19.698591 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabdf966-a15a-428c-9e67-605df0935b0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr\" (UID: \"cabdf966-a15a-428c-9e67-605df0935b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr" Jul 17 02:08:19.799337 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:19.799307 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabdf966-a15a-428c-9e67-605df0935b0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr\" (UID: \"cabdf966-a15a-428c-9e67-605df0935b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr" Jul 17 02:08:19.799887 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:19.799869 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabdf966-a15a-428c-9e67-605df0935b0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr\" (UID: \"cabdf966-a15a-428c-9e67-605df0935b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr" Jul 17 02:08:19.939643 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:19.939551 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr" Jul 17 02:08:20.072439 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:20.072416 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr"] Jul 17 02:08:20.074501 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:08:20.074466 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabdf966_a15a_428c_9e67_605df0935b0e.slice/crio-9506c6084a67139d1dce1297f630ea1055eda12c988ccf32e1a862aa71e509f5 WatchSource:0}: Error finding container 9506c6084a67139d1dce1297f630ea1055eda12c988ccf32e1a862aa71e509f5: Status 404 returned error can't find the container with id 9506c6084a67139d1dce1297f630ea1055eda12c988ccf32e1a862aa71e509f5 Jul 17 02:08:20.278974 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:20.278938 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:08:20.296190 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:20.296165 2583 scope.go:117] "RemoveContainer" containerID="0d3530fc61d26e1c4af39ddd967debf01514c2395b6576966da92389ffde5258" Jul 17 02:08:20.296609 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:08:20.296389 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb_odh-ai-gateway-infra(78733375-7e33-424e-9401-3457a93fbcb9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" podUID="78733375-7e33-424e-9401-3457a93fbcb9" Jul 17 02:08:20.297484 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:20.297457 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr" event={"ID":"cabdf966-a15a-428c-9e67-605df0935b0e","Type":"ContainerStarted","Data":"d272f28228c63d6c425a160fbb575c068fd28782b8446d3d3e59b0f7ba22dbd5"} Jul 17 02:08:20.297610 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:20.297510 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr" event={"ID":"cabdf966-a15a-428c-9e67-605df0935b0e","Type":"ContainerStarted","Data":"9506c6084a67139d1dce1297f630ea1055eda12c988ccf32e1a862aa71e509f5"} Jul 17 02:08:20.329602 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:20.329547 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr" podStartSLOduration=1.3295281970000001 podStartE2EDuration="1.329528197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:08:20.328486813 +0000 UTC m=+3257.724101667" watchObservedRunningTime="2026-07-17 02:08:20.329528197 +0000 UTC m=+3257.725143047" Jul 17 02:08:20.378384 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:20.378349 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:08:20.608881 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:20.608844 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp"] Jul 17 02:08:20.612383 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:20.612364 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" Jul 17 02:08:20.620266 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:20.620235 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp"] Jul 17 02:08:20.706565 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:20.706526 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafdb251-a59d-4b23-a42c-4543c0f14394-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp\" (UID: \"bafdb251-a59d-4b23-a42c-4543c0f14394\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" Jul 17 02:08:20.807880 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:20.807842 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafdb251-a59d-4b23-a42c-4543c0f14394-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp\" (UID: \"bafdb251-a59d-4b23-a42c-4543c0f14394\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" Jul 17 02:08:20.808481 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:20.808462 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafdb251-a59d-4b23-a42c-4543c0f14394-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp\" (UID: \"bafdb251-a59d-4b23-a42c-4543c0f14394\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" Jul 17 02:08:20.925715 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:20.925626 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" Jul 17 02:08:21.060751 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:21.060727 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp"] Jul 17 02:08:21.063039 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:08:21.063009 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafdb251_a59d_4b23_a42c_4543c0f14394.slice/crio-8d7c29d446c65e6e389a977ab385ab296b47e911152a04ce5aa3a6bb5b947434 WatchSource:0}: Error finding container 8d7c29d446c65e6e389a977ab385ab296b47e911152a04ce5aa3a6bb5b947434: Status 404 returned error can't find the container with id 8d7c29d446c65e6e389a977ab385ab296b47e911152a04ce5aa3a6bb5b947434 Jul 17 02:08:21.310376 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:21.310334 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" event={"ID":"bafdb251-a59d-4b23-a42c-4543c0f14394","Type":"ContainerStarted","Data":"5d3e6f41ca1a0854c5d20ec59e400f97f716b98a22c71452c51991961d79744b"} Jul 17 02:08:21.310823 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:21.310429 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" event={"ID":"bafdb251-a59d-4b23-a42c-4543c0f14394","Type":"ContainerStarted","Data":"8d7c29d446c65e6e389a977ab385ab296b47e911152a04ce5aa3a6bb5b947434"} Jul 17 02:08:21.311827 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:21.311799 2583 generic.go:356] "Generic (PLEG): container finished" podID="cabdf966-a15a-428c-9e67-605df0935b0e" containerID="d272f28228c63d6c425a160fbb575c068fd28782b8446d3d3e59b0f7ba22dbd5" exitCode=0 Jul 17 02:08:21.311937 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:21.311828 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr" event={"ID":"cabdf966-a15a-428c-9e67-605df0935b0e","Type":"ContainerDied","Data":"d272f28228c63d6c425a160fbb575c068fd28782b8446d3d3e59b0f7ba22dbd5"} Jul 17 02:08:21.326352 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:21.326306 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" podStartSLOduration=1.326292522 podStartE2EDuration="1.326292522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:08:21.32438366 +0000 UTC m=+3258.719998511" watchObservedRunningTime="2026-07-17 02:08:21.326292522 +0000 UTC m=+3258.721907374" Jul 17 02:08:22.318031 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:22.317992 2583 generic.go:356] "Generic (PLEG): container finished" podID="bafdb251-a59d-4b23-a42c-4543c0f14394" containerID="5d3e6f41ca1a0854c5d20ec59e400f97f716b98a22c71452c51991961d79744b" exitCode=6 Jul 17 02:08:22.318418 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:22.318089 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" event={"ID":"bafdb251-a59d-4b23-a42c-4543c0f14394","Type":"ContainerDied","Data":"5d3e6f41ca1a0854c5d20ec59e400f97f716b98a22c71452c51991961d79744b"} Jul 17 02:08:22.318418 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:22.318297 2583 scope.go:117] "RemoveContainer" containerID="5d3e6f41ca1a0854c5d20ec59e400f97f716b98a22c71452c51991961d79744b" Jul 17 02:08:22.456862 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:22.456836 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr" Jul 17 02:08:22.522343 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:22.522312 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabdf966-a15a-428c-9e67-605df0935b0e-maas-api-service-ca\") pod \"cabdf966-a15a-428c-9e67-605df0935b0e\" (UID: \"cabdf966-a15a-428c-9e67-605df0935b0e\") " Jul 17 02:08:22.522678 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:22.522655 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabdf966-a15a-428c-9e67-605df0935b0e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cabdf966-a15a-428c-9e67-605df0935b0e" (UID: "cabdf966-a15a-428c-9e67-605df0935b0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:08:22.623200 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:22.623092 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabdf966-a15a-428c-9e67-605df0935b0e-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:08:23.325001 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:23.324968 2583 generic.go:356] "Generic (PLEG): container finished" podID="bafdb251-a59d-4b23-a42c-4543c0f14394" containerID="f569417e62a9ae65eabdd963541e4cb48fb5c7d67076851a3f0d06eb18a97d0b" exitCode=6 Jul 17 02:08:23.325468 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:23.325061 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" event={"ID":"bafdb251-a59d-4b23-a42c-4543c0f14394","Type":"ContainerDied","Data":"f569417e62a9ae65eabdd963541e4cb48fb5c7d67076851a3f0d06eb18a97d0b"} Jul 17 02:08:23.325468 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:23.325152 2583 scope.go:117] "RemoveContainer" containerID="5d3e6f41ca1a0854c5d20ec59e400f97f716b98a22c71452c51991961d79744b" Jul 17 02:08:23.325468 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:23.325260 2583 scope.go:117] "RemoveContainer" containerID="f569417e62a9ae65eabdd963541e4cb48fb5c7d67076851a3f0d06eb18a97d0b" Jul 17 02:08:23.325639 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:08:23.325493 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp_odh-ai-gateway-infra(bafdb251-a59d-4b23-a42c-4543c0f14394)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" podUID="bafdb251-a59d-4b23-a42c-4543c0f14394" Jul 17 02:08:23.326460 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:23.326401 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr" Jul 17 02:08:23.326460 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:23.326414 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr" event={"ID":"cabdf966-a15a-428c-9e67-605df0935b0e","Type":"ContainerDied","Data":"9506c6084a67139d1dce1297f630ea1055eda12c988ccf32e1a862aa71e509f5"} Jul 17 02:08:23.326460 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:23.326449 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9506c6084a67139d1dce1297f630ea1055eda12c988ccf32e1a862aa71e509f5" Jul 17 02:08:24.333630 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:24.333605 2583 scope.go:117] "RemoveContainer" containerID="f569417e62a9ae65eabdd963541e4cb48fb5c7d67076851a3f0d06eb18a97d0b" Jul 17 02:08:24.334021 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:08:24.333816 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp_odh-ai-gateway-infra(bafdb251-a59d-4b23-a42c-4543c0f14394)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" podUID="bafdb251-a59d-4b23-a42c-4543c0f14394" Jul 17 02:08:28.163122 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:28.163064 2583 scope.go:117] "RemoveContainer" containerID="5634cdfb3f327b39d7b2f9879f19a093af91bdf2733a11129308150651fde081" Jul 17 02:08:28.354469 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:28.354440 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" event={"ID":"3cdb30c4-34c8-41f5-a050-763a0be5954f","Type":"ContainerStarted","Data":"06894bb22cdda3be629b05a0e09436cfd3f9038f2a73ac83bc22ff52c94e8671"} Jul 17 02:08:28.373784 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:28.373733 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" podStartSLOduration=14.373717475 podStartE2EDuration="14.373717475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:08:28.370979071 +0000 UTC m=+3265.766593921" watchObservedRunningTime="2026-07-17 02:08:28.373717475 +0000 UTC m=+3265.769332325" Jul 17 02:08:28.666791 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:28.666747 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc"] Jul 17 02:08:28.667161 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:28.667147 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cabdf966-a15a-428c-9e67-605df0935b0e" containerName="revoke-keys" Jul 17 02:08:28.667223 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:28.667163 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cabdf966-a15a-428c-9e67-605df0935b0e" containerName="revoke-keys" Jul 17 02:08:28.667259 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:28.667237 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cabdf966-a15a-428c-9e67-605df0935b0e" containerName="revoke-keys" Jul 17 02:08:28.670394 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:28.670375 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" Jul 17 02:08:28.679932 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:28.679905 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc"] Jul 17 02:08:28.777322 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:28.777286 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c6bad8-b808-4715-8769-56fcc21a303c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc\" (UID: \"72c6bad8-b808-4715-8769-56fcc21a303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" Jul 17 02:08:28.877793 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:28.877759 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c6bad8-b808-4715-8769-56fcc21a303c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc\" (UID: \"72c6bad8-b808-4715-8769-56fcc21a303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" Jul 17 02:08:28.878412 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:28.878394 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c6bad8-b808-4715-8769-56fcc21a303c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc\" (UID: \"72c6bad8-b808-4715-8769-56fcc21a303c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" Jul 17 02:08:28.983462 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:28.983375 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" Jul 17 02:08:29.138405 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:29.138377 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc"] Jul 17 02:08:29.140510 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:08:29.140477 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c6bad8_b808_4715_8769_56fcc21a303c.slice/crio-9443900f6079f96232ead25a76424b2950259d14bb93e906504f01032ac3088d WatchSource:0}: Error finding container 9443900f6079f96232ead25a76424b2950259d14bb93e906504f01032ac3088d: Status 404 returned error can't find the container with id 9443900f6079f96232ead25a76424b2950259d14bb93e906504f01032ac3088d Jul 17 02:08:29.360743 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:29.360709 2583 generic.go:356] "Generic (PLEG): container finished" podID="3cdb30c4-34c8-41f5-a050-763a0be5954f" containerID="06894bb22cdda3be629b05a0e09436cfd3f9038f2a73ac83bc22ff52c94e8671" exitCode=6 Jul 17 02:08:29.361253 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:29.360791 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" event={"ID":"3cdb30c4-34c8-41f5-a050-763a0be5954f","Type":"ContainerDied","Data":"06894bb22cdda3be629b05a0e09436cfd3f9038f2a73ac83bc22ff52c94e8671"} Jul 17 02:08:29.361253 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:29.360873 2583 scope.go:117] "RemoveContainer" containerID="5634cdfb3f327b39d7b2f9879f19a093af91bdf2733a11129308150651fde081" Jul 17 02:08:29.361253 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:29.361229 2583 scope.go:117] "RemoveContainer" containerID="06894bb22cdda3be629b05a0e09436cfd3f9038f2a73ac83bc22ff52c94e8671" Jul 17 02:08:29.361475 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:08:29.361449 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp_odh-ai-gateway-infra(3cdb30c4-34c8-41f5-a050-763a0be5954f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" podUID="3cdb30c4-34c8-41f5-a050-763a0be5954f" Jul 17 02:08:29.362276 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:29.362022 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" event={"ID":"72c6bad8-b808-4715-8769-56fcc21a303c","Type":"ContainerStarted","Data":"9443900f6079f96232ead25a76424b2950259d14bb93e906504f01032ac3088d"} Jul 17 02:08:29.382631 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:29.382601 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp"] Jul 17 02:08:30.004747 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.004659 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g"] Jul 17 02:08:30.004914 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.004885 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" podUID="3225281f-d7ca-4f64-b363-6f330d6c6a4f" containerName="maas-api" containerID="cri-o://6b9ed85784052b80b1ba4d45b88313287d58994d8610ca341c519e6c73465ef4" gracePeriod=30 Jul 17 02:08:30.027298 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.027258 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" podUID="3225281f-d7ca-4f64-b363-6f330d6c6a4f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.92:8443/health\": dial tcp 10.134.0.92:8443: connect: connection refused" Jul 17 02:08:30.040608 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.040574 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:08:30.251686 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.251658 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:08:30.370136 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.370089 2583 generic.go:356] "Generic (PLEG): container finished" podID="3225281f-d7ca-4f64-b363-6f330d6c6a4f" containerID="6b9ed85784052b80b1ba4d45b88313287d58994d8610ca341c519e6c73465ef4" exitCode=0 Jul 17 02:08:30.370493 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.370135 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" event={"ID":"3225281f-d7ca-4f64-b363-6f330d6c6a4f","Type":"ContainerDied","Data":"6b9ed85784052b80b1ba4d45b88313287d58994d8610ca341c519e6c73465ef4"} Jul 17 02:08:30.370493 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.370144 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" Jul 17 02:08:30.370493 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.370194 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g" event={"ID":"3225281f-d7ca-4f64-b363-6f330d6c6a4f","Type":"ContainerDied","Data":"cd5627976e96a10085dac67b26bbb7db38a5b876452ef5b2bee827513b214ca9"} Jul 17 02:08:30.370493 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.370208 2583 scope.go:117] "RemoveContainer" containerID="6b9ed85784052b80b1ba4d45b88313287d58994d8610ca341c519e6c73465ef4" Jul 17 02:08:30.371882 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.371856 2583 generic.go:356] "Generic (PLEG): container finished" podID="72c6bad8-b808-4715-8769-56fcc21a303c" containerID="ff25b93f2edcd8096e3c915aca6d47e3702fa70d98e230c3fd8686dfe9773413" exitCode=6 Jul 17 02:08:30.372019 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.371925 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" event={"ID":"72c6bad8-b808-4715-8769-56fcc21a303c","Type":"ContainerDied","Data":"ff25b93f2edcd8096e3c915aca6d47e3702fa70d98e230c3fd8686dfe9773413"} Jul 17 02:08:30.372139 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.372122 2583 scope.go:117] "RemoveContainer" containerID="ff25b93f2edcd8096e3c915aca6d47e3702fa70d98e230c3fd8686dfe9773413" Jul 17 02:08:30.383059 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.383036 2583 scope.go:117] "RemoveContainer" containerID="6b9ed85784052b80b1ba4d45b88313287d58994d8610ca341c519e6c73465ef4" Jul 17 02:08:30.383375 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:08:30.383353 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9ed85784052b80b1ba4d45b88313287d58994d8610ca341c519e6c73465ef4\": container with ID starting with 6b9ed85784052b80b1ba4d45b88313287d58994d8610ca341c519e6c73465ef4 not found: ID does not exist" containerID="6b9ed85784052b80b1ba4d45b88313287d58994d8610ca341c519e6c73465ef4" Jul 17 02:08:30.383469 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.383379 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9ed85784052b80b1ba4d45b88313287d58994d8610ca341c519e6c73465ef4"} err="failed to get container status \"6b9ed85784052b80b1ba4d45b88313287d58994d8610ca341c519e6c73465ef4\": rpc error: code = NotFound desc = could not find container \"6b9ed85784052b80b1ba4d45b88313287d58994d8610ca341c519e6c73465ef4\": container with ID starting with 6b9ed85784052b80b1ba4d45b88313287d58994d8610ca341c519e6c73465ef4 not found: ID does not exist" Jul 17 02:08:30.390764 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.390734 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn8zz\" (UniqueName: \"kubernetes.io/projected/3225281f-d7ca-4f64-b363-6f330d6c6a4f-kube-api-access-tn8zz\") pod \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\" (UID: \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\") " Jul 17 02:08:30.390883 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.390819 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3225281f-d7ca-4f64-b363-6f330d6c6a4f-openshift-service-ca\") pod \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\" (UID: \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\") " Jul 17 02:08:30.390883 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.390862 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3225281f-d7ca-4f64-b363-6f330d6c6a4f-maas-api-tls\") pod \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\" (UID: \"3225281f-d7ca-4f64-b363-6f330d6c6a4f\") " Jul 17 02:08:30.391414 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.391377 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3225281f-d7ca-4f64-b363-6f330d6c6a4f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3225281f-d7ca-4f64-b363-6f330d6c6a4f" (UID: "3225281f-d7ca-4f64-b363-6f330d6c6a4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:08:30.393291 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.393261 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3225281f-d7ca-4f64-b363-6f330d6c6a4f-kube-api-access-tn8zz" (OuterVolumeSpecName: "kube-api-access-tn8zz") pod "3225281f-d7ca-4f64-b363-6f330d6c6a4f" (UID: "3225281f-d7ca-4f64-b363-6f330d6c6a4f"). InnerVolumeSpecName "kube-api-access-tn8zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 02:08:30.393757 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.393732 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3225281f-d7ca-4f64-b363-6f330d6c6a4f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3225281f-d7ca-4f64-b363-6f330d6c6a4f" (UID: "3225281f-d7ca-4f64-b363-6f330d6c6a4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 02:08:30.492142 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.492115 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3225281f-d7ca-4f64-b363-6f330d6c6a4f-openshift-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:08:30.492261 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.492144 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3225281f-d7ca-4f64-b363-6f330d6c6a4f-maas-api-tls\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:08:30.492261 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.492157 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn8zz\" (UniqueName: \"kubernetes.io/projected/3225281f-d7ca-4f64-b363-6f330d6c6a4f-kube-api-access-tn8zz\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:08:30.498513 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.498490 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" Jul 17 02:08:30.593237 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.593209 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdb30c4-34c8-41f5-a050-763a0be5954f-maas-api-service-ca\") pod \"3cdb30c4-34c8-41f5-a050-763a0be5954f\" (UID: \"3cdb30c4-34c8-41f5-a050-763a0be5954f\") " Jul 17 02:08:30.593553 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.593531 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cdb30c4-34c8-41f5-a050-763a0be5954f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3cdb30c4-34c8-41f5-a050-763a0be5954f" (UID: "3cdb30c4-34c8-41f5-a050-763a0be5954f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:08:30.691922 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.691896 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g"] Jul 17 02:08:30.693944 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.693920 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdb30c4-34c8-41f5-a050-763a0be5954f-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:08:30.695880 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:30.695858 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f2720-654c9c48dd-6hx2g"] Jul 17 02:08:31.162785 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:31.162740 2583 scope.go:117] "RemoveContainer" containerID="0d3530fc61d26e1c4af39ddd967debf01514c2395b6576966da92389ffde5258" Jul 17 02:08:31.168044 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:31.168022 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3225281f-d7ca-4f64-b363-6f330d6c6a4f" path="/var/lib/kubelet/pods/3225281f-d7ca-4f64-b363-6f330d6c6a4f/volumes" Jul 17 02:08:31.379440 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:31.379408 2583 generic.go:356] "Generic (PLEG): container finished" podID="72c6bad8-b808-4715-8769-56fcc21a303c" containerID="8c52e250c86d79bcdbe427128d49be584df03c7de596688b2640b2dd75a7d29f" exitCode=6 Jul 17 02:08:31.379896 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:31.379489 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" event={"ID":"72c6bad8-b808-4715-8769-56fcc21a303c","Type":"ContainerDied","Data":"8c52e250c86d79bcdbe427128d49be584df03c7de596688b2640b2dd75a7d29f"} Jul 17 02:08:31.379896 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:31.379547 2583 scope.go:117] "RemoveContainer" containerID="ff25b93f2edcd8096e3c915aca6d47e3702fa70d98e230c3fd8686dfe9773413" Jul 17 02:08:31.379896 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:31.379709 2583 scope.go:117] "RemoveContainer" containerID="8c52e250c86d79bcdbe427128d49be584df03c7de596688b2640b2dd75a7d29f" Jul 17 02:08:31.380050 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:08:31.379934 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc_odh-ai-gateway-infra(72c6bad8-b808-4715-8769-56fcc21a303c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" podUID="72c6bad8-b808-4715-8769-56fcc21a303c" Jul 17 02:08:31.380817 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:31.380801 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" event={"ID":"3cdb30c4-34c8-41f5-a050-763a0be5954f","Type":"ContainerDied","Data":"314e017b4e0ab9252c255f52fa9c5d533c94ff0ca160a5346c83c1e208f01e1a"} Jul 17 02:08:31.380890 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:31.380845 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp" Jul 17 02:08:31.390340 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:31.390324 2583 scope.go:117] "RemoveContainer" containerID="06894bb22cdda3be629b05a0e09436cfd3f9038f2a73ac83bc22ff52c94e8671" Jul 17 02:08:31.411400 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:31.411375 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp"] Jul 17 02:08:31.414071 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:31.414047 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-j2mkp"] Jul 17 02:08:32.387773 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:32.387746 2583 scope.go:117] "RemoveContainer" containerID="8c52e250c86d79bcdbe427128d49be584df03c7de596688b2640b2dd75a7d29f" Jul 17 02:08:32.388212 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:08:32.387958 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc_odh-ai-gateway-infra(72c6bad8-b808-4715-8769-56fcc21a303c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" podUID="72c6bad8-b808-4715-8769-56fcc21a303c" Jul 17 02:08:32.389332 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:32.389310 2583 generic.go:356] "Generic (PLEG): container finished" podID="78733375-7e33-424e-9401-3457a93fbcb9" containerID="083392e4f239193f78f2dfa67bfa16dbf883d4e775377fae1123f6e2560a7f21" exitCode=6 Jul 17 02:08:32.389432 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:32.389379 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" event={"ID":"78733375-7e33-424e-9401-3457a93fbcb9","Type":"ContainerDied","Data":"083392e4f239193f78f2dfa67bfa16dbf883d4e775377fae1123f6e2560a7f21"} Jul 17 02:08:32.389474 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:32.389430 2583 scope.go:117] "RemoveContainer" containerID="0d3530fc61d26e1c4af39ddd967debf01514c2395b6576966da92389ffde5258" Jul 17 02:08:32.389660 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:32.389644 2583 scope.go:117] "RemoveContainer" containerID="083392e4f239193f78f2dfa67bfa16dbf883d4e775377fae1123f6e2560a7f21" Jul 17 02:08:32.389869 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:08:32.389851 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb_odh-ai-gateway-infra(78733375-7e33-424e-9401-3457a93fbcb9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" podUID="78733375-7e33-424e-9401-3457a93fbcb9" Jul 17 02:08:33.168525 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:33.168493 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cdb30c4-34c8-41f5-a050-763a0be5954f" path="/var/lib/kubelet/pods/3cdb30c4-34c8-41f5-a050-763a0be5954f/volumes" Jul 17 02:08:33.435654 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:33.435576 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb"] Jul 17 02:08:33.567275 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:33.567249 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" Jul 17 02:08:33.718825 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:33.718729 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78733375-7e33-424e-9401-3457a93fbcb9-maas-api-service-ca\") pod \"78733375-7e33-424e-9401-3457a93fbcb9\" (UID: \"78733375-7e33-424e-9401-3457a93fbcb9\") " Jul 17 02:08:33.719076 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:33.719054 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78733375-7e33-424e-9401-3457a93fbcb9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78733375-7e33-424e-9401-3457a93fbcb9" (UID: "78733375-7e33-424e-9401-3457a93fbcb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:08:33.820305 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:33.820273 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78733375-7e33-424e-9401-3457a93fbcb9-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:08:34.403130 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:34.403086 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" Jul 17 02:08:34.403283 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:34.403085 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb" event={"ID":"78733375-7e33-424e-9401-3457a93fbcb9","Type":"ContainerDied","Data":"7c4f430be5f6da54d62d8450b4eb79a2c96f1acacc9fb4d3133681575aff4612"} Jul 17 02:08:34.403283 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:34.403230 2583 scope.go:117] "RemoveContainer" containerID="083392e4f239193f78f2dfa67bfa16dbf883d4e775377fae1123f6e2560a7f21" Jul 17 02:08:34.424270 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:34.424239 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb"] Jul 17 02:08:34.428529 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:34.428509 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-2ldqb"] Jul 17 02:08:34.716689 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:34.716602 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:08:35.162821 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:35.162789 2583 scope.go:117] "RemoveContainer" containerID="f569417e62a9ae65eabdd963541e4cb48fb5c7d67076851a3f0d06eb18a97d0b" Jul 17 02:08:35.167823 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:35.167800 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78733375-7e33-424e-9401-3457a93fbcb9" path="/var/lib/kubelet/pods/78733375-7e33-424e-9401-3457a93fbcb9/volumes" Jul 17 02:08:35.410869 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:35.410841 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" event={"ID":"bafdb251-a59d-4b23-a42c-4543c0f14394","Type":"ContainerStarted","Data":"b8d7b1d4d5b3cd33c3a0cf5af1cc474eff602669806839283702f29ada85cc46"} Jul 17 02:08:36.417805 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:36.417771 2583 generic.go:356] "Generic (PLEG): container finished" podID="bafdb251-a59d-4b23-a42c-4543c0f14394" containerID="b8d7b1d4d5b3cd33c3a0cf5af1cc474eff602669806839283702f29ada85cc46" exitCode=6 Jul 17 02:08:36.418256 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:36.417849 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" event={"ID":"bafdb251-a59d-4b23-a42c-4543c0f14394","Type":"ContainerDied","Data":"b8d7b1d4d5b3cd33c3a0cf5af1cc474eff602669806839283702f29ada85cc46"} Jul 17 02:08:36.418256 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:36.417909 2583 scope.go:117] "RemoveContainer" containerID="f569417e62a9ae65eabdd963541e4cb48fb5c7d67076851a3f0d06eb18a97d0b" Jul 17 02:08:36.418256 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:36.418174 2583 scope.go:117] "RemoveContainer" containerID="b8d7b1d4d5b3cd33c3a0cf5af1cc474eff602669806839283702f29ada85cc46" Jul 17 02:08:36.418417 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:08:36.418355 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp_odh-ai-gateway-infra(bafdb251-a59d-4b23-a42c-4543c0f14394)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" podUID="bafdb251-a59d-4b23-a42c-4543c0f14394" Jul 17 02:08:36.434666 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:36.434643 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp"] Jul 17 02:08:37.554288 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:37.554259 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" Jul 17 02:08:37.654867 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:37.654833 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafdb251-a59d-4b23-a42c-4543c0f14394-maas-api-service-ca\") pod \"bafdb251-a59d-4b23-a42c-4543c0f14394\" (UID: \"bafdb251-a59d-4b23-a42c-4543c0f14394\") " Jul 17 02:08:37.655226 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:37.655200 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bafdb251-a59d-4b23-a42c-4543c0f14394-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bafdb251-a59d-4b23-a42c-4543c0f14394" (UID: "bafdb251-a59d-4b23-a42c-4543c0f14394"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:08:37.755923 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:37.755838 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafdb251-a59d-4b23-a42c-4543c0f14394-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:08:38.430474 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:38.430441 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" Jul 17 02:08:38.430730 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:38.430443 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp" event={"ID":"bafdb251-a59d-4b23-a42c-4543c0f14394","Type":"ContainerDied","Data":"8d7c29d446c65e6e389a977ab385ab296b47e911152a04ce5aa3a6bb5b947434"} Jul 17 02:08:38.430730 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:38.430589 2583 scope.go:117] "RemoveContainer" containerID="b8d7b1d4d5b3cd33c3a0cf5af1cc474eff602669806839283702f29ada85cc46" Jul 17 02:08:38.454372 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:38.454338 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp"] Jul 17 02:08:38.459690 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:38.459667 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7122b7f-8k5kp"] Jul 17 02:08:39.167932 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:39.167903 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bafdb251-a59d-4b23-a42c-4543c0f14394" path="/var/lib/kubelet/pods/bafdb251-a59d-4b23-a42c-4543c0f14394/volumes" Jul 17 02:08:40.603155 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:40.603122 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:08:40.702748 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:40.702718 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:08:45.384063 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:45.384022 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:08:45.622710 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:45.622672 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:08:47.163123 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:47.163072 2583 scope.go:117] "RemoveContainer" containerID="8c52e250c86d79bcdbe427128d49be584df03c7de596688b2640b2dd75a7d29f" Jul 17 02:08:47.478235 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:47.478141 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" event={"ID":"72c6bad8-b808-4715-8769-56fcc21a303c","Type":"ContainerStarted","Data":"be81b404366fbf726af20002470dbd50f3ec4a076ecbd5d546d9dfcc4feddb04"} Jul 17 02:08:47.495186 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:47.495128 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" podStartSLOduration=19.495075769 podStartE2EDuration="19.495075769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:08:47.49407607 +0000 UTC m=+3284.889690923" watchObservedRunningTime="2026-07-17 02:08:47.495075769 +0000 UTC m=+3284.890690619" Jul 17 02:08:48.484339 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:48.484306 2583 generic.go:356] "Generic (PLEG): container finished" podID="72c6bad8-b808-4715-8769-56fcc21a303c" containerID="be81b404366fbf726af20002470dbd50f3ec4a076ecbd5d546d9dfcc4feddb04" exitCode=6 Jul 17 02:08:48.484762 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:48.484374 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" event={"ID":"72c6bad8-b808-4715-8769-56fcc21a303c","Type":"ContainerDied","Data":"be81b404366fbf726af20002470dbd50f3ec4a076ecbd5d546d9dfcc4feddb04"} Jul 17 02:08:48.484762 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:48.484435 2583 scope.go:117] "RemoveContainer" containerID="8c52e250c86d79bcdbe427128d49be584df03c7de596688b2640b2dd75a7d29f" Jul 17 02:08:48.484846 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:48.484780 2583 scope.go:117] "RemoveContainer" containerID="be81b404366fbf726af20002470dbd50f3ec4a076ecbd5d546d9dfcc4feddb04" Jul 17 02:08:48.484987 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:08:48.484969 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc_odh-ai-gateway-infra(72c6bad8-b808-4715-8769-56fcc21a303c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" podUID="72c6bad8-b808-4715-8769-56fcc21a303c" Jul 17 02:08:48.510773 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:48.510740 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc"] Jul 17 02:08:49.621612 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:49.621590 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" Jul 17 02:08:49.762381 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:49.762305 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c6bad8-b808-4715-8769-56fcc21a303c-maas-api-service-ca\") pod \"72c6bad8-b808-4715-8769-56fcc21a303c\" (UID: \"72c6bad8-b808-4715-8769-56fcc21a303c\") " Jul 17 02:08:49.762625 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:49.762602 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c6bad8-b808-4715-8769-56fcc21a303c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72c6bad8-b808-4715-8769-56fcc21a303c" (UID: "72c6bad8-b808-4715-8769-56fcc21a303c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:08:49.863668 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:49.863644 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c6bad8-b808-4715-8769-56fcc21a303c-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:08:50.496254 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:50.496206 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" Jul 17 02:08:50.496254 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:50.496240 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc" event={"ID":"72c6bad8-b808-4715-8769-56fcc21a303c","Type":"ContainerDied","Data":"9443900f6079f96232ead25a76424b2950259d14bb93e906504f01032ac3088d"} Jul 17 02:08:50.496477 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:50.496309 2583 scope.go:117] "RemoveContainer" containerID="be81b404366fbf726af20002470dbd50f3ec4a076ecbd5d546d9dfcc4feddb04" Jul 17 02:08:50.519568 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:50.519543 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc"] Jul 17 02:08:50.523445 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:50.523426 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-l6qtc"] Jul 17 02:08:51.172945 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:51.172912 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72c6bad8-b808-4715-8769-56fcc21a303c" path="/var/lib/kubelet/pods/72c6bad8-b808-4715-8769-56fcc21a303c/volumes" Jul 17 02:08:52.204223 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:52.204194 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56ddbf9bf4-smgsc_e07729fb-8c24-4db7-9030-45f84fc5fbef/copy-manifests/0.log" Jul 17 02:08:52.211720 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:52.211692 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56ddbf9bf4-smgsc_e07729fb-8c24-4db7-9030-45f84fc5fbef/manager/0.log" Jul 17 02:08:52.805939 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:52.805903 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737530-dxdf8_2563445a-1d64-4165-a849-a50edd24de77/cleanup/0.log" Jul 17 02:08:52.920912 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:52.920880 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737545-rdh5q_a4eacf93-c3e3-425c-9522-862425ddd03b/cleanup/0.log" Jul 17 02:08:53.029651 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:53.029621 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737560-qqs98_26d5bc47-89fc-4447-af12-be3281ad0001/cleanup/0.log" Jul 17 02:08:53.141200 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:53.141093 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-3413e7-vmrh8_50d34eef-a2de-4df8-aa17-5c2958a86ba6/revoke-keys/0.log" Jul 17 02:08:53.253587 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:53.253556 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-d8cc4a-fcsgg_9ad84ef6-4a2b-45ae-8d5f-9c1c3d062dfa/revoke-keys/0.log" Jul 17 02:08:53.368332 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:53.368310 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-089e15-pmjnz_e38dd9c9-2c0b-4681-9986-2c66d32ef99b/revoke-keys/0.log" Jul 17 02:08:53.481743 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:53.481661 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-ab7f72-j4cwg_dd6fc007-10a9-4bdc-89a3-8fff93799964/revoke-keys/0.log" Jul 17 02:08:53.594640 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:53.594613 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-8c5b7a14-8l9ng_1581115b-9eb5-4a33-8448-851186f870b5/revoke-keys/0.log" Jul 17 02:08:53.709440 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:53.709403 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-ac7a9695-vktkh_d79d5de9-ccdb-4365-a5b7-9cb8f21f53c8/revoke-keys/0.log" Jul 17 02:08:53.820830 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:53.820802 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-fd4ae7-s667b_6318e69c-61c9-413b-83bd-8588898b114b/revoke-keys/0.log" Jul 17 02:08:53.929731 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:53.929700 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-1f2720-dqbmr_cabdf966-a15a-428c-9e67-605df0935b0e/revoke-keys/0.log" Jul 17 02:08:56.667043 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:56.667011 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-87p9j_b6e3bd69-af0f-4c90-a537-b2cc0244e085/discovery/0.log" Jul 17 02:08:56.782117 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:08:56.782067 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f596cfdd-n7zfw_2535c4a4-5a00-4efe-bed9-dd567cbe576f/kube-auth-proxy/0.log" Jul 17 02:09:03.319178 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:03.319152 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 02:09:03.323631 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:03.323610 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2w6g5_ba8f230b-ad43-4811-9a71-a33c5fb4a918/ovn-acl-logging/0.log" Jul 17 02:09:04.034726 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.034695 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:09:04.136645 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.136605 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr"] Jul 17 02:09:04.136986 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.136970 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cdb30c4-34c8-41f5-a050-763a0be5954f" containerName="revoke-keys" Jul 17 02:09:04.137071 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.136989 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cdb30c4-34c8-41f5-a050-763a0be5954f" containerName="revoke-keys" Jul 17 02:09:04.137071 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137006 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3225281f-d7ca-4f64-b363-6f330d6c6a4f" containerName="maas-api" Jul 17 02:09:04.137071 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137015 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3225281f-d7ca-4f64-b363-6f330d6c6a4f" containerName="maas-api" Jul 17 02:09:04.137071 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137032 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78733375-7e33-424e-9401-3457a93fbcb9" containerName="revoke-keys" Jul 17 02:09:04.137071 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137040 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="78733375-7e33-424e-9401-3457a93fbcb9" containerName="revoke-keys" Jul 17 02:09:04.137071 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137050 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c6bad8-b808-4715-8769-56fcc21a303c" containerName="revoke-keys" Jul 17 02:09:04.137071 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137058 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c6bad8-b808-4715-8769-56fcc21a303c" containerName="revoke-keys" Jul 17 02:09:04.137071 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137071 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78733375-7e33-424e-9401-3457a93fbcb9" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137079 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="78733375-7e33-424e-9401-3457a93fbcb9" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137088 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bafdb251-a59d-4b23-a42c-4543c0f14394" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137114 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bafdb251-a59d-4b23-a42c-4543c0f14394" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137125 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c6bad8-b808-4715-8769-56fcc21a303c" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137133 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c6bad8-b808-4715-8769-56fcc21a303c" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137154 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cdb30c4-34c8-41f5-a050-763a0be5954f" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137162 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cdb30c4-34c8-41f5-a050-763a0be5954f" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137177 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c6bad8-b808-4715-8769-56fcc21a303c" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137184 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c6bad8-b808-4715-8769-56fcc21a303c" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137198 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bafdb251-a59d-4b23-a42c-4543c0f14394" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137205 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bafdb251-a59d-4b23-a42c-4543c0f14394" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137216 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78733375-7e33-424e-9401-3457a93fbcb9" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137224 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="78733375-7e33-424e-9401-3457a93fbcb9" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137306 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bafdb251-a59d-4b23-a42c-4543c0f14394" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137319 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cdb30c4-34c8-41f5-a050-763a0be5954f" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137328 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cdb30c4-34c8-41f5-a050-763a0be5954f" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137337 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="72c6bad8-b808-4715-8769-56fcc21a303c" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137345 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="72c6bad8-b808-4715-8769-56fcc21a303c" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137354 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3225281f-d7ca-4f64-b363-6f330d6c6a4f" containerName="maas-api" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137373 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="78733375-7e33-424e-9401-3457a93fbcb9" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137381 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="78733375-7e33-424e-9401-3457a93fbcb9" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137391 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cdb30c4-34c8-41f5-a050-763a0be5954f" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137400 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="78733375-7e33-424e-9401-3457a93fbcb9" containerName="revoke-keys" Jul 17 02:09:04.137515 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.137408 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bafdb251-a59d-4b23-a42c-4543c0f14394" containerName="revoke-keys" Jul 17 02:09:04.143595 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.143561 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" Jul 17 02:09:04.148442 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.148409 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 02:09:04.148554 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.148497 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 02:09:04.154089 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.154063 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr"] Jul 17 02:09:04.163547 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.163520 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:09:04.164657 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.164632 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673c229d-396e-42a6-b09c-7df7c17c2e96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr\" (UID: \"673c229d-396e-42a6-b09c-7df7c17c2e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" Jul 17 02:09:04.265007 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.264975 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673c229d-396e-42a6-b09c-7df7c17c2e96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr\" (UID: \"673c229d-396e-42a6-b09c-7df7c17c2e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" Jul 17 02:09:04.265618 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.265599 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673c229d-396e-42a6-b09c-7df7c17c2e96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr\" (UID: \"673c229d-396e-42a6-b09c-7df7c17c2e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" Jul 17 02:09:04.455906 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.455802 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" Jul 17 02:09:04.597192 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:04.597169 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr"] Jul 17 02:09:05.578772 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:05.578736 2583 generic.go:356] "Generic (PLEG): container finished" podID="673c229d-396e-42a6-b09c-7df7c17c2e96" containerID="a0b8ad599f1c88f11272968f84ce622c7e2ac4d6b2903015ff6dfbee463a28ad" exitCode=6 Jul 17 02:09:05.579225 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:05.578828 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" event={"ID":"673c229d-396e-42a6-b09c-7df7c17c2e96","Type":"ContainerDied","Data":"a0b8ad599f1c88f11272968f84ce622c7e2ac4d6b2903015ff6dfbee463a28ad"} Jul 17 02:09:05.579225 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:05.578894 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" event={"ID":"673c229d-396e-42a6-b09c-7df7c17c2e96","Type":"ContainerStarted","Data":"d7a95d5e9e3240121f5f486643f2c2a4b1484c01fe0ce7ced517d08860af32d3"} Jul 17 02:09:05.579225 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:05.579004 2583 scope.go:117] "RemoveContainer" containerID="a0b8ad599f1c88f11272968f84ce622c7e2ac4d6b2903015ff6dfbee463a28ad" Jul 17 02:09:06.586087 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.586058 2583 generic.go:356] "Generic (PLEG): container finished" podID="673c229d-396e-42a6-b09c-7df7c17c2e96" containerID="c276a30afc4249b6fb90ea11192d511c1376365b566eb6fcdfb44fb7bd0047a3" exitCode=6 Jul 17 02:09:06.586499 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.586155 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" event={"ID":"673c229d-396e-42a6-b09c-7df7c17c2e96","Type":"ContainerDied","Data":"c276a30afc4249b6fb90ea11192d511c1376365b566eb6fcdfb44fb7bd0047a3"} Jul 17 02:09:06.586499 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.586219 2583 scope.go:117] "RemoveContainer" containerID="a0b8ad599f1c88f11272968f84ce622c7e2ac4d6b2903015ff6dfbee463a28ad" Jul 17 02:09:06.586499 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.586441 2583 scope.go:117] "RemoveContainer" containerID="c276a30afc4249b6fb90ea11192d511c1376365b566eb6fcdfb44fb7bd0047a3" Jul 17 02:09:06.586668 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:09:06.586647 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr_odh-ai-gateway-infra(673c229d-396e-42a6-b09c-7df7c17c2e96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" podUID="673c229d-396e-42a6-b09c-7df7c17c2e96" Jul 17 02:09:06.687822 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.687789 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv"] Jul 17 02:09:06.688136 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.688123 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cdb30c4-34c8-41f5-a050-763a0be5954f" containerName="revoke-keys" Jul 17 02:09:06.688182 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.688137 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cdb30c4-34c8-41f5-a050-763a0be5954f" containerName="revoke-keys" Jul 17 02:09:06.688182 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.688153 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bafdb251-a59d-4b23-a42c-4543c0f14394" containerName="revoke-keys" Jul 17 02:09:06.688182 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.688158 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bafdb251-a59d-4b23-a42c-4543c0f14394" containerName="revoke-keys" Jul 17 02:09:06.688278 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.688218 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="72c6bad8-b808-4715-8769-56fcc21a303c" containerName="revoke-keys" Jul 17 02:09:06.688278 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.688228 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bafdb251-a59d-4b23-a42c-4543c0f14394" containerName="revoke-keys" Jul 17 02:09:06.692309 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.692292 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" Jul 17 02:09:06.700117 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.700073 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv"] Jul 17 02:09:06.782883 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.782841 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9882f236-bd10-4e17-bf1c-458c6c0ba5b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv\" (UID: \"9882f236-bd10-4e17-bf1c-458c6c0ba5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" Jul 17 02:09:06.884057 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.883970 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9882f236-bd10-4e17-bf1c-458c6c0ba5b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv\" (UID: \"9882f236-bd10-4e17-bf1c-458c6c0ba5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" Jul 17 02:09:06.884595 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.884577 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9882f236-bd10-4e17-bf1c-458c6c0ba5b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv\" (UID: \"9882f236-bd10-4e17-bf1c-458c6c0ba5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" Jul 17 02:09:06.992109 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:06.992070 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-59v85_71f7c662-3fa3-4691-ad0a-34f6bfdc4d9a/global-pull-secret-syncer/0.log" Jul 17 02:09:07.004042 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:07.004016 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" Jul 17 02:09:07.134505 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:07.134428 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" Jul 17 02:09:07.141438 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:07.141263 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv"] Jul 17 02:09:07.143503 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:09:07.143482 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9882f236_bd10_4e17_bf1c_458c6c0ba5b0.slice/crio-eeec29793e6ea4d6252de8184e6e0e13cf09c86e9662ab24868c53b7a129bfff WatchSource:0}: Error finding container eeec29793e6ea4d6252de8184e6e0e13cf09c86e9662ab24868c53b7a129bfff: Status 404 returned error can't find the container with id eeec29793e6ea4d6252de8184e6e0e13cf09c86e9662ab24868c53b7a129bfff Jul 17 02:09:07.256944 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:07.256916 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-44.ec2.internal_031589d9c1527982687d362e9909b534/haproxy/0.log" Jul 17 02:09:07.593624 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:07.593600 2583 scope.go:117] "RemoveContainer" containerID="c276a30afc4249b6fb90ea11192d511c1376365b566eb6fcdfb44fb7bd0047a3" Jul 17 02:09:07.594135 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:09:07.593824 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr_odh-ai-gateway-infra(673c229d-396e-42a6-b09c-7df7c17c2e96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" podUID="673c229d-396e-42a6-b09c-7df7c17c2e96" Jul 17 02:09:07.594948 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:07.594919 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" event={"ID":"9882f236-bd10-4e17-bf1c-458c6c0ba5b0","Type":"ContainerStarted","Data":"12599f6438b736b79a6bfb1846063f33a10c4c5fca1d7d8a90cc76de85a5033f"} Jul 17 02:09:07.595058 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:07.594968 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" event={"ID":"9882f236-bd10-4e17-bf1c-458c6c0ba5b0","Type":"ContainerStarted","Data":"eeec29793e6ea4d6252de8184e6e0e13cf09c86e9662ab24868c53b7a129bfff"} Jul 17 02:09:07.625459 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:07.625406 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" podStartSLOduration=1.6253914900000002 podStartE2EDuration="1.62539149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:09:07.622853403 +0000 UTC m=+3305.018468254" watchObservedRunningTime="2026-07-17 02:09:07.62539149 +0000 UTC m=+3305.021006341" Jul 17 02:09:08.602248 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:08.602215 2583 generic.go:356] "Generic (PLEG): container finished" podID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" containerID="12599f6438b736b79a6bfb1846063f33a10c4c5fca1d7d8a90cc76de85a5033f" exitCode=6 Jul 17 02:09:08.602634 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:08.602303 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" event={"ID":"9882f236-bd10-4e17-bf1c-458c6c0ba5b0","Type":"ContainerDied","Data":"12599f6438b736b79a6bfb1846063f33a10c4c5fca1d7d8a90cc76de85a5033f"} Jul 17 02:09:08.602634 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:08.602461 2583 scope.go:117] "RemoveContainer" containerID="12599f6438b736b79a6bfb1846063f33a10c4c5fca1d7d8a90cc76de85a5033f" Jul 17 02:09:09.608810 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:09.608777 2583 generic.go:356] "Generic (PLEG): container finished" podID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" containerID="3af16dbe931fa0560318cad9fef7da07942eaff75c5b8fe1c97f46e85b9cc307" exitCode=6 Jul 17 02:09:09.609382 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:09.608875 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" event={"ID":"9882f236-bd10-4e17-bf1c-458c6c0ba5b0","Type":"ContainerDied","Data":"3af16dbe931fa0560318cad9fef7da07942eaff75c5b8fe1c97f46e85b9cc307"} Jul 17 02:09:09.609382 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:09.608950 2583 scope.go:117] "RemoveContainer" containerID="12599f6438b736b79a6bfb1846063f33a10c4c5fca1d7d8a90cc76de85a5033f" Jul 17 02:09:09.609382 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:09.609137 2583 scope.go:117] "RemoveContainer" containerID="3af16dbe931fa0560318cad9fef7da07942eaff75c5b8fe1c97f46e85b9cc307" Jul 17 02:09:09.609382 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:09:09.609366 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv_odh-ai-gateway-infra(9882f236-bd10-4e17-bf1c-458c6c0ba5b0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" podUID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" Jul 17 02:09:10.617022 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:10.616994 2583 scope.go:117] "RemoveContainer" containerID="3af16dbe931fa0560318cad9fef7da07942eaff75c5b8fe1c97f46e85b9cc307" Jul 17 02:09:10.617420 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:09:10.617196 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv_odh-ai-gateway-infra(9882f236-bd10-4e17-bf1c-458c6c0ba5b0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" podUID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" Jul 17 02:09:12.621477 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:12.621437 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:09:14.794944 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:14.794913 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-rcr67_f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb/kube-state-metrics/0.log" Jul 17 02:09:14.810624 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:14.810597 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-rcr67_f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb/kube-rbac-proxy-main/0.log" Jul 17 02:09:14.831508 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:14.831459 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-rcr67_f2cb6ab3-36a7-4db1-9c37-4db4da6b73bb/kube-rbac-proxy-self/0.log" Jul 17 02:09:14.858874 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:14.858851 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-66c9bc58bf-q57jn_a6349637-3750-4429-a596-65de3a5824b6/metrics-server/0.log" Jul 17 02:09:14.891029 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:14.891001 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-cm6lr_c9c7d960-19bd-410d-aab3-f811fc713b28/monitoring-plugin/0.log" Jul 17 02:09:14.919819 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:14.919784 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kvmg6_74496ef6-aa60-4933-ba21-7fc27f0f2549/node-exporter/0.log" Jul 17 02:09:14.941496 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:14.941469 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kvmg6_74496ef6-aa60-4933-ba21-7fc27f0f2549/kube-rbac-proxy/0.log" Jul 17 02:09:14.961885 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:14.961855 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kvmg6_74496ef6-aa60-4933-ba21-7fc27f0f2549/init-textfile/0.log" Jul 17 02:09:15.139622 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.139537 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-nfzqk_5cf44875-46d9-4e1a-b594-87fd5ebe44b0/kube-rbac-proxy-main/0.log" Jul 17 02:09:15.163581 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.163552 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-nfzqk_5cf44875-46d9-4e1a-b594-87fd5ebe44b0/kube-rbac-proxy-self/0.log" Jul 17 02:09:15.184082 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.184049 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-nfzqk_5cf44875-46d9-4e1a-b594-87fd5ebe44b0/openshift-state-metrics/0.log" Jul 17 02:09:15.228814 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.228785 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dc7e42c6-77b7-46f6-ba5c-ddb640c19830/prometheus/0.log" Jul 17 02:09:15.249723 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.249696 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dc7e42c6-77b7-46f6-ba5c-ddb640c19830/config-reloader/0.log" Jul 17 02:09:15.270463 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.270428 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dc7e42c6-77b7-46f6-ba5c-ddb640c19830/thanos-sidecar/0.log" Jul 17 02:09:15.295358 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.295326 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dc7e42c6-77b7-46f6-ba5c-ddb640c19830/kube-rbac-proxy-web/0.log" Jul 17 02:09:15.315609 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.315586 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dc7e42c6-77b7-46f6-ba5c-ddb640c19830/kube-rbac-proxy/0.log" Jul 17 02:09:15.335228 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.335197 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dc7e42c6-77b7-46f6-ba5c-ddb640c19830/kube-rbac-proxy-thanos/0.log" Jul 17 02:09:15.356674 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.356648 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dc7e42c6-77b7-46f6-ba5c-ddb640c19830/init-config-reloader/0.log" Jul 17 02:09:15.558083 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.558058 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6449bf7ffd-plq9l_6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7/thanos-query/0.log" Jul 17 02:09:15.578395 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.578361 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6449bf7ffd-plq9l_6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7/kube-rbac-proxy-web/0.log" Jul 17 02:09:15.601931 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.601899 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6449bf7ffd-plq9l_6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7/kube-rbac-proxy/0.log" Jul 17 02:09:15.623710 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.623684 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6449bf7ffd-plq9l_6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7/prom-label-proxy/0.log" Jul 17 02:09:15.648482 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.648453 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6449bf7ffd-plq9l_6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7/kube-rbac-proxy-rules/0.log" Jul 17 02:09:15.671227 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:15.671193 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6449bf7ffd-plq9l_6ae2abd7-4ab7-4903-9b72-9d19e9dc31f7/kube-rbac-proxy-metrics/0.log" Jul 17 02:09:17.234468 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.234435 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh"] Jul 17 02:09:17.238087 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.238070 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.240409 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.240387 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l2j4k\"/\"kube-root-ca.crt\"" Jul 17 02:09:17.241304 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.241284 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l2j4k\"/\"openshift-service-ca.crt\"" Jul 17 02:09:17.241391 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.241284 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l2j4k\"/\"default-dockercfg-5srgd\"" Jul 17 02:09:17.248846 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.248822 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh"] Jul 17 02:09:17.272121 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.272065 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e223a30-b69a-48d0-9e4d-79bafa2946b7-sys\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.272295 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.272177 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e223a30-b69a-48d0-9e4d-79bafa2946b7-podres\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.272349 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.272327 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e223a30-b69a-48d0-9e4d-79bafa2946b7-proc\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.272442 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.272428 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e223a30-b69a-48d0-9e4d-79bafa2946b7-lib-modules\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.272476 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.272453 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b4tk\" (UniqueName: \"kubernetes.io/projected/7e223a30-b69a-48d0-9e4d-79bafa2946b7-kube-api-access-9b4tk\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.373223 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.373179 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e223a30-b69a-48d0-9e4d-79bafa2946b7-podres\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.373223 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.373230 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e223a30-b69a-48d0-9e4d-79bafa2946b7-proc\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.373485 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.373267 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e223a30-b69a-48d0-9e4d-79bafa2946b7-lib-modules\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.373485 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.373293 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b4tk\" (UniqueName: \"kubernetes.io/projected/7e223a30-b69a-48d0-9e4d-79bafa2946b7-kube-api-access-9b4tk\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.373485 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.373340 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e223a30-b69a-48d0-9e4d-79bafa2946b7-sys\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.373485 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.373391 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e223a30-b69a-48d0-9e4d-79bafa2946b7-proc\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.373485 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.373416 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e223a30-b69a-48d0-9e4d-79bafa2946b7-sys\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.373485 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.373391 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e223a30-b69a-48d0-9e4d-79bafa2946b7-podres\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.373485 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.373436 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e223a30-b69a-48d0-9e4d-79bafa2946b7-lib-modules\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.381879 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.381848 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b4tk\" (UniqueName: \"kubernetes.io/projected/7e223a30-b69a-48d0-9e4d-79bafa2946b7-kube-api-access-9b4tk\") pod \"perf-node-gather-daemonset-tnfhh\" (UID: \"7e223a30-b69a-48d0-9e4d-79bafa2946b7\") " pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.549188 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.549151 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:17.686988 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:17.686961 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh"] Jul 17 02:09:17.688897 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:09:17.688868 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7e223a30_b69a_48d0_9e4d_79bafa2946b7.slice/crio-302ed7d80cbb662f626b17d1eadb17d0083c39c5ecf6d2ba50d245a7f5ade2cd WatchSource:0}: Error finding container 302ed7d80cbb662f626b17d1eadb17d0083c39c5ecf6d2ba50d245a7f5ade2cd: Status 404 returned error can't find the container with id 302ed7d80cbb662f626b17d1eadb17d0083c39c5ecf6d2ba50d245a7f5ade2cd Jul 17 02:09:18.655113 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:18.655061 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" event={"ID":"7e223a30-b69a-48d0-9e4d-79bafa2946b7","Type":"ContainerStarted","Data":"0379e0ecbb76f9b233783cba05ef50a552eaf16ddc96118a37a0529793840e9f"} Jul 17 02:09:18.655507 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:18.655158 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" event={"ID":"7e223a30-b69a-48d0-9e4d-79bafa2946b7","Type":"ContainerStarted","Data":"302ed7d80cbb662f626b17d1eadb17d0083c39c5ecf6d2ba50d245a7f5ade2cd"} Jul 17 02:09:18.655507 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:18.655197 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:18.670029 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:18.669981 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" podStartSLOduration=1.669965839 podStartE2EDuration="1.669965839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 02:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:09:18.668866916 +0000 UTC m=+3316.064481771" watchObservedRunningTime="2026-07-17 02:09:18.669965839 +0000 UTC m=+3316.065580690" Jul 17 02:09:19.742223 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:19.742197 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xjl9w_dfeaf340-95bf-443c-a180-f6b883794466/dns/0.log" Jul 17 02:09:19.763540 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:19.763507 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xjl9w_dfeaf340-95bf-443c-a180-f6b883794466/kube-rbac-proxy/0.log" Jul 17 02:09:19.826890 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:19.826857 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pzn6s_2f571d61-7f50-41c1-aabc-19b67211094d/dns-node-resolver/0.log" Jul 17 02:09:20.409290 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:20.409260 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lnzqz_6fafa6fe-e9ba-4f2e-8043-8e3413afa6de/node-ca/0.log" Jul 17 02:09:21.163211 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:21.163183 2583 scope.go:117] "RemoveContainer" containerID="c276a30afc4249b6fb90ea11192d511c1376365b566eb6fcdfb44fb7bd0047a3" Jul 17 02:09:21.559751 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:21.559724 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-87p9j_b6e3bd69-af0f-4c90-a537-b2cc0244e085/discovery/0.log" Jul 17 02:09:21.592464 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:21.592436 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f596cfdd-n7zfw_2535c4a4-5a00-4efe-bed9-dd567cbe576f/kube-auth-proxy/0.log" Jul 17 02:09:21.672083 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:21.672047 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" event={"ID":"673c229d-396e-42a6-b09c-7df7c17c2e96","Type":"ContainerStarted","Data":"439502425d5b2e449ad4d2795328ebec1802e389588268a7eb4d4c04e8971b8e"} Jul 17 02:09:21.689424 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:21.689375 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" podStartSLOduration=17.689359243 podStartE2EDuration="17.689359243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:09:21.688195315 +0000 UTC m=+3319.083810167" watchObservedRunningTime="2026-07-17 02:09:21.689359243 +0000 UTC m=+3319.084974093" Jul 17 02:09:22.162887 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:22.162802 2583 scope.go:117] "RemoveContainer" containerID="3af16dbe931fa0560318cad9fef7da07942eaff75c5b8fe1c97f46e85b9cc307" Jul 17 02:09:22.564388 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:22.564353 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qhmb7_a6a8c41d-9ea9-4ec3-a61c-9911f9c0ff83/serve-healthcheck-canary/0.log" Jul 17 02:09:22.678674 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:22.678592 2583 generic.go:356] "Generic (PLEG): container finished" podID="673c229d-396e-42a6-b09c-7df7c17c2e96" containerID="439502425d5b2e449ad4d2795328ebec1802e389588268a7eb4d4c04e8971b8e" exitCode=6 Jul 17 02:09:22.678674 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:22.678661 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" event={"ID":"673c229d-396e-42a6-b09c-7df7c17c2e96","Type":"ContainerDied","Data":"439502425d5b2e449ad4d2795328ebec1802e389588268a7eb4d4c04e8971b8e"} Jul 17 02:09:22.678909 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:22.678722 2583 scope.go:117] "RemoveContainer" containerID="c276a30afc4249b6fb90ea11192d511c1376365b566eb6fcdfb44fb7bd0047a3" Jul 17 02:09:22.679041 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:22.679026 2583 scope.go:117] "RemoveContainer" containerID="439502425d5b2e449ad4d2795328ebec1802e389588268a7eb4d4c04e8971b8e" Jul 17 02:09:22.679307 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:09:22.679286 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr_odh-ai-gateway-infra(673c229d-396e-42a6-b09c-7df7c17c2e96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" podUID="673c229d-396e-42a6-b09c-7df7c17c2e96" Jul 17 02:09:22.680591 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:22.680570 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" event={"ID":"9882f236-bd10-4e17-bf1c-458c6c0ba5b0","Type":"ContainerStarted","Data":"0ddae9f6e0a8f9d2e54281c864f6f302d595a78ac13d74423a618eaddc665dcb"} Jul 17 02:09:22.699406 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:22.699384 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr"] Jul 17 02:09:23.148566 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:23.148540 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q9bjq_e9ee2220-5d23-4a71-b795-7ff9e1a57868/kube-rbac-proxy/0.log" Jul 17 02:09:23.170545 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:23.170522 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q9bjq_e9ee2220-5d23-4a71-b795-7ff9e1a57868/exporter/0.log" Jul 17 02:09:23.192744 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:23.192720 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q9bjq_e9ee2220-5d23-4a71-b795-7ff9e1a57868/extractor/0.log" Jul 17 02:09:23.688675 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:23.688646 2583 generic.go:356] "Generic (PLEG): container finished" podID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" containerID="0ddae9f6e0a8f9d2e54281c864f6f302d595a78ac13d74423a618eaddc665dcb" exitCode=6 Jul 17 02:09:23.689125 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:23.688699 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" event={"ID":"9882f236-bd10-4e17-bf1c-458c6c0ba5b0","Type":"ContainerDied","Data":"0ddae9f6e0a8f9d2e54281c864f6f302d595a78ac13d74423a618eaddc665dcb"} Jul 17 02:09:23.689125 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:23.688746 2583 scope.go:117] "RemoveContainer" containerID="3af16dbe931fa0560318cad9fef7da07942eaff75c5b8fe1c97f46e85b9cc307" Jul 17 02:09:23.689125 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:23.689091 2583 scope.go:117] "RemoveContainer" containerID="0ddae9f6e0a8f9d2e54281c864f6f302d595a78ac13d74423a618eaddc665dcb" Jul 17 02:09:23.689376 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:09:23.689353 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv_odh-ai-gateway-infra(9882f236-bd10-4e17-bf1c-458c6c0ba5b0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" podUID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" Jul 17 02:09:23.717608 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:23.717559 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv"] Jul 17 02:09:23.829641 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:23.829612 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" Jul 17 02:09:23.926811 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:23.926766 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673c229d-396e-42a6-b09c-7df7c17c2e96-maas-api-service-ca\") pod \"673c229d-396e-42a6-b09c-7df7c17c2e96\" (UID: \"673c229d-396e-42a6-b09c-7df7c17c2e96\") " Jul 17 02:09:23.927190 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:23.927165 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673c229d-396e-42a6-b09c-7df7c17c2e96-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "673c229d-396e-42a6-b09c-7df7c17c2e96" (UID: "673c229d-396e-42a6-b09c-7df7c17c2e96"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:09:24.027692 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:24.027606 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673c229d-396e-42a6-b09c-7df7c17c2e96-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:09:24.670068 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:24.670043 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l2j4k/perf-node-gather-daemonset-tnfhh" Jul 17 02:09:24.695463 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:24.695434 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" Jul 17 02:09:24.695895 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:24.695473 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr" event={"ID":"673c229d-396e-42a6-b09c-7df7c17c2e96","Type":"ContainerDied","Data":"d7a95d5e9e3240121f5f486643f2c2a4b1484c01fe0ce7ced517d08860af32d3"} Jul 17 02:09:24.695895 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:24.695551 2583 scope.go:117] "RemoveContainer" containerID="439502425d5b2e449ad4d2795328ebec1802e389588268a7eb4d4c04e8971b8e" Jul 17 02:09:24.733523 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:24.733489 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr"] Jul 17 02:09:24.737130 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:24.737074 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e02ab63-4h2vr"] Jul 17 02:09:24.831906 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:24.831884 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" Jul 17 02:09:24.935897 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:24.935809 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9882f236-bd10-4e17-bf1c-458c6c0ba5b0-maas-api-service-ca\") pod \"9882f236-bd10-4e17-bf1c-458c6c0ba5b0\" (UID: \"9882f236-bd10-4e17-bf1c-458c6c0ba5b0\") " Jul 17 02:09:24.936194 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:24.936170 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9882f236-bd10-4e17-bf1c-458c6c0ba5b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9882f236-bd10-4e17-bf1c-458c6c0ba5b0" (UID: "9882f236-bd10-4e17-bf1c-458c6c0ba5b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 02:09:25.037365 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:25.037318 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9882f236-bd10-4e17-bf1c-458c6c0ba5b0-maas-api-service-ca\") on node \"ip-10-0-128-44.ec2.internal\" DevicePath \"\"" Jul 17 02:09:25.168117 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:25.168076 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="673c229d-396e-42a6-b09c-7df7c17c2e96" path="/var/lib/kubelet/pods/673c229d-396e-42a6-b09c-7df7c17c2e96/volumes" Jul 17 02:09:25.703995 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:25.703967 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" Jul 17 02:09:25.704512 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:25.704006 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv" event={"ID":"9882f236-bd10-4e17-bf1c-458c6c0ba5b0","Type":"ContainerDied","Data":"eeec29793e6ea4d6252de8184e6e0e13cf09c86e9662ab24868c53b7a129bfff"} Jul 17 02:09:25.704512 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:25.704064 2583 scope.go:117] "RemoveContainer" containerID="0ddae9f6e0a8f9d2e54281c864f6f302d595a78ac13d74423a618eaddc665dcb" Jul 17 02:09:25.721132 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:25.721092 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv"] Jul 17 02:09:25.724498 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:25.724475 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ef2c65d-jxvdv"] Jul 17 02:09:26.120443 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:26.120417 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56ddbf9bf4-smgsc_e07729fb-8c24-4db7-9030-45f84fc5fbef/manager/0.log" Jul 17 02:09:26.143914 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:26.143890 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56ddbf9bf4-smgsc_e07729fb-8c24-4db7-9030-45f84fc5fbef/copy-manifests/0.log" Jul 17 02:09:27.167347 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:27.167318 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" path="/var/lib/kubelet/pods/9882f236-bd10-4e17-bf1c-458c6c0ba5b0/volumes" Jul 17 02:09:27.587018 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:27.586983 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-66ffdbf46d-zk9p7_78792b02-f8d0-47f8-a104-644d97cf3d6a/manager/0.log" Jul 17 02:09:27.615243 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:27.615212 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-trfjl_b71eb19d-c886-4c3e-a52b-da0f09282c3d/openshift-lws-operator/0.log" Jul 17 02:09:28.057388 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.057350 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-87p9j_b6e3bd69-af0f-4c90-a537-b2cc0244e085/discovery/0.log" Jul 17 02:09:28.067388 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.067353 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f596cfdd-n7zfw_2535c4a4-5a00-4efe-bed9-dd567cbe576f/kube-auth-proxy/0.log" Jul 17 02:09:28.709655 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.709626 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l"] Jul 17 02:09:28.709998 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.709959 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" containerName="revoke-keys" Jul 17 02:09:28.709998 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.709970 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" containerName="revoke-keys" Jul 17 02:09:28.709998 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.709978 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" containerName="revoke-keys" Jul 17 02:09:28.709998 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.709984 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" containerName="revoke-keys" Jul 17 02:09:28.709998 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.709994 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" containerName="revoke-keys" Jul 17 02:09:28.709998 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.709999 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" containerName="revoke-keys" Jul 17 02:09:28.710218 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.710008 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673c229d-396e-42a6-b09c-7df7c17c2e96" containerName="revoke-keys" Jul 17 02:09:28.710218 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.710013 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="673c229d-396e-42a6-b09c-7df7c17c2e96" containerName="revoke-keys" Jul 17 02:09:28.710218 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.710027 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673c229d-396e-42a6-b09c-7df7c17c2e96" containerName="revoke-keys" Jul 17 02:09:28.710218 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.710032 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="673c229d-396e-42a6-b09c-7df7c17c2e96" containerName="revoke-keys" Jul 17 02:09:28.710218 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.710091 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="673c229d-396e-42a6-b09c-7df7c17c2e96" containerName="revoke-keys" Jul 17 02:09:28.710218 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.710121 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" containerName="revoke-keys" Jul 17 02:09:28.710218 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.710128 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9882f236-bd10-4e17-bf1c-458c6c0ba5b0" containerName="revoke-keys" Jul 17 02:09:28.710218 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.710134 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="673c229d-396e-42a6-b09c-7df7c17c2e96" containerName="revoke-keys" Jul 17 02:09:28.714182 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.714163 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l" Jul 17 02:09:28.718482 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.718453 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 02:09:28.718652 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.718636 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qvpl\"" Jul 17 02:09:28.720037 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.720015 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l"] Jul 17 02:09:28.768386 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.768336 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261ccbf9-90b3-4771-99f0-cba64665bda8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l\" (UID: \"261ccbf9-90b3-4771-99f0-cba64665bda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l" Jul 17 02:09:28.869347 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.869311 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261ccbf9-90b3-4771-99f0-cba64665bda8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l\" (UID: \"261ccbf9-90b3-4771-99f0-cba64665bda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l" Jul 17 02:09:28.869954 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:28.869927 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261ccbf9-90b3-4771-99f0-cba64665bda8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l\" (UID: \"261ccbf9-90b3-4771-99f0-cba64665bda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l" Jul 17 02:09:29.027129 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:29.027004 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l" Jul 17 02:09:29.166300 ip-10-0-128-44 kubenswrapper[2583]: W0717 02:09:29.166274 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261ccbf9_90b3_4771_99f0_cba64665bda8.slice/crio-8e6f9f42a10e950f59f32c21883a50ee2b21fbe5e8fc02c7d0017d1bdc70db87 WatchSource:0}: Error finding container 8e6f9f42a10e950f59f32c21883a50ee2b21fbe5e8fc02c7d0017d1bdc70db87: Status 404 returned error can't find the container with id 8e6f9f42a10e950f59f32c21883a50ee2b21fbe5e8fc02c7d0017d1bdc70db87 Jul 17 02:09:29.169818 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:29.169799 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l"] Jul 17 02:09:29.172452 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:29.172426 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:09:29.222963 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:29.222928 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g4pc2_aa0e084d-6d85-492d-a50a-cf1e654d04a9/konnectivity-agent/0.log" limit=1024 Jul 17 02:09:29.603974 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:29.603889 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-87p9j_b6e3bd69-af0f-4c90-a537-b2cc0244e085/discovery/0.log" Jul 17 02:09:29.615792 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:29.615761 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f596cfdd-n7zfw_2535c4a4-5a00-4efe-bed9-dd567cbe576f/kube-auth-proxy/0.log" Jul 17 02:09:29.729837 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:29.729799 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l" event={"ID":"261ccbf9-90b3-4771-99f0-cba64665bda8","Type":"ContainerStarted","Data":"65df5020235f77952883e3703fcc11d89e4d84d409bd4cdd1150995705f4bd90"} Jul 17 02:09:29.730329 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:29.729868 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l" event={"ID":"261ccbf9-90b3-4771-99f0-cba64665bda8","Type":"ContainerStarted","Data":"8e6f9f42a10e950f59f32c21883a50ee2b21fbe5e8fc02c7d0017d1bdc70db87"} Jul 17 02:09:29.748267 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:29.748211 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l" podStartSLOduration=1.748197082 podStartE2EDuration="1.748197082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 02:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 02:09:29.745873847 +0000 UTC m=+3327.141488699" watchObservedRunningTime="2026-07-17 02:09:29.748197082 +0000 UTC m=+3327.143811932" Jul 17 02:09:30.741500 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:30.741461 2583 generic.go:356] "Generic (PLEG): container finished" podID="261ccbf9-90b3-4771-99f0-cba64665bda8" containerID="65df5020235f77952883e3703fcc11d89e4d84d409bd4cdd1150995705f4bd90" exitCode=6 Jul 17 02:09:30.741980 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:30.741519 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l" event={"ID":"261ccbf9-90b3-4771-99f0-cba64665bda8","Type":"ContainerDied","Data":"65df5020235f77952883e3703fcc11d89e4d84d409bd4cdd1150995705f4bd90"} Jul 17 02:09:30.741980 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:30.741810 2583 scope.go:117] "RemoveContainer" containerID="65df5020235f77952883e3703fcc11d89e4d84d409bd4cdd1150995705f4bd90" Jul 17 02:09:31.748866 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:31.748771 2583 generic.go:356] "Generic (PLEG): container finished" podID="261ccbf9-90b3-4771-99f0-cba64665bda8" containerID="c80b1051b012a691ed903aca645affbac8351a9c690aba7c0385b22fea171339" exitCode=6 Jul 17 02:09:31.748866 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:31.748816 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l" event={"ID":"261ccbf9-90b3-4771-99f0-cba64665bda8","Type":"ContainerDied","Data":"c80b1051b012a691ed903aca645affbac8351a9c690aba7c0385b22fea171339"} Jul 17 02:09:31.749363 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:31.748877 2583 scope.go:117] "RemoveContainer" containerID="65df5020235f77952883e3703fcc11d89e4d84d409bd4cdd1150995705f4bd90" Jul 17 02:09:31.749363 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:31.749083 2583 scope.go:117] "RemoveContainer" containerID="c80b1051b012a691ed903aca645affbac8351a9c690aba7c0385b22fea171339" Jul 17 02:09:31.749363 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:09:31.749347 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l_odh-ai-gateway-infra(261ccbf9-90b3-4771-99f0-cba64665bda8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l" podUID="261ccbf9-90b3-4771-99f0-cba64665bda8" Jul 17 02:09:32.755599 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:32.755570 2583 scope.go:117] "RemoveContainer" containerID="c80b1051b012a691ed903aca645affbac8351a9c690aba7c0385b22fea171339" Jul 17 02:09:32.756005 ip-10-0-128-44 kubenswrapper[2583]: E0717 02:09:32.755741 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l_odh-ai-gateway-infra(261ccbf9-90b3-4771-99f0-cba64665bda8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f0d41c05-cr52l" podUID="261ccbf9-90b3-4771-99f0-cba64665bda8" Jul 17 02:09:34.205363 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:34.205338 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w6dvf_0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a/kube-multus-additional-cni-plugins/0.log" Jul 17 02:09:34.224898 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:34.224873 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w6dvf_0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a/egress-router-binary-copy/0.log" Jul 17 02:09:34.250091 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:34.250060 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w6dvf_0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a/cni-plugins/0.log" Jul 17 02:09:34.269297 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:34.269271 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w6dvf_0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a/bond-cni-plugin/0.log" Jul 17 02:09:34.288243 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:34.288218 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w6dvf_0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a/routeoverride-cni/0.log" Jul 17 02:09:34.307907 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:34.307871 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w6dvf_0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a/whereabouts-cni-bincopy/0.log" Jul 17 02:09:34.338133 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:34.338090 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w6dvf_0e763b7c-1b22-4c5e-8dd2-1f06ad72be6a/whereabouts-cni/0.log" Jul 17 02:09:34.456483 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:34.456404 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-kmk5c_5f52d2f1-687d-4203-9c24-788062736e12/kube-multus/0.log" Jul 17 02:09:34.531970 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:34.531940 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9bx9f_8dbc5f6f-120c-42f2-b27f-5644ab623431/network-metrics-daemon/0.log" Jul 17 02:09:34.547319 ip-10-0-128-44 kubenswrapper[2583]: I0717 02:09:34.547285 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9bx9f_8dbc5f6f-120c-42f2-b27f-5644ab623431/kube-rbac-proxy/0.log"