Jul 17 05:41:15.137689 ip-10-0-142-12 systemd[1]: Starting Kubernetes Kubelet... Jul 17 05:41:15.612601 ip-10-0-142-12 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 05:41:15.612601 ip-10-0-142-12 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 05:41:15.612601 ip-10-0-142-12 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 05:41:15.612601 ip-10-0-142-12 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 05:41:15.612601 ip-10-0-142-12 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 05:41:15.612601 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:15.552421 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 05:41:15.612601 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559936 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 05:41:15.612601 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559953 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 05:41:15.612601 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559957 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 05:41:15.612601 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559963 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 05:41:15.612601 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559966 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 05:41:15.612601 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559969 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 05:41:15.612601 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559974 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 05:41:15.612601 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559978 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559981 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559984 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559987 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559989 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559992 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559995 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.559997 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560003 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560006 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560009 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560014 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560017 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560020 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560023 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560026 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560029 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560033 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560035 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 05:41:15.613716 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560038 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560041 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560047 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560050 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560053 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560056 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560058 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560061 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560064 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560067 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560069 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560072 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560075 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560078 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560080 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560090 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560093 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560095 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560098 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560101 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 05:41:15.614360 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560103 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560107 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560110 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560113 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560116 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560119 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560125 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560131 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560134 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560137 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560140 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560142 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560145 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560148 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560151 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560154 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560157 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560160 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560162 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 05:41:15.615258 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560167 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560170 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560173 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560176 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560178 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560181 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560183 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560186 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560188 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560191 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560194 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560196 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560201 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560205 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560207 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560210 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560213 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560216 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560219 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 05:41:15.615840 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560222 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560225 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560228 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560231 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560233 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560236 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560241 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560249 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560252 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560254 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560257 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560260 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560263 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560265 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.560268 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561441 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561450 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561453 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561455 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561459 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 17 05:41:15.616911 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561462 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561465 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561467 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561470 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561473 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561478 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561481 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561484 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561486 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561490 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561493 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561496 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561500 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561502 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561505 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561508 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561510 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561513 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561523 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561525 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 05:41:15.617692 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561528 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561531 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561533 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561536 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561539 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561541 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561544 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561547 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561549 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561552 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561557 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561560 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561562 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561565 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561568 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561570 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561573 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561575 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561578 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561580 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 05:41:15.618512 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561583 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561588 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561592 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561599 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561602 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561606 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561609 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561612 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561615 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561617 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561620 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561623 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561626 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561628 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561631 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561636 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561638 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561641 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561643 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 05:41:15.619290 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561646 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561651 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561655 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561660 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561663 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561666 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561669 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561672 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561677 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561681 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561683 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561686 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561688 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561696 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561701 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561704 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561707 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561710 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561712 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 05:41:15.620082 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561718 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561721 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561724 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561727 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561729 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561732 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561735 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561738 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561740 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561743 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561746 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561749 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561751 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561756 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561759 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.561762 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562810 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562817 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562820 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562823 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562825 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 05:41:15.620687 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562828 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562831 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562833 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562836 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562839 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562844 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562848 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562853 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562856 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562859 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562862 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562865 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562868 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562870 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562873 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562876 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562879 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562881 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562886 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 05:41:15.621555 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562892 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562896 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562899 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562902 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562905 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562907 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562910 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562913 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562915 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562918 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562921 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562923 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562926 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562931 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562934 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562936 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562939 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562946 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562950 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562954 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 05:41:15.622108 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562957 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562961 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562963 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562966 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562971 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562974 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562977 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562979 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562983 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562987 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562990 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562993 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.562997 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563000 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563003 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563005 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563010 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563013 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563018 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 05:41:15.622704 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563021 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563024 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563026 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563029 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563032 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563034 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563037 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563039 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563042 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563046 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563049 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563054 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563057 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563060 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563063 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563065 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563068 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563071 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563073 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563076 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563079 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 05:41:15.623279 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563084 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563086 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563089 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563091 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563094 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563097 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563100 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563105 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563108 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563111 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563115 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563118 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563121 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563125 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:15.563128 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:15.563223 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:15.563231 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:15.563237 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:15.563242 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 05:41:15.624047 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:15.563246 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.141978 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142023 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142032 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142036 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142040 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142046 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142051 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142055 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142058 2580 flags.go:64] FLAG: --cgroup-root="" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142061 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142066 2580 flags.go:64] FLAG: --client-ca-file="" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142081 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142086 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142094 2580 flags.go:64] FLAG: --cluster-domain="" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142097 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142101 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142104 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142108 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142116 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142119 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142125 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142129 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142133 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 17 05:41:16.142395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142138 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142142 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142147 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142151 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142159 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142163 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142168 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142172 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142178 2580 flags.go:64] FLAG: --enable-server="true" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142181 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142189 2580 flags.go:64] FLAG: --event-burst="100" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142192 2580 flags.go:64] FLAG: --event-qps="50" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142197 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142201 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142206 2580 flags.go:64] FLAG: --eviction-hard="" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142223 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142226 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142231 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142235 2580 flags.go:64] FLAG: --eviction-soft="" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142240 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142244 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142247 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142252 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142255 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142261 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 05:41:16.144332 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142264 2580 flags.go:64] FLAG: --feature-gates="" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142269 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142272 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142276 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142279 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142283 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142288 2580 flags.go:64] FLAG: --help="false" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142292 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-142-12.ec2.internal" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142296 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142300 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142306 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142310 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142318 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142322 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142326 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142330 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142335 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142338 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142342 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142347 2580 flags.go:64] FLAG: --kube-reserved="" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142350 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142355 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142359 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142362 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 05:41:16.145221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142366 2580 flags.go:64] FLAG: --lock-file="" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142370 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142374 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142380 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142388 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142391 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142396 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142400 2580 flags.go:64] FLAG: --logging-format="text" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142405 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142408 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142413 2580 flags.go:64] FLAG: --manifest-url="" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142417 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142423 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142426 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142433 2580 flags.go:64] FLAG: --max-pods="110" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142439 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142442 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142448 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142451 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142456 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142473 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142478 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142489 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142493 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142496 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 05:41:16.146705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142499 2580 flags.go:64] FLAG: --pod-cidr="" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142505 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142512 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142516 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142520 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142523 2580 flags.go:64] FLAG: --port="10250" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142526 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142529 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cc684bd948d8fb99" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142532 2580 flags.go:64] FLAG: --qos-reserved="" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142536 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142540 2580 flags.go:64] FLAG: --register-node="true" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142543 2580 flags.go:64] FLAG: --register-with-taints="" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142547 2580 flags.go:64] FLAG: --registry-burst="10" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142551 2580 flags.go:64] FLAG: --registry-qps="5" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142554 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142557 2580 flags.go:64] FLAG: --reserved-memory="" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142561 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142565 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142569 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142572 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142575 2580 flags.go:64] FLAG: --runonce="false" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142578 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142582 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142585 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142588 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 05:41:16.147642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142591 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142594 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142598 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142602 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142605 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142608 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142611 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142614 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142618 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142622 2580 flags.go:64] FLAG: --system-cgroups="" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142625 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142633 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142636 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142640 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142645 2580 flags.go:64] FLAG: --tls-min-version="" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142648 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142651 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142654 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142657 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142661 2580 flags.go:64] FLAG: --v="2" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142666 2580 flags.go:64] FLAG: --version="false" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142671 2580 flags.go:64] FLAG: --vmodule="" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142675 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.142678 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142839 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 05:41:16.149152 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142843 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142847 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142850 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142853 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142855 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142858 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142861 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142865 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142868 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142871 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142874 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142877 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142879 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142883 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142886 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142889 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142891 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142896 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142899 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142901 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 05:41:16.150131 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142904 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142907 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142910 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142913 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142915 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142918 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142920 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142923 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142926 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142929 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142931 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142936 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142939 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142942 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142945 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142948 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142951 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142954 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142956 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 05:41:16.151494 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142959 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142964 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142968 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142971 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142974 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142976 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142980 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142983 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142985 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142988 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142991 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142994 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.142998 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143001 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143004 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143072 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143076 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143078 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143083 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 05:41:16.152159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143086 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143089 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143093 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143113 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143222 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143226 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143251 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143368 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143371 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143374 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143377 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143379 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143382 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143385 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143388 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143545 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143564 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143584 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143601 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 05:41:16.152987 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143674 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143677 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143680 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143683 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143686 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143689 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143692 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143695 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143698 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143713 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143729 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143744 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143761 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143786 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143833 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143849 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143869 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143895 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143899 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143919 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 05:41:16.153634 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.143938 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.144008 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.154049 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.154071 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.154091 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.154098 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154164 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154170 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154173 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154176 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154180 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154183 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154186 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154189 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154193 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 05:41:16.154252 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154196 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154198 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154201 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154204 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154207 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154210 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154214 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154216 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154220 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154223 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154226 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154229 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154232 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154235 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154238 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154240 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154243 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154246 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154249 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154254 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 05:41:16.154844 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154259 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154264 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154269 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154272 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154275 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154278 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154281 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154286 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154290 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154293 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154296 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154299 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154302 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154305 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154309 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154311 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154314 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154317 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154319 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 05:41:16.155419 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154322 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154326 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154329 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154331 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154334 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154337 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154340 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154342 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154345 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154348 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154352 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154356 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154359 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154363 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154366 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154369 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154372 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154375 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154379 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 05:41:16.156072 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154382 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154385 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154388 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154391 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154394 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154396 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154399 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154402 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154404 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154408 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154410 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154413 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154416 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154419 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154422 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154425 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154428 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154431 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154434 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154437 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 05:41:16.156701 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154440 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154442 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154445 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154448 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154450 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154453 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154471 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154476 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154480 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154483 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154486 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154489 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.154495 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154622 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154629 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 05:41:16.157952 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154632 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154636 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154639 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154642 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154645 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154648 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154651 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154654 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154657 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154659 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154662 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154665 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154668 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154671 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154673 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154676 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154679 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154682 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154684 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 05:41:16.161062 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154687 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154690 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154692 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154695 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154698 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154701 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154704 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154707 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154709 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154712 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154715 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154718 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154720 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154723 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154726 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154729 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154731 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154734 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154737 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 05:41:16.161739 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154740 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154744 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154748 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154751 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154754 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154757 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154761 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154764 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154767 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154769 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154772 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154775 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154778 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154781 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154783 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154786 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154789 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154792 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154795 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154797 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 05:41:16.162369 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154800 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154803 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154806 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154808 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154811 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154814 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154817 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154821 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154825 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154829 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154832 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154835 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154838 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154841 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154844 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154846 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154849 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154852 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154855 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154858 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 05:41:16.163564 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154861 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154863 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154866 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154868 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154871 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154873 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154876 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154879 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154881 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154884 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154887 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154889 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154892 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154894 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154898 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154900 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154903 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154907 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 05:41:16.164237 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:16.154910 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 05:41:16.164897 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.154915 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 05:41:16.164897 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.155069 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 05:41:16.164897 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.157379 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 05:41:16.164897 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.158785 2580 server.go:1019] "Starting client certificate rotation" Jul 17 05:41:16.164897 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.158887 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 05:41:16.164897 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.158931 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 05:41:16.164897 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.161163 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 05:41:16.164897 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.162900 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 05:41:16.219241 ip-10-0-142-12 systemd[1]: Started Kubernetes Kubelet. Jul 17 05:41:16.245431 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.177984 2580 log.go:25] "Validated CRI v1 runtime API" Jul 17 05:41:16.245431 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.180329 2580 log.go:25] "Validated CRI v1 image API" Jul 17 05:41:16.245431 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.181273 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 05:41:16.245431 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.187254 2580 fs.go:135] Filesystem UUIDs: map[3353e4ec-23cf-4558-94b7-c8fae56cd584:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9145be99-f8ce-4b3b-a579-e7fff249de31:/dev/nvme0n1p4] Jul 17 05:41:16.245431 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.187276 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.194497 2580 manager.go:216] Machine: {Timestamp:2026-07-17 05:41:16.193033899 +0000 UTC m=+0.937713220 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3196927 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fcaacfe4326eec9bababd8db962d3 SystemUUID:ec2fcaac-fe43-26ee-c9ba-babd8db962d3 BootID:c7de3478-6d8d-4791-8fe1-1f7afe661343 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3e:29:0f:e6:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3e:29:0f:e6:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:28:5f:a7:26:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.194608 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.194699 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.198738 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.198776 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-12.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.199523 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.199533 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.199541 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.200363 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.201685 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.205019 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.205143 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.205895 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.205907 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.205918 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.205928 2580 kubelet.go:398] "Adding apiserver pod source" Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.205939 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 05:41:16.247274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.206317 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.206340 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.209161 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.211355 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.211384 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.212487 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.212502 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.212508 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.212514 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.212522 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.212528 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.212538 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.212544 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.212551 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.212556 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.212571 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.212577 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.213267 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.213274 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.216579 2580 server.go:1268] "Started kubelet" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.216670 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.216755 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.216830 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.218855 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.221479 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.223683 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.226952 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.226967 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.227065 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.227564 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.227583 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.227589 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.227667 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.227678 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.227816 2580 factory.go:55] Registering systemd factory Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.227831 2580 factory.go:223] Registration of the systemd container factory successfully Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.228056 2580 factory.go:153] Registering CRI-O factory Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.228068 2580 factory.go:223] Registration of the crio container factory successfully Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.228120 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.228141 2580 factory.go:103] Registering Raw factory Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.228153 2580 manager.go:1196] Started watching for new ooms in manager Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.228550 2580 manager.go:349] Starting recovery of all containers Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.242270 2580 manager.go:354] Recovery completed Jul 17 05:41:16.248852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.247786 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 05:41:16.252914 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.250604 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientMemory" Jul 17 05:41:16.252914 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.250638 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 05:41:16.252914 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.250651 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientPID" Jul 17 05:41:16.252914 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.251227 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 05:41:16.252914 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.251233 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 05:41:16.252914 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.251239 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 05:41:16.252914 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.251257 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 17 05:41:16.252914 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.252073 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientMemory" Jul 17 05:41:16.252914 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.252101 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 05:41:16.252914 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.252115 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientPID" Jul 17 05:41:16.252914 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.252664 2580 policy_none.go:49] "None policy: Start" Jul 17 05:41:16.252914 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.252689 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 05:41:16.252914 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.252701 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 05:41:16.255251 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.253692 2580 policy_none.go:47] "Start" Jul 17 05:41:16.339218 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.327942 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:16.432212 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.428302 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:16.601317 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.528684 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:16.680674 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.629098 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:16.703693 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.703639 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-12.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 05:41:16.703970 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.703718 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 17 05:41:16.703970 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.703717 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-12.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 05:41:16.703970 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.703853 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-12.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 17 05:41:16.703970 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.703917 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 05:41:16.704769 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.703728 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-12.ec2.internal.18c2fd453d62232e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-12.ec2.internal,UID:ip-10-0-142-12.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-12.ec2.internal,},FirstTimestamp:2026-07-17 05:41:16.21655019 +0000 UTC m=+0.961229509,LastTimestamp:2026-07-17 05:41:16.21655019 +0000 UTC m=+0.961229509,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-12.ec2.internal,}" Jul 17 05:41:16.706569 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.706547 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 05:41:16.706949 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.706854 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-12.ec2.internal.18c2fd453f6a07bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-12.ec2.internal,UID:ip-10-0-142-12.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-12.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-12.ec2.internal,},FirstTimestamp:2026-07-17 05:41:16.250621887 +0000 UTC m=+0.995301210,LastTimestamp:2026-07-17 05:41:16.250621887 +0000 UTC m=+0.995301210,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-12.ec2.internal,}" Jul 17 05:41:16.716058 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.715947 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-12.ec2.internal.18c2fd453f6a5e36 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-12.ec2.internal,UID:ip-10-0-142-12.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-12.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-12.ec2.internal,},FirstTimestamp:2026-07-17 05:41:16.250644022 +0000 UTC m=+0.995323345,LastTimestamp:2026-07-17 05:41:16.250644022 +0000 UTC m=+0.995323345,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-12.ec2.internal,}" Jul 17 05:41:16.727336 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.726068 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-12.ec2.internal.18c2fd453f6a8d33 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-12.ec2.internal,UID:ip-10-0-142-12.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-12.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-12.ec2.internal,},FirstTimestamp:2026-07-17 05:41:16.250656051 +0000 UTC m=+0.995335373,LastTimestamp:2026-07-17 05:41:16.250656051 +0000 UTC m=+0.995335373,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-12.ec2.internal,}" Jul 17 05:41:16.728866 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.728842 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p9bdf" Jul 17 05:41:16.729880 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.729864 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:16.737381 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.737349 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p9bdf" Jul 17 05:41:16.740337 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.740224 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-12.ec2.internal.18c2fd453f6a07bf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-12.ec2.internal.18c2fd453f6a07bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-12.ec2.internal,UID:ip-10-0-142-12.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-12.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-12.ec2.internal,},FirstTimestamp:2026-07-17 05:41:16.250621887 +0000 UTC m=+0.995301210,LastTimestamp:2026-07-17 05:41:16.252085053 +0000 UTC m=+0.996764376,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-12.ec2.internal,}" Jul 17 05:41:16.748325 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.748144 2580 manager.go:337] "Starting Device Plugin manager" Jul 17 05:41:16.748463 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.748354 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 05:41:16.748463 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.748370 2580 server.go:87] "Starting device plugin registration server" Jul 17 05:41:16.748640 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.748628 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 05:41:16.748679 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.748643 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 05:41:16.748767 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.748737 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 05:41:16.748879 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.748824 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 05:41:16.748879 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.748875 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 05:41:16.749481 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.749457 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 05:41:16.749570 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.749518 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:16.849856 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.849817 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 05:41:16.851154 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.851002 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientMemory" Jul 17 05:41:16.851154 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.851032 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 05:41:16.851154 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.851042 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientPID" Jul 17 05:41:16.851154 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.851075 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-12.ec2.internal" Jul 17 05:41:16.859509 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.859456 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-12.ec2.internal" Jul 17 05:41:16.859509 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.859482 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-12.ec2.internal\": node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:16.873940 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.873910 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:16.881921 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.881887 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 05:41:16.883211 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.883194 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 05:41:16.883275 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.883236 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 05:41:16.883275 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.883259 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 05:41:16.883350 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.883295 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 05:41:16.885634 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.885608 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 05:41:16.977450 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:16.977379 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:16.983969 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.983930 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal"] Jul 17 05:41:16.984057 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.984019 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 05:41:16.985838 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.985816 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientMemory" Jul 17 05:41:16.985960 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.985856 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 05:41:16.985960 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.985872 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientPID" Jul 17 05:41:16.987427 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.987408 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 05:41:16.987568 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.987549 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" Jul 17 05:41:16.987622 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.987587 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 05:41:16.992956 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.992938 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientMemory" Jul 17 05:41:16.993043 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.992968 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 05:41:16.993043 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.992977 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientPID" Jul 17 05:41:16.993043 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.992938 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientMemory" Jul 17 05:41:16.993134 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.993056 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 05:41:16.993134 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.993072 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientPID" Jul 17 05:41:16.994328 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.994311 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal" Jul 17 05:41:16.994412 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.994358 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 05:41:16.995186 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.995169 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientMemory" Jul 17 05:41:16.995278 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.995203 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 05:41:16.995278 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:16.995218 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeHasSufficientPID" Jul 17 05:41:17.013116 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:17.013087 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-12.ec2.internal\" not found" node="ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.018109 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:17.018087 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-12.ec2.internal\" not found" node="ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.031024 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.030999 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab400b4fa7d3b1af72fd41bc419aef74-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal\" (UID: \"ab400b4fa7d3b1af72fd41bc419aef74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.031119 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.031029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab400b4fa7d3b1af72fd41bc419aef74-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal\" (UID: \"ab400b4fa7d3b1af72fd41bc419aef74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.031119 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.031046 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba0e125b5dfaf98ff0a6e6b6f46e0aae-config\") pod \"kube-apiserver-proxy-ip-10-0-142-12.ec2.internal\" (UID: \"ba0e125b5dfaf98ff0a6e6b6f46e0aae\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.078442 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:17.078418 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:17.131759 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.131701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab400b4fa7d3b1af72fd41bc419aef74-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal\" (UID: \"ab400b4fa7d3b1af72fd41bc419aef74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.131826 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.131775 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab400b4fa7d3b1af72fd41bc419aef74-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal\" (UID: \"ab400b4fa7d3b1af72fd41bc419aef74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.131826 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.131786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba0e125b5dfaf98ff0a6e6b6f46e0aae-config\") pod \"kube-apiserver-proxy-ip-10-0-142-12.ec2.internal\" (UID: \"ba0e125b5dfaf98ff0a6e6b6f46e0aae\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.131826 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.131815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab400b4fa7d3b1af72fd41bc419aef74-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal\" (UID: \"ab400b4fa7d3b1af72fd41bc419aef74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.131921 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.131839 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ba0e125b5dfaf98ff0a6e6b6f46e0aae-config\") pod \"kube-apiserver-proxy-ip-10-0-142-12.ec2.internal\" (UID: \"ba0e125b5dfaf98ff0a6e6b6f46e0aae\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.131921 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.131856 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab400b4fa7d3b1af72fd41bc419aef74-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal\" (UID: \"ab400b4fa7d3b1af72fd41bc419aef74\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.159959 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.159931 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 05:41:17.160108 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.160089 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 05:41:17.179732 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:17.179693 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:17.283115 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:17.283074 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:17.317889 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.317859 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.322175 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.321894 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.383546 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:17.383448 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:17.486879 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:17.486825 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:17.587804 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:17.587757 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-12.ec2.internal\" not found" Jul 17 05:41:17.661420 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.661341 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 05:41:17.663422 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.663403 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 05:41:17.727592 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.727557 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.739088 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.739055 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 05:41:17.741116 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.741094 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal" Jul 17 05:41:17.741274 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.741249 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 05:36:16 +0000 UTC" deadline="2028-04-26 15:47:14.996759988 +0000 UTC" Jul 17 05:41:17.741313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.741276 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15586h5m57.255487828s" Jul 17 05:41:17.750258 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.750240 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 05:41:17.846210 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:17.846176 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0e125b5dfaf98ff0a6e6b6f46e0aae.slice/crio-dedbd166a9a6ce26fe55d8f15f0f9808cd84bb30b43f7712b684b38446ff9377 WatchSource:0}: Error finding container dedbd166a9a6ce26fe55d8f15f0f9808cd84bb30b43f7712b684b38446ff9377: Status 404 returned error can't find the container with id dedbd166a9a6ce26fe55d8f15f0f9808cd84bb30b43f7712b684b38446ff9377 Jul 17 05:41:17.846609 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:17.846581 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab400b4fa7d3b1af72fd41bc419aef74.slice/crio-911195c011447bbf123279625aff84e1ccd9fddb968877c09bf24731385d5486 WatchSource:0}: Error finding container 911195c011447bbf123279625aff84e1ccd9fddb968877c09bf24731385d5486: Status 404 returned error can't find the container with id 911195c011447bbf123279625aff84e1ccd9fddb968877c09bf24731385d5486 Jul 17 05:41:17.851593 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.851571 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 05:41:17.886539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.886485 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" event={"ID":"ab400b4fa7d3b1af72fd41bc419aef74","Type":"ContainerStarted","Data":"911195c011447bbf123279625aff84e1ccd9fddb968877c09bf24731385d5486"} Jul 17 05:41:17.887678 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.887648 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal" event={"ID":"ba0e125b5dfaf98ff0a6e6b6f46e0aae","Type":"ContainerStarted","Data":"dedbd166a9a6ce26fe55d8f15f0f9808cd84bb30b43f7712b684b38446ff9377"} Jul 17 05:41:17.909589 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:17.909561 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 05:41:18.209055 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.209024 2580 apiserver.go:52] "Watching apiserver" Jul 17 05:41:18.214335 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.214303 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 05:41:18.214731 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.214707 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal","openshift-multus/multus-c56wh","openshift-network-diagnostics/network-check-target-5b6z6","openshift-ovn-kubernetes/ovnkube-node-jvdks","kube-system/global-pull-secret-syncer-t6rpv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg","openshift-dns/node-resolver-gw9qs","openshift-multus/multus-additional-cni-plugins-8wg7p","openshift-multus/network-metrics-daemon-nsrf4","openshift-network-operator/iptables-alerter-hcsh4","kube-system/konnectivity-agent-7phbz","kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bjh7n","openshift-image-registry/node-ca-xqr2c"] Jul 17 05:41:18.216202 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.216171 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:18.216325 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.216260 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:18.217924 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.217893 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7phbz" Jul 17 05:41:18.218964 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.218936 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gw9qs" Jul 17 05:41:18.220091 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.220064 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lxcrg\"" Jul 17 05:41:18.220256 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.220064 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 05:41:18.220256 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.220220 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 05:41:18.220510 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.220487 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.220992 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.220955 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wwp7t\"" Jul 17 05:41:18.221263 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.221242 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 05:41:18.221534 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.221463 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 05:41:18.221770 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.221751 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:18.221852 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.221815 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:18.222239 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.222220 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 05:41:18.223041 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.223020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.223182 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.223161 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 05:41:18.223379 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.223359 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 05:41:18.223485 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.223466 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 05:41:18.223544 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.223364 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 05:41:18.224392 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.224076 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 05:41:18.224883 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.224860 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.226066 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.225528 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 05:41:18.226066 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.225550 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 05:41:18.226066 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.226025 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gfxfr\"" Jul 17 05:41:18.227134 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.227026 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:18.227279 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.227119 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:18.227651 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.227551 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 05:41:18.227742 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.227687 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 05:41:18.227742 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.227733 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-52s9f\"" Jul 17 05:41:18.227917 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.227897 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 05:41:18.228200 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.228069 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 05:41:18.228550 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.228523 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 05:41:18.228799 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.228779 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hcsh4" Jul 17 05:41:18.229002 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.228974 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qjr4m\"" Jul 17 05:41:18.230474 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.230332 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 05:41:18.230924 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.230906 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 05:41:18.231021 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.230978 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 05:41:18.231127 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.231105 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hl865\"" Jul 17 05:41:18.231367 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.231352 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.231666 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.231624 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 05:41:18.231976 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.231952 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 05:41:18.233079 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.232950 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.233597 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.233575 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 05:41:18.233715 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.233583 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-66c5t\"" Jul 17 05:41:18.234365 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.234345 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xqr2c" Jul 17 05:41:18.235509 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.235092 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ht6v2\"" Jul 17 05:41:18.235509 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.235291 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 05:41:18.235509 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.235337 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 05:41:18.236360 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c-agent-certs\") pod \"konnectivity-agent-7phbz\" (UID: \"12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c\") " pod="kube-system/konnectivity-agent-7phbz" Jul 17 05:41:18.236546 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-etc-openvswitch\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.236546 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-cni-netd\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.236546 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ddecbb1a-d4df-4a32-85bb-793654f86198-env-overrides\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.236546 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236480 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55c6188c-0a2e-4607-8c0e-7a5685859c4d-dbus\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:18.236546 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-registration-dir\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.236799 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236567 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-kubelet\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.236799 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-run-systemd\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.236799 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236619 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7t27\" (UniqueName: \"kubernetes.io/projected/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-kube-api-access-v7t27\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.236799 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236645 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0600e268-96d5-4b77-893d-09eb28d97ebd-iptables-alerter-script\") pod \"iptables-alerter-hcsh4\" (UID: \"0600e268-96d5-4b77-893d-09eb28d97ebd\") " pod="openshift-network-operator/iptables-alerter-hcsh4" Jul 17 05:41:18.236799 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236657 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 05:41:18.236799 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236663 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mf65m\"" Jul 17 05:41:18.236799 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236669 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71cd888f-6076-4d46-ba8d-ba55fcee848c-system-cni-dir\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.236799 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236720 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71cd888f-6076-4d46-ba8d-ba55fcee848c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.236799 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236746 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/71cd888f-6076-4d46-ba8d-ba55fcee848c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.236799 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-run-netns\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-run-openvswitch\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-log-socket\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l22pw\" (UniqueName: \"kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw\") pod \"network-check-target-5b6z6\" (UID: \"ac6cc6e6-4369-4fbc-898b-13c449fe4c92\") " pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71cd888f-6076-4d46-ba8d-ba55fcee848c-cnibin\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/71cd888f-6076-4d46-ba8d-ba55fcee848c-os-release\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236960 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-node-log\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ddecbb1a-d4df-4a32-85bb-793654f86198-ovnkube-config\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.236993 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-socket-dir\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237019 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237020 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-etc-selinux\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237115 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6d89cf-4331-4c48-91b6-5e6b570d8927-tmp-dir\") pod \"node-resolver-gw9qs\" (UID: \"ca6d89cf-4331-4c48-91b6-5e6b570d8927\") " pod="openshift-dns/node-resolver-gw9qs" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crdwc\" (UniqueName: \"kubernetes.io/projected/ca6d89cf-4331-4c48-91b6-5e6b570d8927-kube-api-access-crdwc\") pod \"node-resolver-gw9qs\" (UID: \"ca6d89cf-4331-4c48-91b6-5e6b570d8927\") " pod="openshift-dns/node-resolver-gw9qs" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgkgt\" (UniqueName: \"kubernetes.io/projected/71cd888f-6076-4d46-ba8d-ba55fcee848c-kube-api-access-fgkgt\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237163 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-slash\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237178 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-var-lib-openvswitch\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237191 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 05:41:18.237313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237198 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-run-ovn\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237286 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca6d89cf-4331-4c48-91b6-5e6b570d8927-hosts-file\") pod \"node-resolver-gw9qs\" (UID: \"ca6d89cf-4331-4c48-91b6-5e6b570d8927\") " pod="openshift-dns/node-resolver-gw9qs" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237333 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-systemd-units\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55c6188c-0a2e-4607-8c0e-7a5685859c4d-kubelet-config\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47kb\" (UniqueName: \"kubernetes.io/projected/0600e268-96d5-4b77-893d-09eb28d97ebd-kube-api-access-x47kb\") pod \"iptables-alerter-hcsh4\" (UID: \"0600e268-96d5-4b77-893d-09eb28d97ebd\") " pod="openshift-network-operator/iptables-alerter-hcsh4" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237464 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/71cd888f-6076-4d46-ba8d-ba55fcee848c-cni-binary-copy\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237538 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237565 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-sys-fs\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtcfh\" (UniqueName: \"kubernetes.io/projected/3c472db7-71bf-46b2-9452-f936c8c53e67-kube-api-access-mtcfh\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0600e268-96d5-4b77-893d-09eb28d97ebd-host-slash\") pod \"iptables-alerter-hcsh4\" (UID: \"0600e268-96d5-4b77-893d-09eb28d97ebd\") " pod="openshift-network-operator/iptables-alerter-hcsh4" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237648 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c-konnectivity-ca\") pod \"konnectivity-agent-7phbz\" (UID: \"12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c\") " pod="kube-system/konnectivity-agent-7phbz" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-cni-bin\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.237973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237699 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ddecbb1a-d4df-4a32-85bb-793654f86198-ovn-node-metrics-cert\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.238704 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237725 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ddecbb1a-d4df-4a32-85bb-793654f86198-ovnkube-script-lib\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.238704 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237750 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx2fp\" (UniqueName: \"kubernetes.io/projected/ddecbb1a-d4df-4a32-85bb-793654f86198-kube-api-access-kx2fp\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.238704 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-device-dir\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.238704 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.237851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/71cd888f-6076-4d46-ba8d-ba55fcee848c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.238704 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.238173 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 05:41:18.260598 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.260568 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g4qv9" Jul 17 05:41:18.268766 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.268736 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g4qv9" Jul 17 05:41:18.328931 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.328853 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-sysctl-conf\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71cd888f-6076-4d46-ba8d-ba55fcee848c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338367 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/71cd888f-6076-4d46-ba8d-ba55fcee848c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-run-openvswitch\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-log-socket\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22pw\" (UniqueName: \"kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw\") pod \"network-check-target-5b6z6\" (UID: \"ac6cc6e6-4369-4fbc-898b-13c449fe4c92\") " pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-multus-cni-dir\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-sysctl-d\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71cd888f-6076-4d46-ba8d-ba55fcee848c-cnibin\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/71cd888f-6076-4d46-ba8d-ba55fcee848c-os-release\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-node-log\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338590 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ddecbb1a-d4df-4a32-85bb-793654f86198-ovnkube-config\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-etc-selinux\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6d89cf-4331-4c48-91b6-5e6b570d8927-tmp-dir\") pod \"node-resolver-gw9qs\" (UID: \"ca6d89cf-4331-4c48-91b6-5e6b570d8927\") " pod="openshift-dns/node-resolver-gw9qs" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-os-release\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338683 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-var-lib-cni-bin\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.341169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgkgt\" (UniqueName: \"kubernetes.io/projected/71cd888f-6076-4d46-ba8d-ba55fcee848c-kube-api-access-fgkgt\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338770 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-slash\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca6d89cf-4331-4c48-91b6-5e6b570d8927-hosts-file\") pod \"node-resolver-gw9qs\" (UID: \"ca6d89cf-4331-4c48-91b6-5e6b570d8927\") " pod="openshift-dns/node-resolver-gw9qs" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-multus-daemon-config\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338897 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-run-multus-certs\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338920 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-run\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-sys\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.338990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55c6188c-0a2e-4607-8c0e-7a5685859c4d-kubelet-config\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47kb\" (UniqueName: \"kubernetes.io/projected/0600e268-96d5-4b77-893d-09eb28d97ebd-kube-api-access-x47kb\") pod \"iptables-alerter-hcsh4\" (UID: \"0600e268-96d5-4b77-893d-09eb28d97ebd\") " pod="openshift-network-operator/iptables-alerter-hcsh4" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-cni-binary-copy\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339092 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47cf6\" (UniqueName: \"kubernetes.io/projected/7e59750b-d74b-4632-af2b-d509d25811de-kube-api-access-47cf6\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-sys-fs\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-run-netns\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339197 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-sysconfig\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.342539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e59750b-d74b-4632-af2b-d509d25811de-tmp\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ddecbb1a-d4df-4a32-85bb-793654f86198-ovnkube-script-lib\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339259 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-var-lib-kubelet\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339276 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-var-lib-kubelet\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c-agent-certs\") pod \"konnectivity-agent-7phbz\" (UID: \"12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c\") " pod="kube-system/konnectivity-agent-7phbz" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-cni-netd\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ddecbb1a-d4df-4a32-85bb-793654f86198-env-overrides\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7t27\" (UniqueName: \"kubernetes.io/projected/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-kube-api-access-v7t27\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339375 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-multus-conf-dir\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-systemd\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-run-systemd\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339507 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0600e268-96d5-4b77-893d-09eb28d97ebd-iptables-alerter-script\") pod \"iptables-alerter-hcsh4\" (UID: \"0600e268-96d5-4b77-893d-09eb28d97ebd\") " pod="openshift-network-operator/iptables-alerter-hcsh4" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71cd888f-6076-4d46-ba8d-ba55fcee848c-system-cni-dir\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-run-netns\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339610 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-etc-kubernetes\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339635 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-host\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71cd888f-6076-4d46-ba8d-ba55fcee848c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-socket-dir\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crdwc\" (UniqueName: \"kubernetes.io/projected/ca6d89cf-4331-4c48-91b6-5e6b570d8927-kube-api-access-crdwc\") pod \"node-resolver-gw9qs\" (UID: \"ca6d89cf-4331-4c48-91b6-5e6b570d8927\") " pod="openshift-dns/node-resolver-gw9qs" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-modprobe-d\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-kubernetes\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-var-lib-openvswitch\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-run-ovn\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339839 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-socket-dir\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3a2951e-e5a2-49fc-a938-a5617748d2bb-serviceca\") pod \"node-ca-xqr2c\" (UID: \"e3a2951e-e5a2-49fc-a938-a5617748d2bb\") " pod="openshift-image-registry/node-ca-xqr2c" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339927 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv9w7\" (UniqueName: \"kubernetes.io/projected/e3a2951e-e5a2-49fc-a938-a5617748d2bb-kube-api-access-mv9w7\") pod \"node-ca-xqr2c\" (UID: \"e3a2951e-e5a2-49fc-a938-a5617748d2bb\") " pod="openshift-image-registry/node-ca-xqr2c" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.339940 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-systemd-units\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.339988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-var-lib-cni-multus\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.340005 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs podName:3c472db7-71bf-46b2-9452-f936c8c53e67 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:18.839984334 +0000 UTC m=+3.584663658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs") pod "network-metrics-daemon-nsrf4" (UID: "3c472db7-71bf-46b2-9452-f936c8c53e67") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.340270 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ddecbb1a-d4df-4a32-85bb-793654f86198-ovnkube-script-lib\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.340455 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-systemd-units\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.340482 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-run-openvswitch\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.344316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.340498 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-log-socket\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.340529 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71cd888f-6076-4d46-ba8d-ba55fcee848c-cnibin\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.340535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-slash\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.340587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.340752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/71cd888f-6076-4d46-ba8d-ba55fcee848c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.340782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/71cd888f-6076-4d46-ba8d-ba55fcee848c-cni-binary-copy\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.340823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.340866 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtcfh\" (UniqueName: \"kubernetes.io/projected/3c472db7-71bf-46b2-9452-f936c8c53e67-kube-api-access-mtcfh\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.340931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0600e268-96d5-4b77-893d-09eb28d97ebd-host-slash\") pod \"iptables-alerter-hcsh4\" (UID: \"0600e268-96d5-4b77-893d-09eb28d97ebd\") " pod="openshift-network-operator/iptables-alerter-hcsh4" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.340965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-system-cni-dir\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.340998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c-konnectivity-ca\") pod \"konnectivity-agent-7phbz\" (UID: \"12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c\") " pod="kube-system/konnectivity-agent-7phbz" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.341073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-cni-bin\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.341094 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/71cd888f-6076-4d46-ba8d-ba55fcee848c-os-release\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.341118 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca6d89cf-4331-4c48-91b6-5e6b570d8927-tmp-dir\") pod \"node-resolver-gw9qs\" (UID: \"ca6d89cf-4331-4c48-91b6-5e6b570d8927\") " pod="openshift-dns/node-resolver-gw9qs" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.342182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ddecbb1a-d4df-4a32-85bb-793654f86198-ovn-node-metrics-cert\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.343689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx2fp\" (UniqueName: \"kubernetes.io/projected/ddecbb1a-d4df-4a32-85bb-793654f86198-kube-api-access-kx2fp\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.343731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-device-dir\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.345198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.343769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-cnibin\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.343800 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-run-k8s-cni-cncf-io\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.343836 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-hostroot\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.343876 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/71cd888f-6076-4d46-ba8d-ba55fcee848c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.343912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-etc-openvswitch\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.343954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55c6188c-0a2e-4607-8c0e-7a5685859c4d-dbus\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.343979 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-var-lib-openvswitch\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.343988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-registration-dir\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.343510 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca6d89cf-4331-4c48-91b6-5e6b570d8927-hosts-file\") pod \"node-resolver-gw9qs\" (UID: \"ca6d89cf-4331-4c48-91b6-5e6b570d8927\") " pod="openshift-dns/node-resolver-gw9qs" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.343534 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-run-ovn\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.344107 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-node-log\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.344158 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-registration-dir\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.344170 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-etc-openvswitch\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.344257 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-etc-selinux\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.344414 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55c6188c-0a2e-4607-8c0e-7a5685859c4d-dbus\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.344579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ddecbb1a-d4df-4a32-85bb-793654f86198-ovnkube-config\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.344633 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/71cd888f-6076-4d46-ba8d-ba55fcee848c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.346035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.344640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqlh\" (UniqueName: \"kubernetes.io/projected/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-kube-api-access-gkqlh\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.344680 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-lib-modules\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.344977 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71cd888f-6076-4d46-ba8d-ba55fcee848c-system-cni-dir\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.344978 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0600e268-96d5-4b77-893d-09eb28d97ebd-iptables-alerter-script\") pod \"iptables-alerter-hcsh4\" (UID: \"0600e268-96d5-4b77-893d-09eb28d97ebd\") " pod="openshift-network-operator/iptables-alerter-hcsh4" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.345087 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/71cd888f-6076-4d46-ba8d-ba55fcee848c-cni-binary-copy\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.345216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.345126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e59750b-d74b-4632-af2b-d509d25811de-etc-tuned\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.345275 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-kubelet\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.345341 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.345864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-multus-socket-dir-parent\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.346048 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-device-dir\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.346263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-cni-netd\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.346676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ddecbb1a-d4df-4a32-85bb-793654f86198-env-overrides\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.346741 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55c6188c-0a2e-4607-8c0e-7a5685859c4d-kubelet-config\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.346788 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-run-ovn-kubernetes\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.346834 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-run-netns\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.346902 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-sys-fs\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.347006 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.346944 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-kubelet\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.348004 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.347006 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-host-cni-bin\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.348004 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.347120 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:18.348004 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.347185 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret podName:55c6188c-0a2e-4607-8c0e-7a5685859c4d nodeName:}" failed. No retries permitted until 2026-07-17 05:41:18.847166388 +0000 UTC m=+3.591845717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret") pod "global-pull-secret-syncer-t6rpv" (UID: "55c6188c-0a2e-4607-8c0e-7a5685859c4d") : object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:18.348004 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.347833 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c-konnectivity-ca\") pod \"konnectivity-agent-7phbz\" (UID: \"12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c\") " pod="kube-system/konnectivity-agent-7phbz" Jul 17 05:41:18.348004 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.347894 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ddecbb1a-d4df-4a32-85bb-793654f86198-run-systemd\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.348300 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.348035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3a2951e-e5a2-49fc-a938-a5617748d2bb-host\") pod \"node-ca-xqr2c\" (UID: \"e3a2951e-e5a2-49fc-a938-a5617748d2bb\") " pod="openshift-image-registry/node-ca-xqr2c" Jul 17 05:41:18.348300 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.348196 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0600e268-96d5-4b77-893d-09eb28d97ebd-host-slash\") pod \"iptables-alerter-hcsh4\" (UID: \"0600e268-96d5-4b77-893d-09eb28d97ebd\") " pod="openshift-network-operator/iptables-alerter-hcsh4" Jul 17 05:41:18.348300 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.348246 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 05:41:18.348300 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.348265 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 05:41:18.348300 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.348284 2580 projected.go:196] Error preparing data for projected volume kube-api-access-l22pw for pod openshift-network-diagnostics/network-check-target-5b6z6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:18.348642 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.348353 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw podName:ac6cc6e6-4369-4fbc-898b-13c449fe4c92 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:18.848334537 +0000 UTC m=+3.593013873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l22pw" (UniqueName: "kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw") pod "network-check-target-5b6z6" (UID: "ac6cc6e6-4369-4fbc-898b-13c449fe4c92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:18.352590 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.352518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx2fp\" (UniqueName: \"kubernetes.io/projected/ddecbb1a-d4df-4a32-85bb-793654f86198-kube-api-access-kx2fp\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.352992 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.352945 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgkgt\" (UniqueName: \"kubernetes.io/projected/71cd888f-6076-4d46-ba8d-ba55fcee848c-kube-api-access-fgkgt\") pod \"multus-additional-cni-plugins-8wg7p\" (UID: \"71cd888f-6076-4d46-ba8d-ba55fcee848c\") " pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.353104 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.353074 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c-agent-certs\") pod \"konnectivity-agent-7phbz\" (UID: \"12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c\") " pod="kube-system/konnectivity-agent-7phbz" Jul 17 05:41:18.353782 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.353737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ddecbb1a-d4df-4a32-85bb-793654f86198-ovn-node-metrics-cert\") pod \"ovnkube-node-jvdks\" (UID: \"ddecbb1a-d4df-4a32-85bb-793654f86198\") " pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.354268 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.354234 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crdwc\" (UniqueName: \"kubernetes.io/projected/ca6d89cf-4331-4c48-91b6-5e6b570d8927-kube-api-access-crdwc\") pod \"node-resolver-gw9qs\" (UID: \"ca6d89cf-4331-4c48-91b6-5e6b570d8927\") " pod="openshift-dns/node-resolver-gw9qs" Jul 17 05:41:18.354874 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.354841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtcfh\" (UniqueName: \"kubernetes.io/projected/3c472db7-71bf-46b2-9452-f936c8c53e67-kube-api-access-mtcfh\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:18.356380 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.356356 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7t27\" (UniqueName: \"kubernetes.io/projected/6517c465-15e5-4290-ac8a-5c9f6d2e05fb-kube-api-access-v7t27\") pod \"aws-ebs-csi-driver-node-s4fwg\" (UID: \"6517c465-15e5-4290-ac8a-5c9f6d2e05fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.360055 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.359526 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47kb\" (UniqueName: \"kubernetes.io/projected/0600e268-96d5-4b77-893d-09eb28d97ebd-kube-api-access-x47kb\") pod \"iptables-alerter-hcsh4\" (UID: \"0600e268-96d5-4b77-893d-09eb28d97ebd\") " pod="openshift-network-operator/iptables-alerter-hcsh4" Jul 17 05:41:18.444850 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.444816 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-cnibin\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-run-k8s-cni-cncf-io\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-hostroot\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448493 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-cnibin\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqlh\" (UniqueName: \"kubernetes.io/projected/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-kube-api-access-gkqlh\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-hostroot\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448658 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-run-k8s-cni-cncf-io\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-lib-modules\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e59750b-d74b-4632-af2b-d509d25811de-etc-tuned\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448763 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-multus-socket-dir-parent\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3a2951e-e5a2-49fc-a938-a5617748d2bb-host\") pod \"node-ca-xqr2c\" (UID: \"e3a2951e-e5a2-49fc-a938-a5617748d2bb\") " pod="openshift-image-registry/node-ca-xqr2c" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448821 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-sysctl-conf\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448851 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-lib-modules\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-multus-cni-dir\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-sysctl-d\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448906 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-multus-socket-dir-parent\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-os-release\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.449847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-var-lib-cni-bin\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.448965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-multus-daemon-config\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-run-multus-certs\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449021 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-run\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449041 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-sys\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-cni-binary-copy\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47cf6\" (UniqueName: \"kubernetes.io/projected/7e59750b-d74b-4632-af2b-d509d25811de-kube-api-access-47cf6\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-run-netns\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449176 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-sysconfig\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e59750b-d74b-4632-af2b-d509d25811de-tmp\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-var-lib-kubelet\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-var-lib-kubelet\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-multus-conf-dir\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449325 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-run-multus-certs\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449377 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3a2951e-e5a2-49fc-a938-a5617748d2bb-host\") pod \"node-ca-xqr2c\" (UID: \"e3a2951e-e5a2-49fc-a938-a5617748d2bb\") " pod="openshift-image-registry/node-ca-xqr2c" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449370 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-multus-conf-dir\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449446 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-run\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449474 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-run-netns\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.450851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-os-release\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-sysctl-conf\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449533 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-multus-cni-dir\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449544 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-sys\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449560 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-systemd\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-etc-kubernetes\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449602 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-sysconfig\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-host\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449630 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-sysctl-d\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-modprobe-d\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449658 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-var-lib-kubelet\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449677 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-systemd\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-kubernetes\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449709 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-kubernetes\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449718 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-var-lib-cni-bin\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3a2951e-e5a2-49fc-a938-a5617748d2bb-serviceca\") pod \"node-ca-xqr2c\" (UID: \"e3a2951e-e5a2-49fc-a938-a5617748d2bb\") " pod="openshift-image-registry/node-ca-xqr2c" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449757 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-etc-kubernetes\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449758 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-host\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.451772 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449770 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv9w7\" (UniqueName: \"kubernetes.io/projected/e3a2951e-e5a2-49fc-a938-a5617748d2bb-kube-api-access-mv9w7\") pod \"node-ca-xqr2c\" (UID: \"e3a2951e-e5a2-49fc-a938-a5617748d2bb\") " pod="openshift-image-registry/node-ca-xqr2c" Jul 17 05:41:18.452631 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449803 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-var-lib-cni-multus\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.452631 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-system-cni-dir\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.452631 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449821 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-var-lib-kubelet\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.452631 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-system-cni-dir\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.452631 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449836 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e59750b-d74b-4632-af2b-d509d25811de-etc-modprobe-d\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.452631 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.449955 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-host-var-lib-cni-multus\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.452631 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.450164 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-cni-binary-copy\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.452631 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.450446 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3a2951e-e5a2-49fc-a938-a5617748d2bb-serviceca\") pod \"node-ca-xqr2c\" (UID: \"e3a2951e-e5a2-49fc-a938-a5617748d2bb\") " pod="openshift-image-registry/node-ca-xqr2c" Jul 17 05:41:18.452631 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.451112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-multus-daemon-config\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.452631 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.452531 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e59750b-d74b-4632-af2b-d509d25811de-tmp\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.452631 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.452567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e59750b-d74b-4632-af2b-d509d25811de-etc-tuned\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.456568 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.456469 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqlh\" (UniqueName: \"kubernetes.io/projected/45246d9d-2c17-4c14-adec-a8e2d2dfa6ac-kube-api-access-gkqlh\") pod \"multus-c56wh\" (UID: \"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac\") " pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.459699 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.459612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47cf6\" (UniqueName: \"kubernetes.io/projected/7e59750b-d74b-4632-af2b-d509d25811de-kube-api-access-47cf6\") pod \"tuned-bjh7n\" (UID: \"7e59750b-d74b-4632-af2b-d509d25811de\") " pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.461200 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.461176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv9w7\" (UniqueName: \"kubernetes.io/projected/e3a2951e-e5a2-49fc-a938-a5617748d2bb-kube-api-access-mv9w7\") pod \"node-ca-xqr2c\" (UID: \"e3a2951e-e5a2-49fc-a938-a5617748d2bb\") " pod="openshift-image-registry/node-ca-xqr2c" Jul 17 05:41:18.534512 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.534476 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7phbz" Jul 17 05:41:18.543066 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.542777 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" Jul 17 05:41:18.545142 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:18.545113 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d8ad5a_f134_43f0_a882_1e4c6eaa9b5c.slice/crio-6cb113ecad558681deace9403b4031e8b9aa1a018aefb6837e5049967a523e9d WatchSource:0}: Error finding container 6cb113ecad558681deace9403b4031e8b9aa1a018aefb6837e5049967a523e9d: Status 404 returned error can't find the container with id 6cb113ecad558681deace9403b4031e8b9aa1a018aefb6837e5049967a523e9d Jul 17 05:41:18.551840 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.551781 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gw9qs" Jul 17 05:41:18.552971 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:18.552893 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6517c465_15e5_4290_ac8a_5c9f6d2e05fb.slice/crio-455a1c65c2b0afb9a6da7b288b4a1e3e516394d46c386306d171ceb060ac132f WatchSource:0}: Error finding container 455a1c65c2b0afb9a6da7b288b4a1e3e516394d46c386306d171ceb060ac132f: Status 404 returned error can't find the container with id 455a1c65c2b0afb9a6da7b288b4a1e3e516394d46c386306d171ceb060ac132f Jul 17 05:41:18.557903 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.557873 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:18.562064 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:18.562021 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6d89cf_4331_4c48_91b6_5e6b570d8927.slice/crio-904719b0d61f80465d4a6fc68043ffdb1eecb9decde6aee843fe0e33228f2320 WatchSource:0}: Error finding container 904719b0d61f80465d4a6fc68043ffdb1eecb9decde6aee843fe0e33228f2320: Status 404 returned error can't find the container with id 904719b0d61f80465d4a6fc68043ffdb1eecb9decde6aee843fe0e33228f2320 Jul 17 05:41:18.565657 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.565465 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8wg7p" Jul 17 05:41:18.569041 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:18.569012 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddecbb1a_d4df_4a32_85bb_793654f86198.slice/crio-0f4dbfee05acf298578e790248f5863c122f0f37194c77c81d97b745adace779 WatchSource:0}: Error finding container 0f4dbfee05acf298578e790248f5863c122f0f37194c77c81d97b745adace779: Status 404 returned error can't find the container with id 0f4dbfee05acf298578e790248f5863c122f0f37194c77c81d97b745adace779 Jul 17 05:41:18.574916 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.574893 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hcsh4" Jul 17 05:41:18.575758 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:18.575731 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71cd888f_6076_4d46_ba8d_ba55fcee848c.slice/crio-ff5807271ecb6174b1ef4e58f91c720c8296617db81519c8d0818e7c7900cee8 WatchSource:0}: Error finding container ff5807271ecb6174b1ef4e58f91c720c8296617db81519c8d0818e7c7900cee8: Status 404 returned error can't find the container with id ff5807271ecb6174b1ef4e58f91c720c8296617db81519c8d0818e7c7900cee8 Jul 17 05:41:18.583388 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.583358 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c56wh" Jul 17 05:41:18.585424 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:18.585264 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0600e268_96d5_4b77_893d_09eb28d97ebd.slice/crio-30131986eefd0bfbf9e8a9c081ce757dc172967a2014ff9663e4579110e5b524 WatchSource:0}: Error finding container 30131986eefd0bfbf9e8a9c081ce757dc172967a2014ff9663e4579110e5b524: Status 404 returned error can't find the container with id 30131986eefd0bfbf9e8a9c081ce757dc172967a2014ff9663e4579110e5b524 Jul 17 05:41:18.591747 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.591557 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" Jul 17 05:41:18.592889 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:18.592859 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45246d9d_2c17_4c14_adec_a8e2d2dfa6ac.slice/crio-3655657a7e91c0c6f135648cd0ebaeb937884507343281e59d854901bb23cb04 WatchSource:0}: Error finding container 3655657a7e91c0c6f135648cd0ebaeb937884507343281e59d854901bb23cb04: Status 404 returned error can't find the container with id 3655657a7e91c0c6f135648cd0ebaeb937884507343281e59d854901bb23cb04 Jul 17 05:41:18.597287 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.597160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xqr2c" Jul 17 05:41:18.603317 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:18.603286 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e59750b_d74b_4632_af2b_d509d25811de.slice/crio-f173938f8a6bb7dd8ce0481d005deb2078e84e4a85a531342af6c70c33f0b5c2 WatchSource:0}: Error finding container f173938f8a6bb7dd8ce0481d005deb2078e84e4a85a531342af6c70c33f0b5c2: Status 404 returned error can't find the container with id f173938f8a6bb7dd8ce0481d005deb2078e84e4a85a531342af6c70c33f0b5c2 Jul 17 05:41:18.609102 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:18.609075 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a2951e_e5a2_49fc_a938_a5617748d2bb.slice/crio-6005859c1cba5d3db6ba32437fb395328dc589d7a2771ed2b3c9d038b127888b WatchSource:0}: Error finding container 6005859c1cba5d3db6ba32437fb395328dc589d7a2771ed2b3c9d038b127888b: Status 404 returned error can't find the container with id 6005859c1cba5d3db6ba32437fb395328dc589d7a2771ed2b3c9d038b127888b Jul 17 05:41:18.854121 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.854033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:18.854121 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.854089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22pw\" (UniqueName: \"kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw\") pod \"network-check-target-5b6z6\" (UID: \"ac6cc6e6-4369-4fbc-898b-13c449fe4c92\") " pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:18.854121 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.854114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:18.854922 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.854212 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:18.854922 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.854214 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:18.854922 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.854266 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret podName:55c6188c-0a2e-4607-8c0e-7a5685859c4d nodeName:}" failed. No retries permitted until 2026-07-17 05:41:19.854252534 +0000 UTC m=+4.598931840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret") pod "global-pull-secret-syncer-t6rpv" (UID: "55c6188c-0a2e-4607-8c0e-7a5685859c4d") : object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:18.854922 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.854285 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs podName:3c472db7-71bf-46b2-9452-f936c8c53e67 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:19.854276045 +0000 UTC m=+4.598955363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs") pod "network-metrics-daemon-nsrf4" (UID: "3c472db7-71bf-46b2-9452-f936c8c53e67") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:18.854922 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.854226 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 05:41:18.854922 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.854332 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 05:41:18.854922 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.854345 2580 projected.go:196] Error preparing data for projected volume kube-api-access-l22pw for pod openshift-network-diagnostics/network-check-target-5b6z6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:18.854922 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:18.854417 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw podName:ac6cc6e6-4369-4fbc-898b-13c449fe4c92 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:19.854396668 +0000 UTC m=+4.599075990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l22pw" (UniqueName: "kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw") pod "network-check-target-5b6z6" (UID: "ac6cc6e6-4369-4fbc-898b-13c449fe4c92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:18.891069 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.891030 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7phbz" event={"ID":"12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c","Type":"ContainerStarted","Data":"6cb113ecad558681deace9403b4031e8b9aa1a018aefb6837e5049967a523e9d"} Jul 17 05:41:18.892552 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.892512 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" event={"ID":"7e59750b-d74b-4632-af2b-d509d25811de","Type":"ContainerStarted","Data":"f173938f8a6bb7dd8ce0481d005deb2078e84e4a85a531342af6c70c33f0b5c2"} Jul 17 05:41:18.893755 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.893728 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hcsh4" event={"ID":"0600e268-96d5-4b77-893d-09eb28d97ebd","Type":"ContainerStarted","Data":"30131986eefd0bfbf9e8a9c081ce757dc172967a2014ff9663e4579110e5b524"} Jul 17 05:41:18.894909 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.894882 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wg7p" event={"ID":"71cd888f-6076-4d46-ba8d-ba55fcee848c","Type":"ContainerStarted","Data":"ff5807271ecb6174b1ef4e58f91c720c8296617db81519c8d0818e7c7900cee8"} Jul 17 05:41:18.896118 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.896094 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gw9qs" event={"ID":"ca6d89cf-4331-4c48-91b6-5e6b570d8927","Type":"ContainerStarted","Data":"904719b0d61f80465d4a6fc68043ffdb1eecb9decde6aee843fe0e33228f2320"} Jul 17 05:41:18.897304 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.897271 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" event={"ID":"6517c465-15e5-4290-ac8a-5c9f6d2e05fb","Type":"ContainerStarted","Data":"455a1c65c2b0afb9a6da7b288b4a1e3e516394d46c386306d171ceb060ac132f"} Jul 17 05:41:18.898482 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.898460 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xqr2c" event={"ID":"e3a2951e-e5a2-49fc-a938-a5617748d2bb","Type":"ContainerStarted","Data":"6005859c1cba5d3db6ba32437fb395328dc589d7a2771ed2b3c9d038b127888b"} Jul 17 05:41:18.899644 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.899611 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c56wh" event={"ID":"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac","Type":"ContainerStarted","Data":"3655657a7e91c0c6f135648cd0ebaeb937884507343281e59d854901bb23cb04"} Jul 17 05:41:18.900895 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:18.900851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" event={"ID":"ddecbb1a-d4df-4a32-85bb-793654f86198","Type":"ContainerStarted","Data":"0f4dbfee05acf298578e790248f5863c122f0f37194c77c81d97b745adace779"} Jul 17 05:41:19.271101 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:19.270967 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 05:36:18 +0000 UTC" deadline="2028-01-24 21:56:29.146569436 +0000 UTC" Jul 17 05:41:19.271101 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:19.271046 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13360h15m9.875528181s" Jul 17 05:41:19.863645 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:19.862746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22pw\" (UniqueName: \"kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw\") pod \"network-check-target-5b6z6\" (UID: \"ac6cc6e6-4369-4fbc-898b-13c449fe4c92\") " pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:19.863645 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:19.862813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:19.863645 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:19.862856 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:19.863645 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:19.862987 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:19.863645 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:19.863049 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs podName:3c472db7-71bf-46b2-9452-f936c8c53e67 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:21.863031133 +0000 UTC m=+6.607710444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs") pod "network-metrics-daemon-nsrf4" (UID: "3c472db7-71bf-46b2-9452-f936c8c53e67") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:19.863645 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:19.863131 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 05:41:19.863645 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:19.863148 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 05:41:19.863645 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:19.863160 2580 projected.go:196] Error preparing data for projected volume kube-api-access-l22pw for pod openshift-network-diagnostics/network-check-target-5b6z6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:19.863645 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:19.863194 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw podName:ac6cc6e6-4369-4fbc-898b-13c449fe4c92 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:21.863183829 +0000 UTC m=+6.607863149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l22pw" (UniqueName: "kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw") pod "network-check-target-5b6z6" (UID: "ac6cc6e6-4369-4fbc-898b-13c449fe4c92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:19.863645 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:19.863251 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:19.863645 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:19.863280 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret podName:55c6188c-0a2e-4607-8c0e-7a5685859c4d nodeName:}" failed. No retries permitted until 2026-07-17 05:41:21.863270416 +0000 UTC m=+6.607949726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret") pod "global-pull-secret-syncer-t6rpv" (UID: "55c6188c-0a2e-4607-8c0e-7a5685859c4d") : object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:19.884338 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:19.884293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:19.884537 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:19.884453 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:19.884891 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:19.884870 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:19.885008 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:19.884986 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:19.885079 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:19.885063 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:19.885168 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:19.885150 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:20.272130 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:20.272076 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 05:36:18 +0000 UTC" deadline="2028-02-26 12:48:34.669335201 +0000 UTC" Jul 17 05:41:20.272130 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:20.272109 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14143h7m14.397230537s" Jul 17 05:41:20.923151 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:20.922143 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal" event={"ID":"ba0e125b5dfaf98ff0a6e6b6f46e0aae","Type":"ContainerStarted","Data":"f91a226eca237781599e43b10868bd00d1ed94218756abd44e1ca997f7cb7b12"} Jul 17 05:41:21.883747 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:21.883533 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:21.883747 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:21.883613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22pw\" (UniqueName: \"kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw\") pod \"network-check-target-5b6z6\" (UID: \"ac6cc6e6-4369-4fbc-898b-13c449fe4c92\") " pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:21.883747 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:21.883670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:21.883747 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:21.883685 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:21.883747 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:21.883722 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:21.884122 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:21.883770 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs podName:3c472db7-71bf-46b2-9452-f936c8c53e67 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:25.883747988 +0000 UTC m=+10.628427304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs") pod "network-metrics-daemon-nsrf4" (UID: "3c472db7-71bf-46b2-9452-f936c8c53e67") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:21.884122 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:21.883829 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:21.884122 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:21.883838 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:21.884122 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:21.883879 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret podName:55c6188c-0a2e-4607-8c0e-7a5685859c4d nodeName:}" failed. No retries permitted until 2026-07-17 05:41:25.883862545 +0000 UTC m=+10.628541851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret") pod "global-pull-secret-syncer-t6rpv" (UID: "55c6188c-0a2e-4607-8c0e-7a5685859c4d") : object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:21.884122 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:21.883891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:21.884122 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:21.883963 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 05:41:21.884122 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:21.883971 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:21.884122 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:21.883981 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 05:41:21.884122 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:21.883691 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:21.884122 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:21.883995 2580 projected.go:196] Error preparing data for projected volume kube-api-access-l22pw for pod openshift-network-diagnostics/network-check-target-5b6z6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:21.884122 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:21.884032 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw podName:ac6cc6e6-4369-4fbc-898b-13c449fe4c92 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:25.884020689 +0000 UTC m=+10.628699998 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l22pw" (UniqueName: "kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw") pod "network-check-target-5b6z6" (UID: "ac6cc6e6-4369-4fbc-898b-13c449fe4c92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:21.884122 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:21.884046 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:21.927532 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:21.927248 2580 generic.go:356] "Generic (PLEG): container finished" podID="ab400b4fa7d3b1af72fd41bc419aef74" containerID="678677de4196c009c5c7fb600c6eba6b99022689e26318bf7948180745d4ce88" exitCode=0 Jul 17 05:41:21.927532 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:21.927435 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" event={"ID":"ab400b4fa7d3b1af72fd41bc419aef74","Type":"ContainerDied","Data":"678677de4196c009c5c7fb600c6eba6b99022689e26318bf7948180745d4ce88"} Jul 17 05:41:21.943097 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:21.942865 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-12.ec2.internal" podStartSLOduration=4.942845335 podStartE2EDuration="4.942845335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:20.937067908 +0000 UTC m=+5.681747236" watchObservedRunningTime="2026-07-17 05:41:21.942845335 +0000 UTC m=+6.687524663" Jul 17 05:41:23.884399 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:23.884320 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:23.884399 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:23.884363 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:23.884905 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:23.884457 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:23.884905 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:23.884471 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:23.884905 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:23.884576 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:23.884905 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:23.884654 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:25.884044 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:25.884007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:25.884044 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:25.884042 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:25.884569 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:25.884007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:25.884569 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:25.884165 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:25.884569 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:25.884244 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:25.884569 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:25.884360 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:25.917152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:25.917117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:25.917313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:25.917169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22pw\" (UniqueName: \"kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw\") pod \"network-check-target-5b6z6\" (UID: \"ac6cc6e6-4369-4fbc-898b-13c449fe4c92\") " pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:25.917313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:25.917194 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:25.917313 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:25.917295 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:25.917496 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:25.917340 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret podName:55c6188c-0a2e-4607-8c0e-7a5685859c4d nodeName:}" failed. No retries permitted until 2026-07-17 05:41:33.917327638 +0000 UTC m=+18.662006943 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret") pod "global-pull-secret-syncer-t6rpv" (UID: "55c6188c-0a2e-4607-8c0e-7a5685859c4d") : object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:25.917676 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:25.917630 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:25.917676 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:25.917675 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs podName:3c472db7-71bf-46b2-9452-f936c8c53e67 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:33.917661272 +0000 UTC m=+18.662340586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs") pod "network-metrics-daemon-nsrf4" (UID: "3c472db7-71bf-46b2-9452-f936c8c53e67") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:25.917784 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:25.917750 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 05:41:25.917784 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:25.917770 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 05:41:25.917784 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:25.917782 2580 projected.go:196] Error preparing data for projected volume kube-api-access-l22pw for pod openshift-network-diagnostics/network-check-target-5b6z6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:25.917921 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:25.917823 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw podName:ac6cc6e6-4369-4fbc-898b-13c449fe4c92 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:33.917810406 +0000 UTC m=+18.662489725 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l22pw" (UniqueName: "kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw") pod "network-check-target-5b6z6" (UID: "ac6cc6e6-4369-4fbc-898b-13c449fe4c92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:27.884445 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:27.884318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:27.884445 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:27.884335 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:27.884981 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:27.884477 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:27.884981 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:27.884466 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:27.884981 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:27.884588 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:27.884981 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:27.884696 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:29.883932 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:29.883838 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:29.883932 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:29.883886 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:29.884404 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:29.883875 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:29.884404 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:29.883990 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:29.884404 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:29.884068 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:29.884404 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:29.884156 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:31.884520 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:31.884477 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:31.884970 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:31.884528 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:31.884970 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:31.884631 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:31.884970 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:31.884649 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:31.884970 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:31.884712 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:31.884970 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:31.884799 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:33.884034 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:33.883995 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:33.884034 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:33.884025 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:33.884596 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:33.883995 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:33.884596 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:33.884134 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:33.884596 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:33.884197 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:33.884596 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:33.884280 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:33.976047 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:33.976011 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:33.976213 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:33.976074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:33.976213 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:33.976126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22pw\" (UniqueName: \"kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw\") pod \"network-check-target-5b6z6\" (UID: \"ac6cc6e6-4369-4fbc-898b-13c449fe4c92\") " pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:33.976213 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:33.976180 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:33.976213 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:33.976187 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:33.976416 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:33.976233 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 05:41:33.976416 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:33.976245 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret podName:55c6188c-0a2e-4607-8c0e-7a5685859c4d nodeName:}" failed. No retries permitted until 2026-07-17 05:41:49.976225386 +0000 UTC m=+34.720904706 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret") pod "global-pull-secret-syncer-t6rpv" (UID: "55c6188c-0a2e-4607-8c0e-7a5685859c4d") : object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:33.976416 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:33.976268 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs podName:3c472db7-71bf-46b2-9452-f936c8c53e67 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:49.976256172 +0000 UTC m=+34.720935480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs") pod "network-metrics-daemon-nsrf4" (UID: "3c472db7-71bf-46b2-9452-f936c8c53e67") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:33.976416 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:33.976247 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 05:41:33.976416 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:33.976310 2580 projected.go:196] Error preparing data for projected volume kube-api-access-l22pw for pod openshift-network-diagnostics/network-check-target-5b6z6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:33.976416 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:33.976357 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw podName:ac6cc6e6-4369-4fbc-898b-13c449fe4c92 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:49.976345901 +0000 UTC m=+34.721025219 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l22pw" (UniqueName: "kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw") pod "network-check-target-5b6z6" (UID: "ac6cc6e6-4369-4fbc-898b-13c449fe4c92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:35.884060 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:35.884015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:35.884060 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:35.884038 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:35.884060 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:35.884043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:35.884592 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:35.884140 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:35.884592 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:35.884219 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:35.884592 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:35.884293 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:37.884293 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:37.884256 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:37.884722 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:37.884360 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:37.884722 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:37.884364 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:37.884722 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:37.884406 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:37.884722 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:37.884515 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:37.884722 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:37.884626 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:38.960749 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.960294 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gw9qs" event={"ID":"ca6d89cf-4331-4c48-91b6-5e6b570d8927","Type":"ContainerStarted","Data":"7e52c44386bf83120ee6e6964dad5de421353f0fefc0d3e2defa75ea58fdcf7c"} Jul 17 05:41:38.961801 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.961778 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" event={"ID":"6517c465-15e5-4290-ac8a-5c9f6d2e05fb","Type":"ContainerStarted","Data":"44845be9d6bac3e030d71be4232409cc7040d9ccf61976f01333405b748e209b"} Jul 17 05:41:38.963190 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.963144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xqr2c" event={"ID":"e3a2951e-e5a2-49fc-a938-a5617748d2bb","Type":"ContainerStarted","Data":"1d9770e27f184de61390388c862cd6b6f591a28e6434991129bac3a9f84080d9"} Jul 17 05:41:38.964595 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.964573 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c56wh" event={"ID":"45246d9d-2c17-4c14-adec-a8e2d2dfa6ac","Type":"ContainerStarted","Data":"f0679bb62499937ce8bd8313ddc819cb94d674c68bea0b563a369ff9c6d38c29"} Jul 17 05:41:38.967554 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.967522 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" event={"ID":"ddecbb1a-d4df-4a32-85bb-793654f86198","Type":"ContainerStarted","Data":"a8f9f8c5fb5b88d367fd156edc2c756651e45b4fe83aa0a53cc38312eeacdec3"} Jul 17 05:41:38.967692 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.967582 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" event={"ID":"ddecbb1a-d4df-4a32-85bb-793654f86198","Type":"ContainerStarted","Data":"8c09c150c1b39e63d56654b06cb30500151ab791d9a97ad09fe2da8717b4771d"} Jul 17 05:41:38.967692 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.967639 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" event={"ID":"ddecbb1a-d4df-4a32-85bb-793654f86198","Type":"ContainerStarted","Data":"089e028f5d42345d7b2d2b7699e9d9468d9f0eef4e69d176b62848ea0678e660"} Jul 17 05:41:38.967692 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.967686 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" event={"ID":"ddecbb1a-d4df-4a32-85bb-793654f86198","Type":"ContainerStarted","Data":"dcf822f39e642ab666753bd7bf5ce57a60843f842153a1ea8c54cce155b8471e"} Jul 17 05:41:38.967843 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.967719 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" event={"ID":"ddecbb1a-d4df-4a32-85bb-793654f86198","Type":"ContainerStarted","Data":"f0ca00d7a421d0d9a3b82be20c8f13031117f15577b7569040622c8b9e738f1a"} Jul 17 05:41:38.967843 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.967748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" event={"ID":"ddecbb1a-d4df-4a32-85bb-793654f86198","Type":"ContainerStarted","Data":"355dff6f1692827256547dbf4223edc24ddd0b583a8c457a12bb2abcaa466750"} Jul 17 05:41:38.968807 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.968787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7phbz" event={"ID":"12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c","Type":"ContainerStarted","Data":"20d64821f27329f9b10ac6da93734632d2741fa0c26bd4cd23edb3879d1211f7"} Jul 17 05:41:38.970492 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.970471 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" event={"ID":"ab400b4fa7d3b1af72fd41bc419aef74","Type":"ContainerStarted","Data":"0dd6984e0e85550b74fe10ee125633fb92b73a24d7535d4bc6694216e706e130"} Jul 17 05:41:38.971853 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.971834 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" event={"ID":"7e59750b-d74b-4632-af2b-d509d25811de","Type":"ContainerStarted","Data":"2586d4eed04350a952f5ca88aae46ab111e7068cb2ee7174bb0483b6aa391e47"} Jul 17 05:41:38.973175 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.973155 2580 generic.go:356] "Generic (PLEG): container finished" podID="71cd888f-6076-4d46-ba8d-ba55fcee848c" containerID="caf0eefab8dd8679eb0742f2fbf2c9263e82b38d770220830c16db9e31863b5e" exitCode=0 Jul 17 05:41:38.973265 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.973182 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wg7p" event={"ID":"71cd888f-6076-4d46-ba8d-ba55fcee848c","Type":"ContainerDied","Data":"caf0eefab8dd8679eb0742f2fbf2c9263e82b38d770220830c16db9e31863b5e"} Jul 17 05:41:38.974676 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.974403 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gw9qs" podStartSLOduration=2.605718903 podStartE2EDuration="21.974371856s" totalImagesPullingTime="19.368652953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:38.973895895 +0000 UTC m=+23.718575234" watchObservedRunningTime="2026-07-17 05:41:38.974371856 +0000 UTC m=+23.719051183" Jul 17 05:41:38.989954 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:38.989881 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c56wh" podStartSLOduration=2.627266798 podStartE2EDuration="21.989857463s" totalImagesPullingTime="19.362590665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:38.989433638 +0000 UTC m=+23.734112965" watchObservedRunningTime="2026-07-17 05:41:38.989857463 +0000 UTC m=+23.734536791" Jul 17 05:41:39.005643 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:39.005574 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bjh7n" podStartSLOduration=2.67535527 podStartE2EDuration="22.005552728s" totalImagesPullingTime="19.330197458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:39.005207236 +0000 UTC m=+23.749886565" watchObservedRunningTime="2026-07-17 05:41:39.005552728 +0000 UTC m=+23.750232052" Jul 17 05:41:39.018547 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:39.018494 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-12.ec2.internal" podStartSLOduration=22.018475024 podStartE2EDuration="22.018475024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:39.018052073 +0000 UTC m=+23.762731401" watchObservedRunningTime="2026-07-17 05:41:39.018475024 +0000 UTC m=+23.763154351" Jul 17 05:41:39.055087 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:39.055032 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xqr2c" podStartSLOduration=2.732801326 podStartE2EDuration="22.055015753s" totalImagesPullingTime="19.322214427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:39.05458303 +0000 UTC m=+23.799262358" watchObservedRunningTime="2026-07-17 05:41:39.055015753 +0000 UTC m=+23.799695080" Jul 17 05:41:39.069776 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:39.069732 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7phbz" podStartSLOduration=12.000256041 podStartE2EDuration="23.069717788s" totalImagesPullingTime="11.069461747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:39.069215699 +0000 UTC m=+23.813895030" watchObservedRunningTime="2026-07-17 05:41:39.069717788 +0000 UTC m=+23.814397116" Jul 17 05:41:39.420094 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:39.420067 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 05:41:39.760878 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:39.760716 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T05:41:39.420085711Z","UUID":"703dce12-8ea0-4ca0-88ed-1aa5c10472e6","Handler":null,"Name":"","Endpoint":""} Jul 17 05:41:39.762397 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:39.762359 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 05:41:39.762519 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:39.762410 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 05:41:39.884463 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:39.884431 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:39.884728 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:39.884431 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:39.884728 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:39.884577 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:39.884728 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:39.884434 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:39.884728 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:39.884645 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:39.884728 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:39.884704 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:39.979309 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:39.979269 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hcsh4" event={"ID":"0600e268-96d5-4b77-893d-09eb28d97ebd","Type":"ContainerStarted","Data":"0c1cee71867ace4758ef81863eb63f14ab64af6a0c32833d607d2d1e9d7ae738"} Jul 17 05:41:39.981668 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:39.981642 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" event={"ID":"6517c465-15e5-4290-ac8a-5c9f6d2e05fb","Type":"ContainerStarted","Data":"74cce12bf359b58dec1920f9824f3a3246fa6c4d5d9bf97f281815dbc246747b"} Jul 17 05:41:39.993248 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:39.993200 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hcsh4" podStartSLOduration=3.648612052 podStartE2EDuration="22.993185709s" totalImagesPullingTime="19.344573657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:39.992887705 +0000 UTC m=+24.737567032" watchObservedRunningTime="2026-07-17 05:41:39.993185709 +0000 UTC m=+24.737865037" Jul 17 05:41:40.986773 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:40.986735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" event={"ID":"6517c465-15e5-4290-ac8a-5c9f6d2e05fb","Type":"ContainerStarted","Data":"dc4e292473b6313b4f367e1ccdd670fa841f25e6d6b7d411438e6b2cede3e672"} Jul 17 05:41:40.990750 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:40.990720 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" event={"ID":"ddecbb1a-d4df-4a32-85bb-793654f86198","Type":"ContainerStarted","Data":"5a9cc1f4a3fb4d0c064f250242f0224ed82a4fc2c01f561fb39d3e3289275ff3"} Jul 17 05:41:41.003146 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:41.003084 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4fwg" podStartSLOduration=2.428098005 podStartE2EDuration="24.003067946s" totalImagesPullingTime="21.574969941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:41.002687765 +0000 UTC m=+25.747367102" watchObservedRunningTime="2026-07-17 05:41:41.003067946 +0000 UTC m=+25.747747273" Jul 17 05:41:41.139312 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:41.139234 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7phbz" Jul 17 05:41:41.139904 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:41.139881 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7phbz" Jul 17 05:41:41.883789 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:41.883750 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:41.883789 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:41.883786 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:41.884057 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:41.883892 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:41.884057 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:41.883944 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:41.884057 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:41.883960 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:41.884186 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:41.884056 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:41.993869 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:41.993809 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7phbz" Jul 17 05:41:41.994346 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:41.994172 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7phbz" Jul 17 05:41:43.884461 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:43.884088 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:43.885115 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:43.884088 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:43.885115 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:43.884481 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:43.885115 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:43.884579 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:43.885115 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:43.884148 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:43.885115 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:43.884679 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:44.001698 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:44.001662 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" event={"ID":"ddecbb1a-d4df-4a32-85bb-793654f86198","Type":"ContainerStarted","Data":"4c0e035432b7c6ddd487cc56942060571e79466922de358727e07e8b2819bc28"} Jul 17 05:41:44.001903 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:44.001887 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:44.003682 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:44.003655 2580 generic.go:356] "Generic (PLEG): container finished" podID="71cd888f-6076-4d46-ba8d-ba55fcee848c" containerID="e6b22b4636923413554e1dfb33378826b1c0ab27002dc8333fe1774412fbbbd4" exitCode=0 Jul 17 05:41:44.003807 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:44.003745 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wg7p" event={"ID":"71cd888f-6076-4d46-ba8d-ba55fcee848c","Type":"ContainerDied","Data":"e6b22b4636923413554e1dfb33378826b1c0ab27002dc8333fe1774412fbbbd4"} Jul 17 05:41:44.017875 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:44.017853 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:44.025081 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:44.025045 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" podStartSLOduration=7.487307504 podStartE2EDuration="27.025033978s" totalImagesPullingTime="19.537726474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:44.02466615 +0000 UTC m=+28.769345478" watchObservedRunningTime="2026-07-17 05:41:44.025033978 +0000 UTC m=+28.769713305" Jul 17 05:41:45.006815 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:45.006785 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 17 05:41:45.007622 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:45.007214 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:45.024024 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:45.024002 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:45.563666 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:45.563630 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t6rpv"] Jul 17 05:41:45.563845 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:45.563768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:45.563911 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:45.563876 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:45.566739 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:45.566707 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5b6z6"] Jul 17 05:41:45.567007 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:45.566855 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:45.567082 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:45.567004 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:45.567339 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:45.567319 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nsrf4"] Jul 17 05:41:45.567476 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:45.567439 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:45.567562 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:45.567543 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:46.010401 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:46.010349 2580 generic.go:356] "Generic (PLEG): container finished" podID="71cd888f-6076-4d46-ba8d-ba55fcee848c" containerID="88f169488111b8d6d3e37366330fafb43b25fbac492678534e10967b2254f0b0" exitCode=0 Jul 17 05:41:46.010792 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:46.010415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wg7p" event={"ID":"71cd888f-6076-4d46-ba8d-ba55fcee848c","Type":"ContainerDied","Data":"88f169488111b8d6d3e37366330fafb43b25fbac492678534e10967b2254f0b0"} Jul 17 05:41:46.010792 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:46.010631 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 17 05:41:46.073515 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:46.073482 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:41:46.884978 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:46.884940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:46.885142 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:46.884988 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:46.885142 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:46.885069 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:46.885240 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:46.885192 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:47.883989 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:47.883751 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:47.884470 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:47.884012 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:48.019868 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:48.019827 2580 generic.go:356] "Generic (PLEG): container finished" podID="71cd888f-6076-4d46-ba8d-ba55fcee848c" containerID="bce3c460b6f6676cb09e3ecb073a04079b1b9fa10885de7eac82696e11d10a4c" exitCode=0 Jul 17 05:41:48.020046 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:48.019911 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wg7p" event={"ID":"71cd888f-6076-4d46-ba8d-ba55fcee848c","Type":"ContainerDied","Data":"bce3c460b6f6676cb09e3ecb073a04079b1b9fa10885de7eac82696e11d10a4c"} Jul 17 05:41:48.033669 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:48.033635 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" podUID="ddecbb1a-d4df-4a32-85bb-793654f86198" containerName="ovnkube-controller" probeResult="failure" output="" Jul 17 05:41:48.887697 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:48.887663 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:48.888220 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:48.887782 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:41:48.888220 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:48.887833 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:48.888220 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:48.887943 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6rpv" podUID="55c6188c-0a2e-4607-8c0e-7a5685859c4d" Jul 17 05:41:49.883747 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:49.883708 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:49.883950 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:49.883824 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5b6z6" podUID="ac6cc6e6-4369-4fbc-898b-13c449fe4c92" Jul 17 05:41:50.000272 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.000212 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22pw\" (UniqueName: \"kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw\") pod \"network-check-target-5b6z6\" (UID: \"ac6cc6e6-4369-4fbc-898b-13c449fe4c92\") " pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:50.000272 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.000277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:50.000860 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.000317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:50.000860 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:50.000452 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:50.000860 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:50.000453 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:50.000860 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:50.000457 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 05:41:50.000860 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:50.000616 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 05:41:50.000860 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:50.000632 2580 projected.go:196] Error preparing data for projected volume kube-api-access-l22pw for pod openshift-network-diagnostics/network-check-target-5b6z6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:50.000860 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:50.000542 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs podName:3c472db7-71bf-46b2-9452-f936c8c53e67 nodeName:}" failed. No retries permitted until 2026-07-17 05:42:22.000513907 +0000 UTC m=+66.745193217 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs") pod "network-metrics-daemon-nsrf4" (UID: "3c472db7-71bf-46b2-9452-f936c8c53e67") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:50.000860 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:50.000698 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret podName:55c6188c-0a2e-4607-8c0e-7a5685859c4d nodeName:}" failed. No retries permitted until 2026-07-17 05:42:22.000660685 +0000 UTC m=+66.745339998 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret") pod "global-pull-secret-syncer-t6rpv" (UID: "55c6188c-0a2e-4607-8c0e-7a5685859c4d") : object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:50.000860 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:50.000722 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw podName:ac6cc6e6-4369-4fbc-898b-13c449fe4c92 nodeName:}" failed. No retries permitted until 2026-07-17 05:42:22.000711494 +0000 UTC m=+66.745390800 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l22pw" (UniqueName: "kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw") pod "network-check-target-5b6z6" (UID: "ac6cc6e6-4369-4fbc-898b-13c449fe4c92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:50.539033 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.539000 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-12.ec2.internal" event="NodeReady" Jul 17 05:41:50.539195 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.539163 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 05:41:50.577967 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.577922 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nhv84"] Jul 17 05:41:50.589759 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.589736 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dsrvg"] Jul 17 05:41:50.589915 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.589880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:50.592643 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.592615 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 05:41:50.592753 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.592722 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jx7lr\"" Jul 17 05:41:50.592844 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.592829 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 05:41:50.598079 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.598053 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nhv84"] Jul 17 05:41:50.598226 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.598098 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dsrvg"] Jul 17 05:41:50.598226 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.598180 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dsrvg" Jul 17 05:41:50.600493 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.600473 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 05:41:50.600599 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.600475 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tmkvs\"" Jul 17 05:41:50.600599 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.600536 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 05:41:50.600599 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.600557 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 05:41:50.705278 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.705244 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx6pb\" (UniqueName: \"kubernetes.io/projected/3fde696a-f615-4e9f-b866-84073d4a923b-kube-api-access-lx6pb\") pod \"ingress-canary-dsrvg\" (UID: \"3fde696a-f615-4e9f-b866-84073d4a923b\") " pod="openshift-ingress-canary/ingress-canary-dsrvg" Jul 17 05:41:50.705465 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.705297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z89qf\" (UniqueName: \"kubernetes.io/projected/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-kube-api-access-z89qf\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:50.705465 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.705324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fde696a-f615-4e9f-b866-84073d4a923b-cert\") pod \"ingress-canary-dsrvg\" (UID: \"3fde696a-f615-4e9f-b866-84073d4a923b\") " pod="openshift-ingress-canary/ingress-canary-dsrvg" Jul 17 05:41:50.705465 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.705416 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-config-volume\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:50.705615 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.705468 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:50.705615 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.705494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-tmp-dir\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:50.805957 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.805856 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z89qf\" (UniqueName: \"kubernetes.io/projected/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-kube-api-access-z89qf\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:50.805957 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.805904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fde696a-f615-4e9f-b866-84073d4a923b-cert\") pod \"ingress-canary-dsrvg\" (UID: \"3fde696a-f615-4e9f-b866-84073d4a923b\") " pod="openshift-ingress-canary/ingress-canary-dsrvg" Jul 17 05:41:50.805957 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.805945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-config-volume\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:50.806194 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.806001 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:50.806298 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.806269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-tmp-dir\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:50.806431 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.806332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx6pb\" (UniqueName: \"kubernetes.io/projected/3fde696a-f615-4e9f-b866-84073d4a923b-kube-api-access-lx6pb\") pod \"ingress-canary-dsrvg\" (UID: \"3fde696a-f615-4e9f-b866-84073d4a923b\") " pod="openshift-ingress-canary/ingress-canary-dsrvg" Jul 17 05:41:50.806546 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:50.806500 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:41:50.806628 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:50.806571 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls podName:f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe nodeName:}" failed. No retries permitted until 2026-07-17 05:41:51.306550522 +0000 UTC m=+36.051229840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls") pod "dns-default-nhv84" (UID: "f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe") : secret "dns-default-metrics-tls" not found Jul 17 05:41:50.806696 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.806646 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-config-volume\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:50.806753 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.806710 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-tmp-dir\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:50.811264 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.811235 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fde696a-f615-4e9f-b866-84073d4a923b-cert\") pod \"ingress-canary-dsrvg\" (UID: \"3fde696a-f615-4e9f-b866-84073d4a923b\") " pod="openshift-ingress-canary/ingress-canary-dsrvg" Jul 17 05:41:50.814799 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.814772 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z89qf\" (UniqueName: \"kubernetes.io/projected/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-kube-api-access-z89qf\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:50.814998 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.814976 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx6pb\" (UniqueName: \"kubernetes.io/projected/3fde696a-f615-4e9f-b866-84073d4a923b-kube-api-access-lx6pb\") pod \"ingress-canary-dsrvg\" (UID: \"3fde696a-f615-4e9f-b866-84073d4a923b\") " pod="openshift-ingress-canary/ingress-canary-dsrvg" Jul 17 05:41:50.883982 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.883942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:41:50.884191 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.883942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:41:50.886482 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.886451 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 05:41:50.886611 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.886502 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 05:41:50.886611 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.886517 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7x5wz\"" Jul 17 05:41:50.908472 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:50.908449 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dsrvg" Jul 17 05:41:51.311178 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:51.311147 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:51.311896 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:51.311303 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:41:51.311896 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:51.311368 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls podName:f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe nodeName:}" failed. No retries permitted until 2026-07-17 05:41:52.311351755 +0000 UTC m=+37.056031060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls") pod "dns-default-nhv84" (UID: "f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe") : secret "dns-default-metrics-tls" not found Jul 17 05:41:51.883742 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:51.883704 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:41:51.886275 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:51.886249 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 05:41:51.886441 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:51.886306 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 05:41:51.886441 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:51.886246 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9jxzr\"" Jul 17 05:41:52.319104 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:52.319073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:52.319486 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:52.319227 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:41:52.319486 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:52.319290 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls podName:f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe nodeName:}" failed. No retries permitted until 2026-07-17 05:41:54.319273685 +0000 UTC m=+39.063953002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls") pod "dns-default-nhv84" (UID: "f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe") : secret "dns-default-metrics-tls" not found Jul 17 05:41:52.349628 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:52.349587 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dsrvg"] Jul 17 05:41:52.406547 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:41:52.406513 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fde696a_f615_4e9f_b866_84073d4a923b.slice/crio-4fa52f31e22ee33ef90b5d7f723e046d7840687f772bd6974c2bb5e6a8d354ac WatchSource:0}: Error finding container 4fa52f31e22ee33ef90b5d7f723e046d7840687f772bd6974c2bb5e6a8d354ac: Status 404 returned error can't find the container with id 4fa52f31e22ee33ef90b5d7f723e046d7840687f772bd6974c2bb5e6a8d354ac Jul 17 05:41:53.036081 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:53.036039 2580 generic.go:356] "Generic (PLEG): container finished" podID="71cd888f-6076-4d46-ba8d-ba55fcee848c" containerID="016b98c48ca28aa21c3fac5ee6fa2ab1b612b5fe9174aa0880ea0dd3f4c193ca" exitCode=0 Jul 17 05:41:53.036277 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:53.036122 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wg7p" event={"ID":"71cd888f-6076-4d46-ba8d-ba55fcee848c","Type":"ContainerDied","Data":"016b98c48ca28aa21c3fac5ee6fa2ab1b612b5fe9174aa0880ea0dd3f4c193ca"} Jul 17 05:41:53.037461 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:53.037423 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dsrvg" event={"ID":"3fde696a-f615-4e9f-b866-84073d4a923b","Type":"ContainerStarted","Data":"4fa52f31e22ee33ef90b5d7f723e046d7840687f772bd6974c2bb5e6a8d354ac"} Jul 17 05:41:53.657879 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:53.657841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:41:53.739434 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:53.739400 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:41:53.787325 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:53.787291 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:41:54.044506 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:54.044424 2580 generic.go:356] "Generic (PLEG): container finished" podID="71cd888f-6076-4d46-ba8d-ba55fcee848c" containerID="2730a81d768dfa93f0d9fa24dae24a0a30df710e7eb1bb14c043fae3817b42e5" exitCode=0 Jul 17 05:41:54.044669 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:54.044502 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wg7p" event={"ID":"71cd888f-6076-4d46-ba8d-ba55fcee848c","Type":"ContainerDied","Data":"2730a81d768dfa93f0d9fa24dae24a0a30df710e7eb1bb14c043fae3817b42e5"} Jul 17 05:41:54.117823 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:54.117785 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:41:54.168873 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:54.168838 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:41:54.334261 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:54.334168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:54.334469 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:54.334339 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:41:54.334469 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:54.334432 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls podName:f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe nodeName:}" failed. No retries permitted until 2026-07-17 05:41:58.334409702 +0000 UTC m=+43.079089011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls") pod "dns-default-nhv84" (UID: "f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe") : secret "dns-default-metrics-tls" not found Jul 17 05:41:55.049587 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:55.049401 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8wg7p" event={"ID":"71cd888f-6076-4d46-ba8d-ba55fcee848c","Type":"ContainerStarted","Data":"0d5e9961b0bf2b249655e2b14feab90584c766a37f7717ccf3f94f22dca67c4d"} Jul 17 05:41:55.050905 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:55.050881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dsrvg" event={"ID":"3fde696a-f615-4e9f-b866-84073d4a923b","Type":"ContainerStarted","Data":"a8cc80834abc8a17340d275f397d0c8463e2d37e1da8aa4e3fa17d2721b23964"} Jul 17 05:41:55.071710 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:55.070895 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8wg7p" podStartSLOduration=7.765003773 podStartE2EDuration="38.070867117s" totalImagesPullingTime="30.305863344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:55.069667891 +0000 UTC m=+39.814347218" watchObservedRunningTime="2026-07-17 05:41:55.070867117 +0000 UTC m=+39.815546443" Jul 17 05:41:55.084577 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:55.084485 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dsrvg" podStartSLOduration=2.979093729 podStartE2EDuration="5.084461792s" totalImagesPullingTime="2.105368063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:55.083941429 +0000 UTC m=+39.828620757" watchObservedRunningTime="2026-07-17 05:41:55.084461792 +0000 UTC m=+39.829141098" Jul 17 05:41:55.198073 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:55.198038 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:41:55.298444 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:55.298353 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:41:55.349875 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:55.349839 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:41:57.401112 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:57.401081 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:41:57.451642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:57.451608 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:41:57.503075 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:57.503042 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:41:58.247268 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:58.247226 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:41:58.301229 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:58.301195 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:41:58.363444 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:41:58.363409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:41:58.363608 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:58.363553 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:41:58.363655 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:41:58.363626 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls podName:f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe nodeName:}" failed. No retries permitted until 2026-07-17 05:42:06.363608307 +0000 UTC m=+51.108287616 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls") pod "dns-default-nhv84" (UID: "f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe") : secret "dns-default-metrics-tls" not found Jul 17 05:42:06.413308 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:06.413257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:42:06.413903 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:42:06.413410 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:42:06.413903 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:42:06.413491 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls podName:f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe nodeName:}" failed. No retries permitted until 2026-07-17 05:42:22.413470363 +0000 UTC m=+67.158149672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls") pod "dns-default-nhv84" (UID: "f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe") : secret "dns-default-metrics-tls" not found Jul 17 05:42:15.627242 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:15.627206 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:42:15.676917 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:15.676884 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:42:18.029753 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:18.029725 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jvdks" Jul 17 05:42:18.177910 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:18.177847 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:42:18.228150 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:18.228116 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:42:19.417627 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:19.417596 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:42:19.513466 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:19.513434 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:42:20.399585 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.399550 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps"] Jul 17 05:42:20.402395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.402363 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps" Jul 17 05:42:20.403087 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.403065 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj"] Jul 17 05:42:20.405479 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.405462 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.405978 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.405947 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 17 05:42:20.405978 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.405974 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 17 05:42:20.406099 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.405989 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9942l\"" Jul 17 05:42:20.406099 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.405973 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 17 05:42:20.406099 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.405954 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 17 05:42:20.407903 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.407886 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 17 05:42:20.408088 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.407888 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 17 05:42:20.408246 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.408226 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 17 05:42:20.408246 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.408238 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 17 05:42:20.411759 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.411736 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps"] Jul 17 05:42:20.415458 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.415434 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj"] Jul 17 05:42:20.499499 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.499460 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.499499 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.499500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-additional-service-ca\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.499993 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.499527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7gbh\" (UniqueName: \"kubernetes.io/projected/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-kube-api-access-s7gbh\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.499993 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.499560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.499993 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.499658 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-ca\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.499993 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.499684 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-hub\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.499993 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.499716 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh9dj\" (UniqueName: \"kubernetes.io/projected/0d2fe440-baae-411a-aef5-059f4e6f2cc8-kube-api-access-fh9dj\") pod \"managed-serviceaccount-addon-agent-65cb699598-tm5ps\" (UID: \"0d2fe440-baae-411a-aef5-059f4e6f2cc8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps" Jul 17 05:42:20.499993 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.499764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d2fe440-baae-411a-aef5-059f4e6f2cc8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65cb699598-tm5ps\" (UID: \"0d2fe440-baae-411a-aef5-059f4e6f2cc8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps" Jul 17 05:42:20.600993 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.600959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.601140 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.601021 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-ca\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.601140 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.601045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-hub\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.601140 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.601078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh9dj\" (UniqueName: \"kubernetes.io/projected/0d2fe440-baae-411a-aef5-059f4e6f2cc8-kube-api-access-fh9dj\") pod \"managed-serviceaccount-addon-agent-65cb699598-tm5ps\" (UID: \"0d2fe440-baae-411a-aef5-059f4e6f2cc8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps" Jul 17 05:42:20.601140 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.601120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d2fe440-baae-411a-aef5-059f4e6f2cc8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65cb699598-tm5ps\" (UID: \"0d2fe440-baae-411a-aef5-059f4e6f2cc8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps" Jul 17 05:42:20.601361 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.601154 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.601361 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.601177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-additional-service-ca\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.601361 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.601205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7gbh\" (UniqueName: \"kubernetes.io/projected/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-kube-api-access-s7gbh\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.602186 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.602158 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-additional-service-ca\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.605237 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.605209 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.605237 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.605217 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-ca\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.605358 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.605215 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.605358 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.605282 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d2fe440-baae-411a-aef5-059f4e6f2cc8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65cb699598-tm5ps\" (UID: \"0d2fe440-baae-411a-aef5-059f4e6f2cc8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps" Jul 17 05:42:20.605452 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.605402 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-hub\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.608356 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.608335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7gbh\" (UniqueName: \"kubernetes.io/projected/5be596d5-da4f-4d3a-81a8-ecac9b05ffda-kube-api-access-s7gbh\") pod \"cluster-proxy-proxy-agent-778cd776d-njfzj\" (UID: \"5be596d5-da4f-4d3a-81a8-ecac9b05ffda\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.608589 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.608563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh9dj\" (UniqueName: \"kubernetes.io/projected/0d2fe440-baae-411a-aef5-059f4e6f2cc8-kube-api-access-fh9dj\") pod \"managed-serviceaccount-addon-agent-65cb699598-tm5ps\" (UID: \"0d2fe440-baae-411a-aef5-059f4e6f2cc8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps" Jul 17 05:42:20.724558 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.724458 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps" Jul 17 05:42:20.729530 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.729504 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:42:20.853106 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.853073 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj"] Jul 17 05:42:20.856722 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:42:20.856681 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5be596d5_da4f_4d3a_81a8_ecac9b05ffda.slice/crio-65ba7611c511c295fc087ba883033228a5607457727cdf6da8b69e64b3147fd4 WatchSource:0}: Error finding container 65ba7611c511c295fc087ba883033228a5607457727cdf6da8b69e64b3147fd4: Status 404 returned error can't find the container with id 65ba7611c511c295fc087ba883033228a5607457727cdf6da8b69e64b3147fd4 Jul 17 05:42:20.866328 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:20.866302 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps"] Jul 17 05:42:20.868884 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:42:20.868862 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d2fe440_baae_411a_aef5_059f4e6f2cc8.slice/crio-b3a30d95af991d702e9d94d9d2f646c3bfaf710a908a38fc64e5530a4f604e9a WatchSource:0}: Error finding container b3a30d95af991d702e9d94d9d2f646c3bfaf710a908a38fc64e5530a4f604e9a: Status 404 returned error can't find the container with id b3a30d95af991d702e9d94d9d2f646c3bfaf710a908a38fc64e5530a4f604e9a Jul 17 05:42:21.113918 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:21.113885 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" event={"ID":"5be596d5-da4f-4d3a-81a8-ecac9b05ffda","Type":"ContainerStarted","Data":"65ba7611c511c295fc087ba883033228a5607457727cdf6da8b69e64b3147fd4"} Jul 17 05:42:21.114855 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:21.114822 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps" event={"ID":"0d2fe440-baae-411a-aef5-059f4e6f2cc8","Type":"ContainerStarted","Data":"b3a30d95af991d702e9d94d9d2f646c3bfaf710a908a38fc64e5530a4f604e9a"} Jul 17 05:42:22.012361 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.012283 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:42:22.012877 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.012551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:42:22.012877 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.012633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22pw\" (UniqueName: \"kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw\") pod \"network-check-target-5b6z6\" (UID: \"ac6cc6e6-4369-4fbc-898b-13c449fe4c92\") " pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:42:22.015553 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.015271 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 05:42:22.015553 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.015280 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 05:42:22.015553 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.015280 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 05:42:22.023333 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:42:22.023303 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 17 05:42:22.023499 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:42:22.023426 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs podName:3c472db7-71bf-46b2-9452-f936c8c53e67 nodeName:}" failed. No retries permitted until 2026-07-17 05:43:26.023400997 +0000 UTC m=+130.768080316 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs") pod "network-metrics-daemon-nsrf4" (UID: "3c472db7-71bf-46b2-9452-f936c8c53e67") : secret "metrics-daemon-secret" not found Jul 17 05:42:22.025625 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.025598 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 05:42:22.030070 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.030012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55c6188c-0a2e-4607-8c0e-7a5685859c4d-original-pull-secret\") pod \"global-pull-secret-syncer-t6rpv\" (UID: \"55c6188c-0a2e-4607-8c0e-7a5685859c4d\") " pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:42:22.042666 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.042592 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l22pw\" (UniqueName: \"kubernetes.io/projected/ac6cc6e6-4369-4fbc-898b-13c449fe4c92-kube-api-access-l22pw\") pod \"network-check-target-5b6z6\" (UID: \"ac6cc6e6-4369-4fbc-898b-13c449fe4c92\") " pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:42:22.101325 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.101016 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6rpv" Jul 17 05:42:22.197125 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.197081 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9jxzr\"" Jul 17 05:42:22.206046 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.206012 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:42:22.279845 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.279684 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t6rpv"] Jul 17 05:42:22.417927 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.417849 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5b6z6"] Jul 17 05:42:22.422064 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:22.422035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:42:22.422516 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:42:22.422213 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:42:22.422516 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:42:22.422290 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls podName:f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe nodeName:}" failed. No retries permitted until 2026-07-17 05:42:54.422268706 +0000 UTC m=+99.166948014 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls") pod "dns-default-nhv84" (UID: "f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe") : secret "dns-default-metrics-tls" not found Jul 17 05:42:22.422925 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:42:22.422899 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6cc6e6_4369_4fbc_898b_13c449fe4c92.slice/crio-2a1965a75c3826530afc596a931a4f591bba79d5b92bf54c15e58bae3d13771f WatchSource:0}: Error finding container 2a1965a75c3826530afc596a931a4f591bba79d5b92bf54c15e58bae3d13771f: Status 404 returned error can't find the container with id 2a1965a75c3826530afc596a931a4f591bba79d5b92bf54c15e58bae3d13771f Jul 17 05:42:23.125668 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:23.125585 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5b6z6" event={"ID":"ac6cc6e6-4369-4fbc-898b-13c449fe4c92","Type":"ContainerStarted","Data":"2a1965a75c3826530afc596a931a4f591bba79d5b92bf54c15e58bae3d13771f"} Jul 17 05:42:23.127169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:23.127140 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t6rpv" event={"ID":"55c6188c-0a2e-4607-8c0e-7a5685859c4d","Type":"ContainerStarted","Data":"122f02aef846f6bf48b1b644030aab26bf669e289c4c5dd5c32a95d6ec7ffba0"} Jul 17 05:42:28.845730 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:28.845690 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:42:28.897457 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:28.897426 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:42:29.146722 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:29.146689 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5b6z6" event={"ID":"ac6cc6e6-4369-4fbc-898b-13c449fe4c92","Type":"ContainerStarted","Data":"518477a8759ef831abd207649c3ea4c1592b65976c48738efc2ab6f08dc22588"} Jul 17 05:42:29.146900 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:29.146805 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:42:29.148771 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:29.148737 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" event={"ID":"5be596d5-da4f-4d3a-81a8-ecac9b05ffda","Type":"ContainerStarted","Data":"680fd245880721175bc16588f1abb51be25e1fdf99c1701a84d958247a56bdbf"} Jul 17 05:42:29.148899 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:29.148775 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" event={"ID":"5be596d5-da4f-4d3a-81a8-ecac9b05ffda","Type":"ContainerStarted","Data":"7da36bc89a8ba94fd190a655db0c26a240296aea204495ae1a5b388a0223e837"} Jul 17 05:42:29.148899 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:29.148791 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" event={"ID":"5be596d5-da4f-4d3a-81a8-ecac9b05ffda","Type":"ContainerStarted","Data":"e572f06919ac489244c1a56d4b155d019f48966d1cab0380de00ec1fa9d7f53e"} Jul 17 05:42:29.150236 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:29.150215 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t6rpv" event={"ID":"55c6188c-0a2e-4607-8c0e-7a5685859c4d","Type":"ContainerStarted","Data":"ab0a383ffee08936b2d8b8a1b30b9d7d43b2c44494243fbd4e82894001c5a860"} Jul 17 05:42:29.151434 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:29.151415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps" event={"ID":"0d2fe440-baae-411a-aef5-059f4e6f2cc8","Type":"ContainerStarted","Data":"8593d63f8fb077aa45791acb5d9a65b37b134b8b08b3315be81ec038b2d4c595"} Jul 17 05:42:29.160944 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:29.160899 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5b6z6" podStartSLOduration=67.332722262 podStartE2EDuration="1m13.16088454s" totalImagesPullingTime="5.828162278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:42:29.16061265 +0000 UTC m=+73.905291979" watchObservedRunningTime="2026-07-17 05:42:29.16088454 +0000 UTC m=+73.905563868" Jul 17 05:42:29.175851 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:29.175813 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" podStartSLOduration=1.7948203600000001 podStartE2EDuration="9.175801327s" totalImagesPullingTime="7.380980967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:42:29.175658298 +0000 UTC m=+73.920337626" watchObservedRunningTime="2026-07-17 05:42:29.175801327 +0000 UTC m=+73.920480655" Jul 17 05:42:29.190219 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:29.190185 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t6rpv" podStartSLOduration=66.224544969 podStartE2EDuration="1m12.190173148s" totalImagesPullingTime="5.965628179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:42:29.188522222 +0000 UTC m=+73.933201549" watchObservedRunningTime="2026-07-17 05:42:29.190173148 +0000 UTC m=+73.934852453" Jul 17 05:42:29.202992 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:29.202955 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps" podStartSLOduration=1.834497101 podStartE2EDuration="9.202943743s" totalImagesPullingTime="7.368446642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:42:29.202067515 +0000 UTC m=+73.946746852" watchObservedRunningTime="2026-07-17 05:42:29.202943743 +0000 UTC m=+73.947623070" Jul 17 05:42:52.003485 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:52.003343 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:42:52.055025 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:52.054993 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:42:54.450515 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:42:54.450472 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:42:54.450930 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:42:54.450632 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:42:54.450930 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:42:54.450703 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls podName:f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe nodeName:}" failed. No retries permitted until 2026-07-17 05:43:58.45068621 +0000 UTC m=+163.195365519 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls") pod "dns-default-nhv84" (UID: "f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe") : secret "dns-default-metrics-tls" not found Jul 17 05:43:00.159041 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:43:00.159009 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5b6z6" Jul 17 05:43:26.070301 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:43:26.070259 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:43:26.070739 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:43:26.070380 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 17 05:43:26.070739 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:43:26.070451 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs podName:3c472db7-71bf-46b2-9452-f936c8c53e67 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:28.070436683 +0000 UTC m=+252.815115989 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs") pod "network-metrics-daemon-nsrf4" (UID: "3c472db7-71bf-46b2-9452-f936c8c53e67") : secret "metrics-daemon-secret" not found Jul 17 05:43:53.601126 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:43:53.601083 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nhv84" podUID="f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe" Jul 17 05:43:53.896017 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:43:53.895980 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:43:54.398325 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:43:54.398290 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nhv84" Jul 17 05:43:58.498863 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:43:58.498810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:43:58.499295 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:43:58.498976 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:43:58.499295 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:43:58.499046 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls podName:f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe nodeName:}" failed. No retries permitted until 2026-07-17 05:46:00.499029118 +0000 UTC m=+285.243708423 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls") pod "dns-default-nhv84" (UID: "f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe") : secret "dns-default-metrics-tls" not found Jul 17 05:44:05.884500 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:44:05.884445 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:44:28.498076 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:44:28.498041 2580 generic.go:356] "Generic (PLEG): container finished" podID="0d2fe440-baae-411a-aef5-059f4e6f2cc8" containerID="8593d63f8fb077aa45791acb5d9a65b37b134b8b08b3315be81ec038b2d4c595" exitCode=255 Jul 17 05:44:28.498076 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:44:28.498080 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps" event={"ID":"0d2fe440-baae-411a-aef5-059f4e6f2cc8","Type":"ContainerDied","Data":"8593d63f8fb077aa45791acb5d9a65b37b134b8b08b3315be81ec038b2d4c595"} Jul 17 05:44:28.498534 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:44:28.498345 2580 scope.go:117] "RemoveContainer" containerID="8593d63f8fb077aa45791acb5d9a65b37b134b8b08b3315be81ec038b2d4c595" Jul 17 05:44:29.502726 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:44:29.502691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65cb699598-tm5ps" event={"ID":"0d2fe440-baae-411a-aef5-059f4e6f2cc8","Type":"ContainerStarted","Data":"9d5903dcb5766f5c1d5cd2979a7e901ec740483b4e24e580da57d143e30c27fc"} Jul 17 05:44:47.474051 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:44:47.474021 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:15.725105 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:15.725071 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:15.776936 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:15.776905 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:18.095175 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:18.095138 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:18.151617 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:18.151585 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:18.961079 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:18.961043 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vfk9k"] Jul 17 05:45:18.962901 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:18.962885 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:18.965722 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:18.965688 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 05:45:18.965870 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:18.965744 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 05:45:18.965870 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:18.965616 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 05:45:18.966154 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:18.965704 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 05:45:18.968284 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:18.967173 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8zw9n\"" Jul 17 05:45:18.973473 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:18.973439 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vfk9k"] Jul 17 05:45:18.982204 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:18.982181 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:19.032328 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.032300 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:19.058143 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.058113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-crio-socket\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.058238 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.058147 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.058238 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.058191 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-data-volume\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.058303 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.058235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.058303 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.058254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mr8b\" (UniqueName: \"kubernetes.io/projected/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-api-access-6mr8b\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.159118 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.159087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-crio-socket\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.159118 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.159122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.159585 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.159155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-data-volume\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.159585 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:19.159225 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 05:45:19.159585 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.159257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.159585 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.159285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mr8b\" (UniqueName: \"kubernetes.io/projected/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-api-access-6mr8b\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.159585 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.159222 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-crio-socket\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.159585 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:19.159320 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm podName:9d52e702-a8bf-44dd-887b-f16b64fa0e7a nodeName:}" failed. No retries permitted until 2026-07-17 05:45:19.659298575 +0000 UTC m=+244.403977923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vfk9k" (UID: "9d52e702-a8bf-44dd-887b-f16b64fa0e7a") : configmap "kube-rbac-proxy" not found Jul 17 05:45:19.159585 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.159458 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-data-volume\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.159585 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:19.159462 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 05:45:19.159585 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:19.159540 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls podName:9d52e702-a8bf-44dd-887b-f16b64fa0e7a nodeName:}" failed. No retries permitted until 2026-07-17 05:45:19.659522222 +0000 UTC m=+244.404201532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfk9k" (UID: "9d52e702-a8bf-44dd-887b-f16b64fa0e7a") : secret "insights-runtime-extractor-tls" not found Jul 17 05:45:19.167060 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.167031 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mr8b\" (UniqueName: \"kubernetes.io/projected/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-api-access-6mr8b\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.662355 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.662310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.662600 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:19.662427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:19.662600 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:19.662457 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 05:45:19.662600 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:19.662504 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 05:45:19.662600 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:19.662535 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls podName:9d52e702-a8bf-44dd-887b-f16b64fa0e7a nodeName:}" failed. No retries permitted until 2026-07-17 05:45:20.662517379 +0000 UTC m=+245.407196701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfk9k" (UID: "9d52e702-a8bf-44dd-887b-f16b64fa0e7a") : secret "insights-runtime-extractor-tls" not found Jul 17 05:45:19.662600 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:19.662554 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm podName:9d52e702-a8bf-44dd-887b-f16b64fa0e7a nodeName:}" failed. No retries permitted until 2026-07-17 05:45:20.662547099 +0000 UTC m=+245.407226405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vfk9k" (UID: "9d52e702-a8bf-44dd-887b-f16b64fa0e7a") : configmap "kube-rbac-proxy" not found Jul 17 05:45:20.179107 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:20.179076 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:20.620780 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:20.620746 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:20.671622 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:20.671588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:20.671810 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:20.671641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:20.671810 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:20.671727 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 05:45:20.671810 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:20.671743 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 05:45:20.671810 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:20.671792 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls podName:9d52e702-a8bf-44dd-887b-f16b64fa0e7a nodeName:}" failed. No retries permitted until 2026-07-17 05:45:22.671777873 +0000 UTC m=+247.416457179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfk9k" (UID: "9d52e702-a8bf-44dd-887b-f16b64fa0e7a") : secret "insights-runtime-extractor-tls" not found Jul 17 05:45:20.671810 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:20.671803 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm podName:9d52e702-a8bf-44dd-887b-f16b64fa0e7a nodeName:}" failed. No retries permitted until 2026-07-17 05:45:22.671797814 +0000 UTC m=+247.416477119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vfk9k" (UID: "9d52e702-a8bf-44dd-887b-f16b64fa0e7a") : configmap "kube-rbac-proxy" not found Jul 17 05:45:20.720039 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:20.720006 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:20.773318 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:20.773285 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:20.886145 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:20.886047 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:20.986860 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:20.986823 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:21.027270 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:21.027250 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gw9qs_ca6d89cf-4331-4c48-91b6-5e6b570d8927/dns-node-resolver/0.log" Jul 17 05:45:22.227209 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.227182 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xqr2c_e3a2951e-e5a2-49fc-a938-a5617748d2bb/node-ca/0.log" Jul 17 05:45:22.481926 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.481850 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-khvwb"] Jul 17 05:45:22.484240 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.484223 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-khvwb" Jul 17 05:45:22.486676 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.486640 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 17 05:45:22.486676 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.486663 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 17 05:45:22.486844 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.486682 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 17 05:45:22.486844 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.486640 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 17 05:45:22.487337 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.487323 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lndxf\"" Jul 17 05:45:22.493677 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.493658 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-khvwb"] Jul 17 05:45:22.530809 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.530779 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:22.586878 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.586844 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5070856-15d5-4b1b-beb5-6d3a1a15eeee-signing-key\") pod \"service-ca-58598695f7-khvwb\" (UID: \"b5070856-15d5-4b1b-beb5-6d3a1a15eeee\") " pod="openshift-service-ca/service-ca-58598695f7-khvwb" Jul 17 05:45:22.587003 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.586891 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5070856-15d5-4b1b-beb5-6d3a1a15eeee-signing-cabundle\") pod \"service-ca-58598695f7-khvwb\" (UID: \"b5070856-15d5-4b1b-beb5-6d3a1a15eeee\") " pod="openshift-service-ca/service-ca-58598695f7-khvwb" Jul 17 05:45:22.587003 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.586982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xp2h\" (UniqueName: \"kubernetes.io/projected/b5070856-15d5-4b1b-beb5-6d3a1a15eeee-kube-api-access-7xp2h\") pod \"service-ca-58598695f7-khvwb\" (UID: \"b5070856-15d5-4b1b-beb5-6d3a1a15eeee\") " pod="openshift-service-ca/service-ca-58598695f7-khvwb" Jul 17 05:45:22.627138 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.627110 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dsrvg_3fde696a-f615-4e9f-b866-84073d4a923b/serve-healthcheck-canary/0.log" Jul 17 05:45:22.687708 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.687663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5070856-15d5-4b1b-beb5-6d3a1a15eeee-signing-key\") pod \"service-ca-58598695f7-khvwb\" (UID: \"b5070856-15d5-4b1b-beb5-6d3a1a15eeee\") " pod="openshift-service-ca/service-ca-58598695f7-khvwb" Jul 17 05:45:22.687893 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.687731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5070856-15d5-4b1b-beb5-6d3a1a15eeee-signing-cabundle\") pod \"service-ca-58598695f7-khvwb\" (UID: \"b5070856-15d5-4b1b-beb5-6d3a1a15eeee\") " pod="openshift-service-ca/service-ca-58598695f7-khvwb" Jul 17 05:45:22.687893 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.687768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xp2h\" (UniqueName: \"kubernetes.io/projected/b5070856-15d5-4b1b-beb5-6d3a1a15eeee-kube-api-access-7xp2h\") pod \"service-ca-58598695f7-khvwb\" (UID: \"b5070856-15d5-4b1b-beb5-6d3a1a15eeee\") " pod="openshift-service-ca/service-ca-58598695f7-khvwb" Jul 17 05:45:22.687893 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.687801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:22.687893 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.687860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:22.688114 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:22.687945 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 05:45:22.688114 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:22.687960 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 05:45:22.688114 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:22.688006 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls podName:9d52e702-a8bf-44dd-887b-f16b64fa0e7a nodeName:}" failed. No retries permitted until 2026-07-17 05:45:26.687986105 +0000 UTC m=+251.432665430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfk9k" (UID: "9d52e702-a8bf-44dd-887b-f16b64fa0e7a") : secret "insights-runtime-extractor-tls" not found Jul 17 05:45:22.688114 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:22.688024 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm podName:9d52e702-a8bf-44dd-887b-f16b64fa0e7a nodeName:}" failed. No retries permitted until 2026-07-17 05:45:26.688015163 +0000 UTC m=+251.432694470 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vfk9k" (UID: "9d52e702-a8bf-44dd-887b-f16b64fa0e7a") : configmap "kube-rbac-proxy" not found Jul 17 05:45:22.688571 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.688550 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5070856-15d5-4b1b-beb5-6d3a1a15eeee-signing-cabundle\") pod \"service-ca-58598695f7-khvwb\" (UID: \"b5070856-15d5-4b1b-beb5-6d3a1a15eeee\") " pod="openshift-service-ca/service-ca-58598695f7-khvwb" Jul 17 05:45:22.690253 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.690230 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5070856-15d5-4b1b-beb5-6d3a1a15eeee-signing-key\") pod \"service-ca-58598695f7-khvwb\" (UID: \"b5070856-15d5-4b1b-beb5-6d3a1a15eeee\") " pod="openshift-service-ca/service-ca-58598695f7-khvwb" Jul 17 05:45:22.695346 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.695327 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xp2h\" (UniqueName: \"kubernetes.io/projected/b5070856-15d5-4b1b-beb5-6d3a1a15eeee-kube-api-access-7xp2h\") pod \"service-ca-58598695f7-khvwb\" (UID: \"b5070856-15d5-4b1b-beb5-6d3a1a15eeee\") " pod="openshift-service-ca/service-ca-58598695f7-khvwb" Jul 17 05:45:22.794197 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.794111 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-khvwb" Jul 17 05:45:22.855813 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.855771 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:22.911496 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.911467 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-khvwb"] Jul 17 05:45:22.914438 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:22.914405 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:22.914758 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:45:22.914739 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5070856_15d5_4b1b_beb5_6d3a1a15eeee.slice/crio-b59cb9afe159b270bc2350f2be89cbdb295d5bb2bb617e1482b6ab7ce546a15c WatchSource:0}: Error finding container b59cb9afe159b270bc2350f2be89cbdb295d5bb2bb617e1482b6ab7ce546a15c: Status 404 returned error can't find the container with id b59cb9afe159b270bc2350f2be89cbdb295d5bb2bb617e1482b6ab7ce546a15c Jul 17 05:45:23.394590 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:23.394554 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:23.652999 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:23.652920 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-khvwb" event={"ID":"b5070856-15d5-4b1b-beb5-6d3a1a15eeee","Type":"ContainerStarted","Data":"b59cb9afe159b270bc2350f2be89cbdb295d5bb2bb617e1482b6ab7ce546a15c"} Jul 17 05:45:25.662366 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:25.662323 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-khvwb" event={"ID":"b5070856-15d5-4b1b-beb5-6d3a1a15eeee","Type":"ContainerStarted","Data":"7f506c611ac29682ff09e6647dc754c163577ea925571e00fce74af93f3db749"} Jul 17 05:45:25.680136 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:25.680081 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-khvwb" podStartSLOduration=1.4738766509999999 podStartE2EDuration="3.680063265s" totalImagesPullingTime="2.206186614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:45:25.679896614 +0000 UTC m=+250.424575955" watchObservedRunningTime="2026-07-17 05:45:25.680063265 +0000 UTC m=+250.424742573" Jul 17 05:45:26.120003 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:26.119965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:26.457206 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:26.457106 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:26.575745 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:26.575687 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:26.721203 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:26.721095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:26.721203 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:26.721189 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:26.721753 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:26.721302 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 05:45:26.721753 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:26.721363 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm podName:9d52e702-a8bf-44dd-887b-f16b64fa0e7a nodeName:}" failed. No retries permitted until 2026-07-17 05:45:34.721345852 +0000 UTC m=+259.466025163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vfk9k" (UID: "9d52e702-a8bf-44dd-887b-f16b64fa0e7a") : configmap "kube-rbac-proxy" not found Jul 17 05:45:26.721871 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:26.721797 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 05:45:26.721871 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:26.721841 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls podName:9d52e702-a8bf-44dd-887b-f16b64fa0e7a nodeName:}" failed. No retries permitted until 2026-07-17 05:45:34.721826141 +0000 UTC m=+259.466505451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfk9k" (UID: "9d52e702-a8bf-44dd-887b-f16b64fa0e7a") : secret "insights-runtime-extractor-tls" not found Jul 17 05:45:26.740418 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:26.740373 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:26.870958 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:26.870928 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:26.977588 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:26.977496 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:27.137174 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:27.137141 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:27.348509 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:27.348421 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:27.558203 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:27.558173 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:28.134319 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:28.134264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:45:28.134744 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:28.134430 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 17 05:45:28.134744 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:28.134493 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs podName:3c472db7-71bf-46b2-9452-f936c8c53e67 nodeName:}" failed. No retries permitted until 2026-07-17 05:47:30.134476239 +0000 UTC m=+374.879155560 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs") pod "network-metrics-daemon-nsrf4" (UID: "3c472db7-71bf-46b2-9452-f936c8c53e67") : secret "metrics-daemon-secret" not found Jul 17 05:45:28.730749 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:28.730714 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:29.111527 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:29.111486 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:29.226541 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:29.226499 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:29.407642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:29.407553 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:29.525647 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:29.525605 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:30.029416 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:30.029371 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:30.324279 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:30.324197 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:30.529467 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:30.529434 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:30.931011 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:30.930975 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:31.130375 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:31.130342 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:31.331590 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:31.331498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:31.730396 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:31.730363 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:32.242101 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:32.242069 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:32.345216 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:32.345183 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:32.725345 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:32.725308 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:32.961719 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:32.961668 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:33.017764 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:33.017686 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:33.567441 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:33.567401 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:33.927923 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:33.927877 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:34.332358 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:34.332267 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:34.434040 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:34.434007 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:34.533091 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:34.533054 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:34.784488 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:34.784442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:34.784949 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:34.784514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:34.784949 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:34.784617 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 05:45:34.784949 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:34.784683 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls podName:9d52e702-a8bf-44dd-887b-f16b64fa0e7a nodeName:}" failed. No retries permitted until 2026-07-17 05:45:50.784666368 +0000 UTC m=+275.529345676 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vfk9k" (UID: "9d52e702-a8bf-44dd-887b-f16b64fa0e7a") : secret "insights-runtime-extractor-tls" not found Jul 17 05:45:34.785167 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:34.785142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:34.925843 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:34.925812 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:35.530399 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:35.530363 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:35.926310 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:35.926277 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:36.129428 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:36.129376 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:36.729019 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:36.728985 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:36.933251 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:36.933219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:37.132963 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:37.132866 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:39.118942 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:39.118892 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:39.219927 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:39.219887 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:41.700938 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:41.700907 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:41.800612 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:41.800582 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:41.928630 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:41.928602 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:42.028009 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:42.027930 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:42.108802 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:42.108770 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:42.212152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:42.212117 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:42.590075 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:42.590043 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:42.642699 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:42.642668 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:42.981328 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:42.981299 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:43.037535 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:43.037507 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:43.809115 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:43.809080 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:44.008012 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:44.007978 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:44.109125 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:44.109085 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:44.445251 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:44.445166 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:44.499835 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:44.499806 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:49.946723 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:49.946692 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:50.044883 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:50.044851 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:50.393181 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:50.393145 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:50.493423 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:50.493380 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:50.564091 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:50.564058 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:50.662107 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:50.662024 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:50.795614 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:50.795562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:50.798339 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:50.798296 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d52e702-a8bf-44dd-887b-f16b64fa0e7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vfk9k\" (UID: \"9d52e702-a8bf-44dd-887b-f16b64fa0e7a\") " pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:51.075664 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:51.075570 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vfk9k" Jul 17 05:45:51.198372 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:51.198338 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vfk9k"] Jul 17 05:45:51.201456 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:45:51.201427 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d52e702_a8bf_44dd_887b_f16b64fa0e7a.slice/crio-0dfef5463c414dbe82800cda7e3bf92a33c202772136b051b2b35a7f66879927 WatchSource:0}: Error finding container 0dfef5463c414dbe82800cda7e3bf92a33c202772136b051b2b35a7f66879927: Status 404 returned error can't find the container with id 0dfef5463c414dbe82800cda7e3bf92a33c202772136b051b2b35a7f66879927 Jul 17 05:45:51.745225 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:51.745041 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfk9k" event={"ID":"9d52e702-a8bf-44dd-887b-f16b64fa0e7a","Type":"ContainerStarted","Data":"0dfef5463c414dbe82800cda7e3bf92a33c202772136b051b2b35a7f66879927"} Jul 17 05:45:53.589101 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:53.589052 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:53.654660 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:53.654614 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:53.845951 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:53.845866 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:55.214876 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:55.214843 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:56.765026 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:56.764956 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfk9k" event={"ID":"9d52e702-a8bf-44dd-887b-f16b64fa0e7a","Type":"ContainerStarted","Data":"01f6ff606840f36961a986c0ba18368a6370c94b85b180bcd8a3551b48045fb3"} Jul 17 05:45:57.398853 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:45:57.398810 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nhv84" podUID="f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe" Jul 17 05:45:57.768445 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:57.768360 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nhv84" Jul 17 05:45:57.816162 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:57.816116 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:57.870016 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:57.869982 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:45:59.777083 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:45:59.777047 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfk9k" event={"ID":"9d52e702-a8bf-44dd-887b-f16b64fa0e7a","Type":"ContainerStarted","Data":"cbd6146adef089499c8f8fc99561757fdb1698838ea9b154d879988e4e827c6a"} Jul 17 05:46:00.576663 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:00.576626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:46:00.579296 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:00.579252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe-metrics-tls\") pod \"dns-default-nhv84\" (UID: \"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe\") " pod="openshift-dns/dns-default-nhv84" Jul 17 05:46:00.771974 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:00.771940 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jx7lr\"" Jul 17 05:46:00.779802 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:00.779770 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nhv84" Jul 17 05:46:00.927290 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:00.927252 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nhv84"] Jul 17 05:46:00.931159 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:46:00.931126 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83f0f4a_a71a_4ce1_a4f6_8ce7ce17f3fe.slice/crio-3785e4b4ad32a482c3901a0497ecbe86734407330027850d4d38618f8fd2359d WatchSource:0}: Error finding container 3785e4b4ad32a482c3901a0497ecbe86734407330027850d4d38618f8fd2359d: Status 404 returned error can't find the container with id 3785e4b4ad32a482c3901a0497ecbe86734407330027850d4d38618f8fd2359d Jul 17 05:46:01.734038 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:01.733998 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:01.785314 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:01.785271 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nhv84" event={"ID":"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe","Type":"ContainerStarted","Data":"3785e4b4ad32a482c3901a0497ecbe86734407330027850d4d38618f8fd2359d"} Jul 17 05:46:03.744024 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:03.743992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:03.794950 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:03.794912 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vfk9k" event={"ID":"9d52e702-a8bf-44dd-887b-f16b64fa0e7a","Type":"ContainerStarted","Data":"52a629a34bbca195b2d35ce6de0cb922656da1ec4e4f7e815205738e113ca68b"} Jul 17 05:46:03.810268 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:03.810212 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vfk9k" podStartSLOduration=34.202115009 podStartE2EDuration="45.81019269s" totalImagesPullingTime="11.608077681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:45:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:03.809981605 +0000 UTC m=+288.554660964" watchObservedRunningTime="2026-07-17 05:46:03.81019269 +0000 UTC m=+288.554872001" Jul 17 05:46:03.949120 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:03.949082 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:03.982595 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:03.982539 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g"] Jul 17 05:46:03.987475 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:03.987445 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:03.989738 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:03.989690 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 17 05:46:03.990366 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:03.990334 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 05:46:03.990520 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:03.990441 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kkrzq\"" Jul 17 05:46:03.990792 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:03.990754 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 05:46:03.990792 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:03.990779 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 17 05:46:03.990946 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:03.990855 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 05:46:03.998251 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:03.998211 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g"] Jul 17 05:46:04.002974 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.001701 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q9fj7"] Jul 17 05:46:04.005886 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.005798 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.008140 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.008119 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 05:46:04.008631 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.008322 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 05:46:04.008631 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.008470 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dzjkz\"" Jul 17 05:46:04.008631 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.008573 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 05:46:04.031530 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.031498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:04.092904 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.092873 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:04.105909 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.105877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-hlp5g\" (UID: \"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:04.106100 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.105917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-root\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.106100 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.105946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-hlp5g\" (UID: \"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:04.106100 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.105969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-sys\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.106100 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.106064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-metrics-client-ca\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.106331 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.106108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-accelerators-collector-config\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.106331 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.106140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-textfile\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.106331 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.106201 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-wtmp\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.106331 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.106231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-tls\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.106331 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.106260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-hlp5g\" (UID: \"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:04.106331 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.106286 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrhsj\" (UniqueName: \"kubernetes.io/projected/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-kube-api-access-jrhsj\") pod \"openshift-state-metrics-79f6958cdf-hlp5g\" (UID: \"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:04.106586 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.106337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.106586 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.106372 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prtt\" (UniqueName: \"kubernetes.io/projected/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-kube-api-access-7prtt\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.206798 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.206724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.206798 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.206754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prtt\" (UniqueName: \"kubernetes.io/projected/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-kube-api-access-7prtt\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.206798 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.206779 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-hlp5g\" (UID: \"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:04.206990 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:46:04.206865 2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 17 05:46:04.206990 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:46:04.206918 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-openshift-state-metrics-tls podName:f8d44361-f45e-42bb-bdb6-d4c6a071f8d6 nodeName:}" failed. No retries permitted until 2026-07-17 05:46:04.706900173 +0000 UTC m=+289.451579495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-hlp5g" (UID: "f8d44361-f45e-42bb-bdb6-d4c6a071f8d6") : secret "openshift-state-metrics-tls" not found Jul 17 05:46:04.206990 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.206945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-root\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.206990 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.206972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-hlp5g\" (UID: \"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:04.207202 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.206996 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-sys\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.207202 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.207040 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-metrics-client-ca\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.207202 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.207067 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-root\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.207202 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.207080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-accelerators-collector-config\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.207202 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.207111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-textfile\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.207202 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.207117 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-sys\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.207585 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.207444 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-textfile\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.207585 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.207524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-wtmp\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.207585 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.207564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-tls\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.207761 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.207597 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-hlp5g\" (UID: \"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:04.207761 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.207677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrhsj\" (UniqueName: \"kubernetes.io/projected/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-kube-api-access-jrhsj\") pod \"openshift-state-metrics-79f6958cdf-hlp5g\" (UID: \"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:04.207863 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.207770 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-accelerators-collector-config\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.207863 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:46:04.207849 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 17 05:46:04.207958 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:46:04.207887 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-tls podName:377bf2c4-8523-4d6d-a342-ffe1ec22daf2 nodeName:}" failed. No retries permitted until 2026-07-17 05:46:04.707874349 +0000 UTC m=+289.452553669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-tls") pod "node-exporter-q9fj7" (UID: "377bf2c4-8523-4d6d-a342-ffe1ec22daf2") : secret "node-exporter-tls" not found Jul 17 05:46:04.207958 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.207910 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-metrics-client-ca\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.208066 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.207956 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-hlp5g\" (UID: \"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:04.208066 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.207995 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-wtmp\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.210364 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.210314 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-hlp5g\" (UID: \"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:04.210945 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.210919 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.220663 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.220292 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prtt\" (UniqueName: \"kubernetes.io/projected/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-kube-api-access-7prtt\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.221191 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.221151 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrhsj\" (UniqueName: \"kubernetes.io/projected/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-kube-api-access-jrhsj\") pod \"openshift-state-metrics-79f6958cdf-hlp5g\" (UID: \"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:04.242869 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.242808 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:04.339327 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.339295 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:04.440576 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.440543 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:04.541757 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.541682 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:04.685905 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.685858 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:04.711077 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.711037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-tls\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.711234 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.711105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-hlp5g\" (UID: \"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:04.713633 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.713599 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/377bf2c4-8523-4d6d-a342-ffe1ec22daf2-node-exporter-tls\") pod \"node-exporter-q9fj7\" (UID: \"377bf2c4-8523-4d6d-a342-ffe1ec22daf2\") " pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.713762 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.713657 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8d44361-f45e-42bb-bdb6-d4c6a071f8d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-hlp5g\" (UID: \"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:04.800504 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.800403 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nhv84" event={"ID":"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe","Type":"ContainerStarted","Data":"aee439735e6d0a254e39eb3e7a06a984794e8390c64322a6e1b842b9a1ad2e51"} Jul 17 05:46:04.800923 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.800732 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nhv84" Jul 17 05:46:04.800923 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.800776 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nhv84" event={"ID":"f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe","Type":"ContainerStarted","Data":"e384f4b2673339f585f7db42e1d53a1e39af0502b9224e13bcb463809d62eed2"} Jul 17 05:46:04.815313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.815263 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nhv84" podStartSLOduration=251.625428409 podStartE2EDuration="4m14.815246911s" totalImagesPullingTime="3.189818502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:04.814274315 +0000 UTC m=+289.558953659" watchObservedRunningTime="2026-07-17 05:46:04.815246911 +0000 UTC m=+289.559926238" Jul 17 05:46:04.900724 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.900691 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" Jul 17 05:46:04.922024 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.921993 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q9fj7" Jul 17 05:46:04.934852 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:46:04.934806 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377bf2c4_8523_4d6d_a342_ffe1ec22daf2.slice/crio-b6d9cf5aed33ea62501208c1ee6413ae97089ad36d612f60f3f87121e6c93d2f WatchSource:0}: Error finding container b6d9cf5aed33ea62501208c1ee6413ae97089ad36d612f60f3f87121e6c93d2f: Status 404 returned error can't find the container with id b6d9cf5aed33ea62501208c1ee6413ae97089ad36d612f60f3f87121e6c93d2f Jul 17 05:46:04.961415 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:04.961370 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:05.034172 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.034136 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g"] Jul 17 05:46:05.037163 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:46:05.037139 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d44361_f45e_42bb_bdb6_d4c6a071f8d6.slice/crio-814cca275ac708105fe857f4b62542e60a071358971169129690fabc269bc389 WatchSource:0}: Error finding container 814cca275ac708105fe857f4b62542e60a071358971169129690fabc269bc389: Status 404 returned error can't find the container with id 814cca275ac708105fe857f4b62542e60a071358971169129690fabc269bc389 Jul 17 05:46:05.055745 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.055715 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 05:46:05.060107 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.060090 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.062526 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.062500 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 05:46:05.062630 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.062542 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 05:46:05.062630 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.062607 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 05:46:05.062745 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.062500 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 05:46:05.062882 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.062861 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:05.062882 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.062874 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 05:46:05.062995 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.062887 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 05:46:05.062995 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.062978 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 05:46:05.063083 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.062979 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 05:46:05.063122 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.063082 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 05:46:05.063122 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.063110 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rdqft\"" Jul 17 05:46:05.076572 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.076549 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 05:46:05.115203 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.115178 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-config-volume\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.115321 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.115207 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.115321 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.115225 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.115321 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.115251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lbvw\" (UniqueName: \"kubernetes.io/projected/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-kube-api-access-4lbvw\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.115506 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.115352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-web-config\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.115506 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.115406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-config-out\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.115506 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.115448 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.115506 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.115485 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.115700 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.115514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.115700 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.115559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.115700 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.115589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.115700 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.115615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.115700 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.115648 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.169867 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.169839 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:05.216497 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.216465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-web-config\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.216497 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.216500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-config-out\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.216740 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.216523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.216740 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.216553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.216740 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.216582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.216740 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.216619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.216740 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.216646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.216740 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.216671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.216740 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.216696 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.217050 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.216753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-config-volume\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.217050 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.216771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.217050 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.216791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.217050 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.216820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lbvw\" (UniqueName: \"kubernetes.io/projected/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-kube-api-access-4lbvw\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.217050 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.216984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.217416 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:46:05.217134 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-trusted-ca-bundle podName:2a849f40-bde6-40b9-b2f4-b6441ec84e0e nodeName:}" failed. No retries permitted until 2026-07-17 05:46:05.717110684 +0000 UTC m=+290.461789990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "2a849f40-bde6-40b9-b2f4-b6441ec84e0e") : configmap references non-existent config key: ca-bundle.crt Jul 17 05:46:05.218209 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.218160 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.219656 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.219586 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-config-out\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.219771 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.219729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.219871 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.219847 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.220309 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.220283 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.220428 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.220328 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-web-config\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.220494 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.220476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-config-volume\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.220635 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.220618 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.220712 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.220670 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.221049 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.221032 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.225193 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.225170 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lbvw\" (UniqueName: \"kubernetes.io/projected/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-kube-api-access-4lbvw\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.720510 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.720463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.721555 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.721529 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:05.807824 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.807778 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" event={"ID":"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6","Type":"ContainerStarted","Data":"c723cac725ff1653ece4da90867cd98bb5df4774dc9a02f49e68b8d3109fb4f0"} Jul 17 05:46:05.808279 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.807864 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" event={"ID":"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6","Type":"ContainerStarted","Data":"673e97e8609fcb744accd959138cfb0940bd9dad9dab87b2ce9af5f0f3b42fcf"} Jul 17 05:46:05.808279 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.807897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" event={"ID":"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6","Type":"ContainerStarted","Data":"814cca275ac708105fe857f4b62542e60a071358971169129690fabc269bc389"} Jul 17 05:46:05.809422 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.809372 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q9fj7" event={"ID":"377bf2c4-8523-4d6d-a342-ffe1ec22daf2","Type":"ContainerStarted","Data":"b6d9cf5aed33ea62501208c1ee6413ae97089ad36d612f60f3f87121e6c93d2f"} Jul 17 05:46:05.973614 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:05.973594 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:46:06.124463 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:06.124431 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 05:46:06.290706 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:46:06.290629 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a849f40_bde6_40b9_b2f4_b6441ec84e0e.slice/crio-99fc9fa1cb7fedcc166baf187e2c9ffe12c606379d80f7b8629e861257c329f8 WatchSource:0}: Error finding container 99fc9fa1cb7fedcc166baf187e2c9ffe12c606379d80f7b8629e861257c329f8: Status 404 returned error can't find the container with id 99fc9fa1cb7fedcc166baf187e2c9ffe12c606379d80f7b8629e861257c329f8 Jul 17 05:46:06.816162 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:06.816118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" event={"ID":"f8d44361-f45e-42bb-bdb6-d4c6a071f8d6","Type":"ContainerStarted","Data":"fc380fa8962b53c9e58d5278b3a9fb8c93d681ce5842119626b39408845a2e5f"} Jul 17 05:46:06.817318 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:06.817297 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerStarted","Data":"99fc9fa1cb7fedcc166baf187e2c9ffe12c606379d80f7b8629e861257c329f8"} Jul 17 05:46:06.818733 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:06.818707 2580 generic.go:356] "Generic (PLEG): container finished" podID="377bf2c4-8523-4d6d-a342-ffe1ec22daf2" containerID="90f73e59a711281f50f97f7db84e049751447aedb5a4c6fa5081020c5fe7198c" exitCode=0 Jul 17 05:46:06.818849 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:06.818791 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q9fj7" event={"ID":"377bf2c4-8523-4d6d-a342-ffe1ec22daf2","Type":"ContainerDied","Data":"90f73e59a711281f50f97f7db84e049751447aedb5a4c6fa5081020c5fe7198c"} Jul 17 05:46:06.833421 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:06.833347 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-hlp5g" podStartSLOduration=2.701476578 podStartE2EDuration="3.833326637s" totalImagesPullingTime="1.131850059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:06.832845985 +0000 UTC m=+291.577525333" watchObservedRunningTime="2026-07-17 05:46:06.833326637 +0000 UTC m=+291.578005966" Jul 17 05:46:06.919961 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:06.919869 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:06.994711 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:06.994676 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:07.056890 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:07.056856 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:07.204024 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:07.203988 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:07.824141 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:07.824054 2580 generic.go:356] "Generic (PLEG): container finished" podID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerID="2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5" exitCode=0 Jul 17 05:46:07.824661 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:07.824134 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerDied","Data":"2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5"} Jul 17 05:46:07.826323 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:07.826299 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q9fj7" event={"ID":"377bf2c4-8523-4d6d-a342-ffe1ec22daf2","Type":"ContainerStarted","Data":"9436714e46198d88a09a52cc26900f4768cb77a4f9553f3dd703e4ea1dd63e93"} Jul 17 05:46:07.826422 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:07.826345 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q9fj7" event={"ID":"377bf2c4-8523-4d6d-a342-ffe1ec22daf2","Type":"ContainerStarted","Data":"ccf9649c51d446a2ff96e3861cda7ebe6a5dda69415223fa96b458fa7617fdbb"} Jul 17 05:46:07.864733 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:07.864678 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q9fj7" podStartSLOduration=3.8168754529999998 podStartE2EDuration="4.864660775s" totalImagesPullingTime="969.010326ms" totalInitContainerRuntime="78.774996ms" isStatefulPod=true podCreationTimestamp="2026-07-17 05:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:07.863934442 +0000 UTC m=+292.608613771" watchObservedRunningTime="2026-07-17 05:46:07.864660775 +0000 UTC m=+292.609340102" Jul 17 05:46:08.768224 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:08.768190 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:08.855687 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:08.855645 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:08.885931 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:46:08.885835 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nsrf4" podUID="3c472db7-71bf-46b2-9452-f936c8c53e67" Jul 17 05:46:09.081129 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:09.081037 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:09.837526 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:09.837490 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerStarted","Data":"32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1"} Jul 17 05:46:09.837692 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:09.837586 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerStarted","Data":"c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d"} Jul 17 05:46:09.837692 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:09.837613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerStarted","Data":"edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28"} Jul 17 05:46:09.837692 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:09.837636 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerStarted","Data":"6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012"} Jul 17 05:46:09.837692 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:09.837659 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerStarted","Data":"dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb"} Jul 17 05:46:10.242701 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.242670 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:10.244344 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.244320 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 05:46:10.246729 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.246706 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.249401 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.249366 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 05:46:10.249522 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.249374 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 05:46:10.249522 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.249461 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 05:46:10.249522 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.249374 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 05:46:10.249522 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.249500 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 05:46:10.250017 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.250002 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 05:46:10.250094 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.250018 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3svbn0ehar4bm\"" Jul 17 05:46:10.250094 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.250033 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 05:46:10.250188 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.250018 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 05:46:10.250380 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.250365 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 05:46:10.250470 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.250416 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 05:46:10.250470 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.250462 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 05:46:10.250572 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.250471 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 05:46:10.250572 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.250487 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 05:46:10.250572 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.250504 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mlkbz\"" Jul 17 05:46:10.250572 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.250557 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 05:46:10.253525 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.253505 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 05:46:10.264396 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.264359 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 05:46:10.361600 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.361566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bdfcf2fb-f8d5-4088-9833-df8c31c77385-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.361785 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.361612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.361785 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.361643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.361785 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.361739 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.361785 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.361781 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-config\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362005 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.361810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362005 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.361840 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362005 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.361867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362005 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.361893 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2792j\" (UniqueName: \"kubernetes.io/projected/bdfcf2fb-f8d5-4088-9833-df8c31c77385-kube-api-access-2792j\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362005 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.361933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bdfcf2fb-f8d5-4088-9833-df8c31c77385-config-out\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362005 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.361978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362005 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.362004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362272 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.362030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362272 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.362075 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362272 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.362096 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362272 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.362136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362272 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.362163 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-web-config\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362272 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.362225 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362272 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.362267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.362574 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.362299 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.463572 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.463572 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.463842 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.463842 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-config\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.463842 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.463842 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.463842 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.463842 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2792j\" (UniqueName: \"kubernetes.io/projected/bdfcf2fb-f8d5-4088-9833-df8c31c77385-kube-api-access-2792j\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.463842 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bdfcf2fb-f8d5-4088-9833-df8c31c77385-config-out\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.463842 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.463842 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.464337 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463866 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.464337 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.464337 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.464337 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.464337 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.463991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-web-config\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.464337 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.464024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.464337 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.464025 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.464337 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.464059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.464337 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.464097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.464337 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.464131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bdfcf2fb-f8d5-4088-9833-df8c31c77385-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.464864 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.464680 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.464864 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.464720 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.464864 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.464823 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.467108 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.466801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.467108 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.467033 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.467108 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.467050 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bdfcf2fb-f8d5-4088-9833-df8c31c77385-config-out\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.467108 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.467087 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-config\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.467458 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.467434 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.467661 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.467634 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.467760 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.467725 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-web-config\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.467892 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.467852 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.468193 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.468168 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.468266 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.468237 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.468612 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.468587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bdfcf2fb-f8d5-4088-9833-df8c31c77385-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.469713 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.469689 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.469812 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.469796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.469879 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.469852 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.470330 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.470307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.471741 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.471708 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2792j\" (UniqueName: \"kubernetes.io/projected/bdfcf2fb-f8d5-4088-9833-df8c31c77385-kube-api-access-2792j\") pod \"prometheus-k8s-0\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.557079 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.556819 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:10.821525 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.821431 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 05:46:10.824955 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:46:10.824924 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdfcf2fb_f8d5_4088_9833_df8c31c77385.slice/crio-00b35dc5ed60600bccb85b5cb88f012252f7894bc9b557682992f826668de423 WatchSource:0}: Error finding container 00b35dc5ed60600bccb85b5cb88f012252f7894bc9b557682992f826668de423: Status 404 returned error can't find the container with id 00b35dc5ed60600bccb85b5cb88f012252f7894bc9b557682992f826668de423 Jul 17 05:46:10.845696 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.845660 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerStarted","Data":"c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628"} Jul 17 05:46:10.847633 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.847608 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerStarted","Data":"00b35dc5ed60600bccb85b5cb88f012252f7894bc9b557682992f826668de423"} Jul 17 05:46:10.868974 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:10.868927 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.149383694 podStartE2EDuration="5.868910964s" totalImagesPullingTime="3.703866817s" totalInitContainerRuntime="15.660453ms" isStatefulPod=false podCreationTimestamp="2026-07-17 05:46:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:10.867927795 +0000 UTC m=+295.612607125" watchObservedRunningTime="2026-07-17 05:46:10.868910964 +0000 UTC m=+295.613590327" Jul 17 05:46:11.227932 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:11.227900 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:11.284301 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:11.284265 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:11.853003 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:11.852972 2580 generic.go:356] "Generic (PLEG): container finished" podID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerID="33756dd6d61e120b1b88190b9a939e4fd47963fa7a47b6f2afdd3eba702c9e64" exitCode=0 Jul 17 05:46:11.853182 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:11.853066 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerDied","Data":"33756dd6d61e120b1b88190b9a939e4fd47963fa7a47b6f2afdd3eba702c9e64"} Jul 17 05:46:15.822164 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:15.822123 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nhv84" Jul 17 05:46:15.873372 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:15.873329 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerStarted","Data":"1e94df5034262ce6b8776dc6f009b8f77242c03c7e80a305f0d9fcbb1ea2c183"} Jul 17 05:46:15.873545 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:15.873462 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerStarted","Data":"a7f9c9bacfb4c047b4fa13f703fc62aa8cf2885b6eecc52b26e776ee18315799"} Jul 17 05:46:16.269410 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:16.269369 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 05:46:17.883595 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:17.883560 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerStarted","Data":"c3bd07546f73a01e3213c4392d9b7b1231dcbf0415b784baf930bc0a17ccd3d0"} Jul 17 05:46:17.884054 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:17.883625 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerStarted","Data":"67a42667cf88cbc2544aa4a1c32e38ba699c32eef60a896426058d0ea555e852"} Jul 17 05:46:17.884054 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:17.883651 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerStarted","Data":"af12e00aee5a9ddbff9f396a08d31ff9d07353924152b309d50c4c47182ca4d5"} Jul 17 05:46:17.884054 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:17.883679 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerStarted","Data":"846a550de1593117c0844b820bdd4474763af10b7e75f1850cefd9b0c6e52c36"} Jul 17 05:46:17.913424 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:17.913357 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.660370537 podStartE2EDuration="7.913340597s" totalImagesPullingTime="5.233362096s" totalInitContainerRuntime="19.607964ms" isStatefulPod=false podCreationTimestamp="2026-07-17 05:46:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:17.91164582 +0000 UTC m=+302.656325148" watchObservedRunningTime="2026-07-17 05:46:17.913340597 +0000 UTC m=+302.658019946" Jul 17 05:46:19.112444 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:19.112367 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:20.469841 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:20.469804 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:20.557205 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:20.557170 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:46:20.731476 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:20.731367 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" podUID="5be596d5-da4f-4d3a-81a8-ecac9b05ffda" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 17 05:46:20.868417 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:20.868357 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:20.884461 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:20.884436 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:46:20.917583 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:20.917551 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:21.321659 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:21.321625 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:21.372642 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:21.372610 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:22.462217 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:22.462182 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:22.563517 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:22.563484 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:24.941995 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:24.941954 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:24.992617 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:24.992589 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:25.133172 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:25.133135 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:25.229996 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:25.229914 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:29.536929 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:29.536895 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:29.637635 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:29.637603 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:29.690588 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:29.690540 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:30.731124 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:30.731085 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" podUID="5be596d5-da4f-4d3a-81a8-ecac9b05ffda" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 17 05:46:32.089407 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:32.089357 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:32.136488 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:32.136455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:32.201844 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:32.201811 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:32.258119 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:32.258088 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:33.946156 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:33.946125 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:34.046823 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:34.046788 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:36.651696 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:36.651664 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:36.703474 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:36.703439 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:39.116907 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:39.116877 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:39.216231 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:39.216202 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:40.731141 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:40.731105 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" podUID="5be596d5-da4f-4d3a-81a8-ecac9b05ffda" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 17 05:46:40.731606 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:40.731173 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" Jul 17 05:46:40.731666 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:40.731642 2580 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"680fd245880721175bc16588f1abb51be25e1fdf99c1701a84d958247a56bdbf"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 17 05:46:40.731709 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:40.731688 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" podUID="5be596d5-da4f-4d3a-81a8-ecac9b05ffda" containerName="service-proxy" containerID="cri-o://680fd245880721175bc16588f1abb51be25e1fdf99c1701a84d958247a56bdbf" gracePeriod=30 Jul 17 05:46:40.969326 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:40.968207 2580 generic.go:356] "Generic (PLEG): container finished" podID="5be596d5-da4f-4d3a-81a8-ecac9b05ffda" containerID="680fd245880721175bc16588f1abb51be25e1fdf99c1701a84d958247a56bdbf" exitCode=2 Jul 17 05:46:40.969326 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:40.968250 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" event={"ID":"5be596d5-da4f-4d3a-81a8-ecac9b05ffda","Type":"ContainerDied","Data":"680fd245880721175bc16588f1abb51be25e1fdf99c1701a84d958247a56bdbf"} Jul 17 05:46:40.969326 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:40.968287 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778cd776d-njfzj" event={"ID":"5be596d5-da4f-4d3a-81a8-ecac9b05ffda","Type":"ContainerStarted","Data":"ff414bfdf4f699718e195a722d23e1a7d26923e97bc8796e22fb4f527b113508"} Jul 17 05:46:45.724173 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:45.724118 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:45.824673 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:45.824633 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:49.260488 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:49.260445 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:49.361226 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:49.361193 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:49.462838 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:49.462807 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:54.123638 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:54.123608 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:54.173516 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:54.173485 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:55.308722 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:55.308686 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:46:56.032682 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:56.032651 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xqr2c_e3a2951e-e5a2-49fc-a938-a5617748d2bb/node-ca/0.log" Jul 17 05:46:56.433970 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:46:56.433940 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dsrvg_3fde696a-f615-4e9f-b866-84073d4a923b/serve-healthcheck-canary/0.log" Jul 17 05:47:01.119797 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:01.119755 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:01.225053 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:01.225024 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:01.277292 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:01.277268 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:09.117944 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:09.117859 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:09.218704 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:09.218668 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:10.391089 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:10.391055 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:10.441488 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:10.441456 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:10.557435 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:10.557402 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:10.577305 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:10.577279 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:11.089560 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:11.089531 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:23.180376 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:23.180336 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:23.335692 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:23.335650 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:24.235646 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:24.235605 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 05:47:24.236212 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:24.235999 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="alertmanager" containerID="cri-o://dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb" gracePeriod=120 Jul 17 05:47:24.236212 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:24.236066 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="kube-rbac-proxy-metric" containerID="cri-o://32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1" gracePeriod=120 Jul 17 05:47:24.236212 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:24.236109 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="kube-rbac-proxy-web" containerID="cri-o://edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28" gracePeriod=120 Jul 17 05:47:24.236212 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:24.236157 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="config-reloader" containerID="cri-o://6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012" gracePeriod=120 Jul 17 05:47:24.236212 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:24.236170 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="kube-rbac-proxy" containerID="cri-o://c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d" gracePeriod=120 Jul 17 05:47:24.236212 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:24.236118 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="prom-label-proxy" containerID="cri-o://c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628" gracePeriod=120 Jul 17 05:47:25.130366 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.130335 2580 generic.go:356] "Generic (PLEG): container finished" podID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerID="c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628" exitCode=0 Jul 17 05:47:25.130366 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.130358 2580 generic.go:356] "Generic (PLEG): container finished" podID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerID="c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d" exitCode=0 Jul 17 05:47:25.130366 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.130365 2580 generic.go:356] "Generic (PLEG): container finished" podID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerID="6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012" exitCode=0 Jul 17 05:47:25.130366 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.130370 2580 generic.go:356] "Generic (PLEG): container finished" podID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerID="dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb" exitCode=0 Jul 17 05:47:25.130664 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.130420 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerDied","Data":"c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628"} Jul 17 05:47:25.130664 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.130480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerDied","Data":"c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d"} Jul 17 05:47:25.130664 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.130508 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerDied","Data":"6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012"} Jul 17 05:47:25.130664 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.130533 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerDied","Data":"dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb"} Jul 17 05:47:25.483705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.483680 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:25.501519 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.501486 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-tls-assets\") pod \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " Jul 17 05:47:25.501648 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.501530 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-web-config\") pod \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " Jul 17 05:47:25.501648 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.501569 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lbvw\" (UniqueName: \"kubernetes.io/projected/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-kube-api-access-4lbvw\") pod \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " Jul 17 05:47:25.501648 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.501591 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-cluster-tls-config\") pod \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " Jul 17 05:47:25.501648 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.501616 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-metrics-client-ca\") pod \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " Jul 17 05:47:25.501841 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.501650 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy-web\") pod \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " Jul 17 05:47:25.501841 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.501688 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-config-volume\") pod \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " Jul 17 05:47:25.501841 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.501711 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy\") pod \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " Jul 17 05:47:25.501841 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.501747 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-main-db\") pod \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " Jul 17 05:47:25.501841 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.501772 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " Jul 17 05:47:25.501841 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.501799 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-main-tls\") pod \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " Jul 17 05:47:25.501841 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.501839 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-trusted-ca-bundle\") pod \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " Jul 17 05:47:25.502169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.501893 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-config-out\") pod \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\" (UID: \"2a849f40-bde6-40b9-b2f4-b6441ec84e0e\") " Jul 17 05:47:25.502169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.502019 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "2a849f40-bde6-40b9-b2f4-b6441ec84e0e" (UID: "2a849f40-bde6-40b9-b2f4-b6441ec84e0e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:47:25.502169 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.502155 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-metrics-client-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:25.504039 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.503105 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "2a849f40-bde6-40b9-b2f4-b6441ec84e0e" (UID: "2a849f40-bde6-40b9-b2f4-b6441ec84e0e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:47:25.504172 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.504045 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "2a849f40-bde6-40b9-b2f4-b6441ec84e0e" (UID: "2a849f40-bde6-40b9-b2f4-b6441ec84e0e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:47:25.504688 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.504641 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2a849f40-bde6-40b9-b2f4-b6441ec84e0e" (UID: "2a849f40-bde6-40b9-b2f4-b6441ec84e0e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:47:25.504962 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.504930 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-config-volume" (OuterVolumeSpecName: "config-volume") pod "2a849f40-bde6-40b9-b2f4-b6441ec84e0e" (UID: "2a849f40-bde6-40b9-b2f4-b6441ec84e0e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:25.505199 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.505158 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "2a849f40-bde6-40b9-b2f4-b6441ec84e0e" (UID: "2a849f40-bde6-40b9-b2f4-b6441ec84e0e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:25.505313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.505218 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "2a849f40-bde6-40b9-b2f4-b6441ec84e0e" (UID: "2a849f40-bde6-40b9-b2f4-b6441ec84e0e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:25.505587 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.505563 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-kube-api-access-4lbvw" (OuterVolumeSpecName: "kube-api-access-4lbvw") pod "2a849f40-bde6-40b9-b2f4-b6441ec84e0e" (UID: "2a849f40-bde6-40b9-b2f4-b6441ec84e0e"). InnerVolumeSpecName "kube-api-access-4lbvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:47:25.505798 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.505770 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "2a849f40-bde6-40b9-b2f4-b6441ec84e0e" (UID: "2a849f40-bde6-40b9-b2f4-b6441ec84e0e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:25.505891 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.505821 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "2a849f40-bde6-40b9-b2f4-b6441ec84e0e" (UID: "2a849f40-bde6-40b9-b2f4-b6441ec84e0e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:25.506366 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.506326 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-config-out" (OuterVolumeSpecName: "config-out") pod "2a849f40-bde6-40b9-b2f4-b6441ec84e0e" (UID: "2a849f40-bde6-40b9-b2f4-b6441ec84e0e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:47:25.509444 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.509269 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "2a849f40-bde6-40b9-b2f4-b6441ec84e0e" (UID: "2a849f40-bde6-40b9-b2f4-b6441ec84e0e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:25.517300 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.517262 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-web-config" (OuterVolumeSpecName: "web-config") pod "2a849f40-bde6-40b9-b2f4-b6441ec84e0e" (UID: "2a849f40-bde6-40b9-b2f4-b6441ec84e0e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:25.603156 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.603125 2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-config-out\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:25.603156 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.603152 2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-tls-assets\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:25.603156 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.603161 2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-web-config\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:25.603369 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.603169 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lbvw\" (UniqueName: \"kubernetes.io/projected/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-kube-api-access-4lbvw\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:25.603369 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.603178 2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-cluster-tls-config\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:25.603369 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.603187 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:25.603369 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.603197 2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-config-volume\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:25.603369 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.603205 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:25.603369 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.603214 2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-main-db\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:25.603369 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.603223 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:25.603369 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.603231 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-secret-alertmanager-main-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:25.603369 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:25.603239 2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a849f40-bde6-40b9-b2f4-b6441ec84e0e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:26.136685 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.136651 2580 generic.go:356] "Generic (PLEG): container finished" podID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerID="32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1" exitCode=0 Jul 17 05:47:26.136685 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.136683 2580 generic.go:356] "Generic (PLEG): container finished" podID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerID="edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28" exitCode=0 Jul 17 05:47:26.136966 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.136714 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerDied","Data":"32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1"} Jul 17 05:47:26.136966 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.136740 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.136966 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.136782 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerDied","Data":"edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28"} Jul 17 05:47:26.136966 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.136811 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a849f40-bde6-40b9-b2f4-b6441ec84e0e","Type":"ContainerDied","Data":"99fc9fa1cb7fedcc166baf187e2c9ffe12c606379d80f7b8629e861257c329f8"} Jul 17 05:47:26.136966 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.136841 2580 scope.go:117] "RemoveContainer" containerID="c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628" Jul 17 05:47:26.144603 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.144562 2580 scope.go:117] "RemoveContainer" containerID="32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1" Jul 17 05:47:26.151602 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.151581 2580 scope.go:117] "RemoveContainer" containerID="c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d" Jul 17 05:47:26.158296 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.158258 2580 scope.go:117] "RemoveContainer" containerID="edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28" Jul 17 05:47:26.158433 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.158408 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 05:47:26.163035 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.163012 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 05:47:26.165459 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.165442 2580 scope.go:117] "RemoveContainer" containerID="6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012" Jul 17 05:47:26.172136 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.172120 2580 scope.go:117] "RemoveContainer" containerID="dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb" Jul 17 05:47:26.181420 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.181401 2580 scope.go:117] "RemoveContainer" containerID="2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5" Jul 17 05:47:26.186606 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.186584 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 05:47:26.186867 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.186853 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="kube-rbac-proxy-web" Jul 17 05:47:26.186935 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.186870 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="kube-rbac-proxy-web" Jul 17 05:47:26.186935 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.186882 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="kube-rbac-proxy" Jul 17 05:47:26.186935 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.186891 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="kube-rbac-proxy" Jul 17 05:47:26.186935 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.186899 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="prom-label-proxy" Jul 17 05:47:26.186935 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.186908 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="prom-label-proxy" Jul 17 05:47:26.186935 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.186921 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="config-reloader" Jul 17 05:47:26.186935 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.186930 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="config-reloader" Jul 17 05:47:26.187256 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.186940 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="kube-rbac-proxy-metric" Jul 17 05:47:26.187256 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.186948 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="kube-rbac-proxy-metric" Jul 17 05:47:26.187256 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.186963 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="alertmanager" Jul 17 05:47:26.187256 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.186972 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="alertmanager" Jul 17 05:47:26.187256 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.186995 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="init-config-reloader" Jul 17 05:47:26.187256 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.187003 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="init-config-reloader" Jul 17 05:47:26.187256 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.187067 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="kube-rbac-proxy-web" Jul 17 05:47:26.187256 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.187079 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="kube-rbac-proxy" Jul 17 05:47:26.187256 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.187088 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="kube-rbac-proxy-metric" Jul 17 05:47:26.187256 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.187099 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="prom-label-proxy" Jul 17 05:47:26.187256 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.187109 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="config-reloader" Jul 17 05:47:26.187256 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.187120 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" containerName="alertmanager" Jul 17 05:47:26.188721 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.188706 2580 scope.go:117] "RemoveContainer" containerID="c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628" Jul 17 05:47:26.188981 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:47:26.188958 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628\": container with ID starting with c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628 not found: ID does not exist" containerID="c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628" Jul 17 05:47:26.189049 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.188986 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628"} err="failed to get container status \"c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628\": rpc error: code = NotFound desc = could not find container \"c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628\": container with ID starting with c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628 not found: ID does not exist" Jul 17 05:47:26.189049 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.189002 2580 scope.go:117] "RemoveContainer" containerID="32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1" Jul 17 05:47:26.189221 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:47:26.189203 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1\": container with ID starting with 32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1 not found: ID does not exist" containerID="32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1" Jul 17 05:47:26.189260 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.189227 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1"} err="failed to get container status \"32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1\": rpc error: code = NotFound desc = could not find container \"32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1\": container with ID starting with 32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1 not found: ID does not exist" Jul 17 05:47:26.189260 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.189243 2580 scope.go:117] "RemoveContainer" containerID="c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d" Jul 17 05:47:26.189461 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:47:26.189440 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d\": container with ID starting with c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d not found: ID does not exist" containerID="c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d" Jul 17 05:47:26.189531 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.189470 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d"} err="failed to get container status \"c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d\": rpc error: code = NotFound desc = could not find container \"c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d\": container with ID starting with c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d not found: ID does not exist" Jul 17 05:47:26.189531 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.189491 2580 scope.go:117] "RemoveContainer" containerID="edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28" Jul 17 05:47:26.189724 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:47:26.189711 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28\": container with ID starting with edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28 not found: ID does not exist" containerID="edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28" Jul 17 05:47:26.189767 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.189729 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28"} err="failed to get container status \"edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28\": rpc error: code = NotFound desc = could not find container \"edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28\": container with ID starting with edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28 not found: ID does not exist" Jul 17 05:47:26.189767 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.189741 2580 scope.go:117] "RemoveContainer" containerID="6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012" Jul 17 05:47:26.189959 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:47:26.189941 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012\": container with ID starting with 6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012 not found: ID does not exist" containerID="6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012" Jul 17 05:47:26.189993 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.189965 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012"} err="failed to get container status \"6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012\": rpc error: code = NotFound desc = could not find container \"6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012\": container with ID starting with 6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012 not found: ID does not exist" Jul 17 05:47:26.189993 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.189979 2580 scope.go:117] "RemoveContainer" containerID="dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb" Jul 17 05:47:26.190188 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:47:26.190174 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb\": container with ID starting with dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb not found: ID does not exist" containerID="dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb" Jul 17 05:47:26.190221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.190193 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb"} err="failed to get container status \"dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb\": rpc error: code = NotFound desc = could not find container \"dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb\": container with ID starting with dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb not found: ID does not exist" Jul 17 05:47:26.190221 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.190209 2580 scope.go:117] "RemoveContainer" containerID="2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5" Jul 17 05:47:26.190560 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:47:26.190545 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5\": container with ID starting with 2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5 not found: ID does not exist" containerID="2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5" Jul 17 05:47:26.190610 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.190563 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5"} err="failed to get container status \"2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5\": rpc error: code = NotFound desc = could not find container \"2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5\": container with ID starting with 2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5 not found: ID does not exist" Jul 17 05:47:26.190610 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.190574 2580 scope.go:117] "RemoveContainer" containerID="c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628" Jul 17 05:47:26.190791 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.190770 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628"} err="failed to get container status \"c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628\": rpc error: code = NotFound desc = could not find container \"c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628\": container with ID starting with c340f0e9be3fa4e312cbf70f673d98d82682f68435e69ac617c94222cc736628 not found: ID does not exist" Jul 17 05:47:26.190791 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.190792 2580 scope.go:117] "RemoveContainer" containerID="32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1" Jul 17 05:47:26.190905 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.190836 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.191100 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.191005 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1"} err="failed to get container status \"32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1\": rpc error: code = NotFound desc = could not find container \"32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1\": container with ID starting with 32cdc2a80b11c2c2ee1c0b4ceed45eb1a9fd76c0f03e464e758302ec9468e2c1 not found: ID does not exist" Jul 17 05:47:26.191100 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.191028 2580 scope.go:117] "RemoveContainer" containerID="c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d" Jul 17 05:47:26.191311 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.191287 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d"} err="failed to get container status \"c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d\": rpc error: code = NotFound desc = could not find container \"c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d\": container with ID starting with c7c821b82c6a0cc922233ce8aeff091bd2db4014ebfbf41a3704104571d9384d not found: ID does not exist" Jul 17 05:47:26.191363 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.191315 2580 scope.go:117] "RemoveContainer" containerID="edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28" Jul 17 05:47:26.191607 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.191587 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28"} err="failed to get container status \"edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28\": rpc error: code = NotFound desc = could not find container \"edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28\": container with ID starting with edc92dce2b28199b7c1dea3cba3e4662c5107e966d51031fb4a0f25dcd4bba28 not found: ID does not exist" Jul 17 05:47:26.191607 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.191606 2580 scope.go:117] "RemoveContainer" containerID="6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012" Jul 17 05:47:26.191841 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.191823 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012"} err="failed to get container status \"6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012\": rpc error: code = NotFound desc = could not find container \"6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012\": container with ID starting with 6083e35adbcaa5e5f8e0019d099b9f4ce7e479eb12d1008a799d4c0483eee012 not found: ID does not exist" Jul 17 05:47:26.191841 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.191841 2580 scope.go:117] "RemoveContainer" containerID="dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb" Jul 17 05:47:26.192055 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.192034 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb"} err="failed to get container status \"dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb\": rpc error: code = NotFound desc = could not find container \"dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb\": container with ID starting with dd553a6552f0b3ac4b82195c02ca5256c9c2ed89fb3388af519f865472c2c7cb not found: ID does not exist" Jul 17 05:47:26.192102 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.192056 2580 scope.go:117] "RemoveContainer" containerID="2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5" Jul 17 05:47:26.192298 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.192280 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5"} err="failed to get container status \"2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5\": rpc error: code = NotFound desc = could not find container \"2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5\": container with ID starting with 2e7195d79ef20878097e8144f627abf9bacc25493ef3a4d16263bcccfcbf4cd5 not found: ID does not exist" Jul 17 05:47:26.192916 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.192901 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 05:47:26.193007 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.192901 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 05:47:26.193007 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.192940 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 05:47:26.193007 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.192946 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 05:47:26.193233 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.193218 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 05:47:26.193297 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.193272 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rdqft\"" Jul 17 05:47:26.193348 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.193314 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 05:47:26.193412 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.193353 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 05:47:26.193665 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.193638 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 05:47:26.198071 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.198053 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 05:47:26.203380 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.203361 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 05:47:26.208474 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.208455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0243024d-ed96-44ac-8b41-e334fbd21648-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.208582 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.208491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0243024d-ed96-44ac-8b41-e334fbd21648-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.208582 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.208523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0243024d-ed96-44ac-8b41-e334fbd21648-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.208694 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.208592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.208694 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.208619 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.208694 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.208649 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-config-volume\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.208694 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.208673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.208876 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.208715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.208876 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.208770 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.208876 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.208801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0243024d-ed96-44ac-8b41-e334fbd21648-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.208876 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.208827 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65r5z\" (UniqueName: \"kubernetes.io/projected/0243024d-ed96-44ac-8b41-e334fbd21648-kube-api-access-65r5z\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.209002 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.208878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0243024d-ed96-44ac-8b41-e334fbd21648-config-out\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.209002 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.208909 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-web-config\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.310110 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.310070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0243024d-ed96-44ac-8b41-e334fbd21648-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.310110 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.310114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0243024d-ed96-44ac-8b41-e334fbd21648-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.310322 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.310137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0243024d-ed96-44ac-8b41-e334fbd21648-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.310322 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.310169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.310322 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.310187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.310322 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.310293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-config-volume\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.310322 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.310314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.310604 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.310339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.310604 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.310377 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.310604 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.310432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0243024d-ed96-44ac-8b41-e334fbd21648-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.310604 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.310454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65r5z\" (UniqueName: \"kubernetes.io/projected/0243024d-ed96-44ac-8b41-e334fbd21648-kube-api-access-65r5z\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.310604 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.310485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0243024d-ed96-44ac-8b41-e334fbd21648-config-out\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.310604 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.310523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-web-config\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.310882 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.310693 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0243024d-ed96-44ac-8b41-e334fbd21648-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.311832 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.311142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0243024d-ed96-44ac-8b41-e334fbd21648-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.311832 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.311148 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0243024d-ed96-44ac-8b41-e334fbd21648-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.313415 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.313372 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.313638 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.313612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-config-volume\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.313737 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.313719 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.313792 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.313725 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.313837 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.313791 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-web-config\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.314222 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.313958 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.314861 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.314838 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0243024d-ed96-44ac-8b41-e334fbd21648-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.314937 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.314861 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0243024d-ed96-44ac-8b41-e334fbd21648-config-out\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.314990 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.314970 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0243024d-ed96-44ac-8b41-e334fbd21648-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.318440 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.318419 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65r5z\" (UniqueName: \"kubernetes.io/projected/0243024d-ed96-44ac-8b41-e334fbd21648-kube-api-access-65r5z\") pod \"alertmanager-main-0\" (UID: \"0243024d-ed96-44ac-8b41-e334fbd21648\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.500308 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.500213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 05:47:26.626861 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.626827 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 05:47:26.629685 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:47:26.629662 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0243024d_ed96_44ac_8b41_e334fbd21648.slice/crio-d6f7235536d9fc8f20aa49824e637f787b61970a2ac3daeaf94d329ff751b4d6 WatchSource:0}: Error finding container d6f7235536d9fc8f20aa49824e637f787b61970a2ac3daeaf94d329ff751b4d6: Status 404 returned error can't find the container with id d6f7235536d9fc8f20aa49824e637f787b61970a2ac3daeaf94d329ff751b4d6 Jul 17 05:47:26.888348 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:26.888313 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a849f40-bde6-40b9-b2f4-b6441ec84e0e" path="/var/lib/kubelet/pods/2a849f40-bde6-40b9-b2f4-b6441ec84e0e/volumes" Jul 17 05:47:27.142294 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:27.142201 2580 generic.go:356] "Generic (PLEG): container finished" podID="0243024d-ed96-44ac-8b41-e334fbd21648" containerID="e3c4f6cce176bfd54f112da2efcb6d955eff9ab1a6914bdfdfbcbd38abd0fd11" exitCode=0 Jul 17 05:47:27.142294 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:27.142254 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0243024d-ed96-44ac-8b41-e334fbd21648","Type":"ContainerDied","Data":"e3c4f6cce176bfd54f112da2efcb6d955eff9ab1a6914bdfdfbcbd38abd0fd11"} Jul 17 05:47:27.142504 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:27.142318 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0243024d-ed96-44ac-8b41-e334fbd21648","Type":"ContainerStarted","Data":"d6f7235536d9fc8f20aa49824e637f787b61970a2ac3daeaf94d329ff751b4d6"} Jul 17 05:47:28.150137 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.150098 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0243024d-ed96-44ac-8b41-e334fbd21648","Type":"ContainerStarted","Data":"0dd6ecbeda104bd30b991dd49ec9b4b2fb1669dbd30d0765fd4fd17e942a40d0"} Jul 17 05:47:28.150597 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.150161 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0243024d-ed96-44ac-8b41-e334fbd21648","Type":"ContainerStarted","Data":"fa789baca58c2e1a64c7412458473e80c2da912bb574343471571d517aa9cb5b"} Jul 17 05:47:28.150597 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.150185 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0243024d-ed96-44ac-8b41-e334fbd21648","Type":"ContainerStarted","Data":"1528a1b85c48cf1866599f5db2d6abfcd54b4e7e30c9b9ec2deed4a3f02561d3"} Jul 17 05:47:28.150597 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.150213 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0243024d-ed96-44ac-8b41-e334fbd21648","Type":"ContainerStarted","Data":"c0692b3a58df6cc75fa06e02afd3fc3e71aaa681268d9a29205c1abeb365eca3"} Jul 17 05:47:28.150597 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.150236 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0243024d-ed96-44ac-8b41-e334fbd21648","Type":"ContainerStarted","Data":"be7a70cc2e1c2a27401ea353ae9354d24792200a74ee3dfbe64c496709320e1b"} Jul 17 05:47:28.150597 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.150259 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0243024d-ed96-44ac-8b41-e334fbd21648","Type":"ContainerStarted","Data":"f88a5fd5eabbd4888b847ef346dddede1dc7df442a4e628f1260a295688a7e75"} Jul 17 05:47:28.175145 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.175081 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.160588063 podStartE2EDuration="2.175067243s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.47918ms" isStatefulPod=false podCreationTimestamp="2026-07-17 05:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:47:28.174775106 +0000 UTC m=+372.919454450" watchObservedRunningTime="2026-07-17 05:47:28.175067243 +0000 UTC m=+372.919746606" Jul 17 05:47:28.600452 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.600362 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 05:47:28.600681 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.600642 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="prometheus" containerID="cri-o://a7f9c9bacfb4c047b4fa13f703fc62aa8cf2885b6eecc52b26e776ee18315799" gracePeriod=600 Jul 17 05:47:28.600820 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.600662 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="thanos-sidecar" containerID="cri-o://846a550de1593117c0844b820bdd4474763af10b7e75f1850cefd9b0c6e52c36" gracePeriod=600 Jul 17 05:47:28.600820 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.600697 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="config-reloader" containerID="cri-o://1e94df5034262ce6b8776dc6f009b8f77242c03c7e80a305f0d9fcbb1ea2c183" gracePeriod=600 Jul 17 05:47:28.600820 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.600720 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="kube-rbac-proxy-web" containerID="cri-o://af12e00aee5a9ddbff9f396a08d31ff9d07353924152b309d50c4c47182ca4d5" gracePeriod=600 Jul 17 05:47:28.600820 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.600669 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="kube-rbac-proxy" containerID="cri-o://67a42667cf88cbc2544aa4a1c32e38ba699c32eef60a896426058d0ea555e852" gracePeriod=600 Jul 17 05:47:28.600820 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.600705 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c3bd07546f73a01e3213c4392d9b7b1231dcbf0415b784baf930bc0a17ccd3d0" gracePeriod=600 Jul 17 05:47:28.686128 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.686097 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:28.752507 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:28.752475 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:29.156649 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.156610 2580 generic.go:356] "Generic (PLEG): container finished" podID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerID="c3bd07546f73a01e3213c4392d9b7b1231dcbf0415b784baf930bc0a17ccd3d0" exitCode=0 Jul 17 05:47:29.156649 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.156638 2580 generic.go:356] "Generic (PLEG): container finished" podID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerID="67a42667cf88cbc2544aa4a1c32e38ba699c32eef60a896426058d0ea555e852" exitCode=0 Jul 17 05:47:29.156649 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.156644 2580 generic.go:356] "Generic (PLEG): container finished" podID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerID="846a550de1593117c0844b820bdd4474763af10b7e75f1850cefd9b0c6e52c36" exitCode=0 Jul 17 05:47:29.156649 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.156652 2580 generic.go:356] "Generic (PLEG): container finished" podID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerID="1e94df5034262ce6b8776dc6f009b8f77242c03c7e80a305f0d9fcbb1ea2c183" exitCode=0 Jul 17 05:47:29.156649 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.156657 2580 generic.go:356] "Generic (PLEG): container finished" podID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerID="a7f9c9bacfb4c047b4fa13f703fc62aa8cf2885b6eecc52b26e776ee18315799" exitCode=0 Jul 17 05:47:29.157196 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.156681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerDied","Data":"c3bd07546f73a01e3213c4392d9b7b1231dcbf0415b784baf930bc0a17ccd3d0"} Jul 17 05:47:29.157196 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.156792 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerDied","Data":"67a42667cf88cbc2544aa4a1c32e38ba699c32eef60a896426058d0ea555e852"} Jul 17 05:47:29.157196 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.156833 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerDied","Data":"846a550de1593117c0844b820bdd4474763af10b7e75f1850cefd9b0c6e52c36"} Jul 17 05:47:29.157196 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.156871 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerDied","Data":"1e94df5034262ce6b8776dc6f009b8f77242c03c7e80a305f0d9fcbb1ea2c183"} Jul 17 05:47:29.157196 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.156908 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerDied","Data":"a7f9c9bacfb4c047b4fa13f703fc62aa8cf2885b6eecc52b26e776ee18315799"} Jul 17 05:47:29.830297 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.830268 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:29.943195 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.943161 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bdfcf2fb-f8d5-4088-9833-df8c31c77385-tls-assets\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.943195 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.943198 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-0\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.943433 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.943229 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bdfcf2fb-f8d5-4088-9833-df8c31c77385-config-out\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.943433 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.943250 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-trusted-ca-bundle\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.943543 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.943431 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-config\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.943543 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.943483 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-thanos-prometheus-http-client-file\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.943648 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.943623 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-kubelet-serving-ca-bundle\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.943700 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.943665 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:47:29.943757 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.943665 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-metrics-client-certs\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.943829 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.943785 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-metrics-client-ca\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.943829 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.943818 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-web-config\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.943925 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.943903 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-db\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.943973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.943928 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2792j\" (UniqueName: \"kubernetes.io/projected/bdfcf2fb-f8d5-4088-9833-df8c31c77385-kube-api-access-2792j\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.943973 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.943965 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-2\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.944070 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.944008 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-kube-rbac-proxy\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.944070 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.944036 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:47:29.944162 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.944067 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-1\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.944162 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.944093 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-serving-certs-ca-bundle\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.944162 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.944127 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.944162 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.944157 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-tls\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.944354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.944194 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.944354 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.944221 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-grpc-tls\") pod \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\" (UID: \"bdfcf2fb-f8d5-4088-9833-df8c31c77385\") " Jul 17 05:47:29.944532 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.944512 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:29.944592 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.944536 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:29.944719 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.944694 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:47:29.945541 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.945161 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:47:29.945541 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.945297 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:47:29.945728 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.945648 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:47:29.945728 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.945697 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:29.946321 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.946064 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfcf2fb-f8d5-4088-9833-df8c31c77385-config-out" (OuterVolumeSpecName: "config-out") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:47:29.946441 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.946419 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:29.946505 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.946434 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:47:29.946505 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.946472 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdfcf2fb-f8d5-4088-9833-df8c31c77385-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:47:29.946813 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.946781 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:47:29.947322 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.947243 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-config" (OuterVolumeSpecName: "config") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:29.947637 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.947600 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:29.947932 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.947908 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdfcf2fb-f8d5-4088-9833-df8c31c77385-kube-api-access-2792j" (OuterVolumeSpecName: "kube-api-access-2792j") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "kube-api-access-2792j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:47:29.947932 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.947908 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:29.948673 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.948652 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:29.948833 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.948816 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:29.948887 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.948835 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:29.978932 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:29.978708 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-web-config" (OuterVolumeSpecName: "web-config") pod "bdfcf2fb-f8d5-4088-9833-df8c31c77385" (UID: "bdfcf2fb-f8d5-4088-9833-df8c31c77385"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:47:30.045250 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045217 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-db\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045250 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045246 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2792j\" (UniqueName: \"kubernetes.io/projected/bdfcf2fb-f8d5-4088-9833-df8c31c77385-kube-api-access-2792j\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045250 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045256 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045265 2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-kube-rbac-proxy\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045274 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045283 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045293 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045304 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045312 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045321 2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-grpc-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045329 2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bdfcf2fb-f8d5-4088-9833-df8c31c77385-tls-assets\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045338 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045346 2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bdfcf2fb-f8d5-4088-9833-df8c31c77385-config-out\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045354 2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-config\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045361 2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045369 2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-secret-metrics-client-certs\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045377 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdfcf2fb-f8d5-4088-9833-df8c31c77385-configmap-metrics-client-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.045489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.045407 2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bdfcf2fb-f8d5-4088-9833-df8c31c77385-web-config\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 05:47:30.146062 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.146028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:47:30.148211 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.148192 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c472db7-71bf-46b2-9452-f936c8c53e67-metrics-certs\") pod \"network-metrics-daemon-nsrf4\" (UID: \"3c472db7-71bf-46b2-9452-f936c8c53e67\") " pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:47:30.163617 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.163588 2580 generic.go:356] "Generic (PLEG): container finished" podID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerID="af12e00aee5a9ddbff9f396a08d31ff9d07353924152b309d50c4c47182ca4d5" exitCode=0 Jul 17 05:47:30.163956 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.163622 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerDied","Data":"af12e00aee5a9ddbff9f396a08d31ff9d07353924152b309d50c4c47182ca4d5"} Jul 17 05:47:30.163956 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.163658 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.163956 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.163675 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bdfcf2fb-f8d5-4088-9833-df8c31c77385","Type":"ContainerDied","Data":"00b35dc5ed60600bccb85b5cb88f012252f7894bc9b557682992f826668de423"} Jul 17 05:47:30.163956 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.163704 2580 scope.go:117] "RemoveContainer" containerID="c3bd07546f73a01e3213c4392d9b7b1231dcbf0415b784baf930bc0a17ccd3d0" Jul 17 05:47:30.172045 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.172028 2580 scope.go:117] "RemoveContainer" containerID="67a42667cf88cbc2544aa4a1c32e38ba699c32eef60a896426058d0ea555e852" Jul 17 05:47:30.178587 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.178572 2580 scope.go:117] "RemoveContainer" containerID="af12e00aee5a9ddbff9f396a08d31ff9d07353924152b309d50c4c47182ca4d5" Jul 17 05:47:30.185367 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.185342 2580 scope.go:117] "RemoveContainer" containerID="846a550de1593117c0844b820bdd4474763af10b7e75f1850cefd9b0c6e52c36" Jul 17 05:47:30.186015 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.185995 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 05:47:30.187429 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.187378 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7x5wz\"" Jul 17 05:47:30.190363 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.190339 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 05:47:30.192665 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.192648 2580 scope.go:117] "RemoveContainer" containerID="1e94df5034262ce6b8776dc6f009b8f77242c03c7e80a305f0d9fcbb1ea2c183" Jul 17 05:47:30.195695 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.195679 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsrf4" Jul 17 05:47:30.199686 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.199666 2580 scope.go:117] "RemoveContainer" containerID="a7f9c9bacfb4c047b4fa13f703fc62aa8cf2885b6eecc52b26e776ee18315799" Jul 17 05:47:30.210475 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.210437 2580 scope.go:117] "RemoveContainer" containerID="33756dd6d61e120b1b88190b9a939e4fd47963fa7a47b6f2afdd3eba702c9e64" Jul 17 05:47:30.214568 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214513 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 05:47:30.214798 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214785 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="kube-rbac-proxy" Jul 17 05:47:30.214839 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214800 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="kube-rbac-proxy" Jul 17 05:47:30.214839 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214810 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="prometheus" Jul 17 05:47:30.214839 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214815 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="prometheus" Jul 17 05:47:30.214839 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214823 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="init-config-reloader" Jul 17 05:47:30.214839 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214828 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="init-config-reloader" Jul 17 05:47:30.214839 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214838 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="kube-rbac-proxy-web" Jul 17 05:47:30.215061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214844 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="kube-rbac-proxy-web" Jul 17 05:47:30.215061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214856 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="config-reloader" Jul 17 05:47:30.215061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214861 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="config-reloader" Jul 17 05:47:30.215061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214868 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="kube-rbac-proxy-thanos" Jul 17 05:47:30.215061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214873 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="kube-rbac-proxy-thanos" Jul 17 05:47:30.215061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214883 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="thanos-sidecar" Jul 17 05:47:30.215061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214887 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="thanos-sidecar" Jul 17 05:47:30.215061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214948 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="config-reloader" Jul 17 05:47:30.215061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214960 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="kube-rbac-proxy-thanos" Jul 17 05:47:30.215061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214971 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="kube-rbac-proxy-web" Jul 17 05:47:30.215061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214978 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="kube-rbac-proxy" Jul 17 05:47:30.215061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214985 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="thanos-sidecar" Jul 17 05:47:30.215061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.214994 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" containerName="prometheus" Jul 17 05:47:30.217998 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.217967 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.218517 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.218469 2580 scope.go:117] "RemoveContainer" containerID="c3bd07546f73a01e3213c4392d9b7b1231dcbf0415b784baf930bc0a17ccd3d0" Jul 17 05:47:30.219010 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:47:30.218867 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3bd07546f73a01e3213c4392d9b7b1231dcbf0415b784baf930bc0a17ccd3d0\": container with ID starting with c3bd07546f73a01e3213c4392d9b7b1231dcbf0415b784baf930bc0a17ccd3d0 not found: ID does not exist" containerID="c3bd07546f73a01e3213c4392d9b7b1231dcbf0415b784baf930bc0a17ccd3d0" Jul 17 05:47:30.219010 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.218902 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3bd07546f73a01e3213c4392d9b7b1231dcbf0415b784baf930bc0a17ccd3d0"} err="failed to get container status \"c3bd07546f73a01e3213c4392d9b7b1231dcbf0415b784baf930bc0a17ccd3d0\": rpc error: code = NotFound desc = could not find container \"c3bd07546f73a01e3213c4392d9b7b1231dcbf0415b784baf930bc0a17ccd3d0\": container with ID starting with c3bd07546f73a01e3213c4392d9b7b1231dcbf0415b784baf930bc0a17ccd3d0 not found: ID does not exist" Jul 17 05:47:30.219010 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.218929 2580 scope.go:117] "RemoveContainer" containerID="67a42667cf88cbc2544aa4a1c32e38ba699c32eef60a896426058d0ea555e852" Jul 17 05:47:30.219251 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:47:30.219228 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a42667cf88cbc2544aa4a1c32e38ba699c32eef60a896426058d0ea555e852\": container with ID starting with 67a42667cf88cbc2544aa4a1c32e38ba699c32eef60a896426058d0ea555e852 not found: ID does not exist" containerID="67a42667cf88cbc2544aa4a1c32e38ba699c32eef60a896426058d0ea555e852" Jul 17 05:47:30.219308 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.219262 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a42667cf88cbc2544aa4a1c32e38ba699c32eef60a896426058d0ea555e852"} err="failed to get container status \"67a42667cf88cbc2544aa4a1c32e38ba699c32eef60a896426058d0ea555e852\": rpc error: code = NotFound desc = could not find container \"67a42667cf88cbc2544aa4a1c32e38ba699c32eef60a896426058d0ea555e852\": container with ID starting with 67a42667cf88cbc2544aa4a1c32e38ba699c32eef60a896426058d0ea555e852 not found: ID does not exist" Jul 17 05:47:30.219308 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.219285 2580 scope.go:117] "RemoveContainer" containerID="af12e00aee5a9ddbff9f396a08d31ff9d07353924152b309d50c4c47182ca4d5" Jul 17 05:47:30.219592 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:47:30.219567 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af12e00aee5a9ddbff9f396a08d31ff9d07353924152b309d50c4c47182ca4d5\": container with ID starting with af12e00aee5a9ddbff9f396a08d31ff9d07353924152b309d50c4c47182ca4d5 not found: ID does not exist" containerID="af12e00aee5a9ddbff9f396a08d31ff9d07353924152b309d50c4c47182ca4d5" Jul 17 05:47:30.219696 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.219601 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af12e00aee5a9ddbff9f396a08d31ff9d07353924152b309d50c4c47182ca4d5"} err="failed to get container status \"af12e00aee5a9ddbff9f396a08d31ff9d07353924152b309d50c4c47182ca4d5\": rpc error: code = NotFound desc = could not find container \"af12e00aee5a9ddbff9f396a08d31ff9d07353924152b309d50c4c47182ca4d5\": container with ID starting with af12e00aee5a9ddbff9f396a08d31ff9d07353924152b309d50c4c47182ca4d5 not found: ID does not exist" Jul 17 05:47:30.219696 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.219620 2580 scope.go:117] "RemoveContainer" containerID="846a550de1593117c0844b820bdd4474763af10b7e75f1850cefd9b0c6e52c36" Jul 17 05:47:30.219911 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:47:30.219892 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846a550de1593117c0844b820bdd4474763af10b7e75f1850cefd9b0c6e52c36\": container with ID starting with 846a550de1593117c0844b820bdd4474763af10b7e75f1850cefd9b0c6e52c36 not found: ID does not exist" containerID="846a550de1593117c0844b820bdd4474763af10b7e75f1850cefd9b0c6e52c36" Jul 17 05:47:30.219981 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.219914 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846a550de1593117c0844b820bdd4474763af10b7e75f1850cefd9b0c6e52c36"} err="failed to get container status \"846a550de1593117c0844b820bdd4474763af10b7e75f1850cefd9b0c6e52c36\": rpc error: code = NotFound desc = could not find container \"846a550de1593117c0844b820bdd4474763af10b7e75f1850cefd9b0c6e52c36\": container with ID starting with 846a550de1593117c0844b820bdd4474763af10b7e75f1850cefd9b0c6e52c36 not found: ID does not exist" Jul 17 05:47:30.219981 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.219931 2580 scope.go:117] "RemoveContainer" containerID="1e94df5034262ce6b8776dc6f009b8f77242c03c7e80a305f0d9fcbb1ea2c183" Jul 17 05:47:30.220218 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:47:30.220176 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e94df5034262ce6b8776dc6f009b8f77242c03c7e80a305f0d9fcbb1ea2c183\": container with ID starting with 1e94df5034262ce6b8776dc6f009b8f77242c03c7e80a305f0d9fcbb1ea2c183 not found: ID does not exist" containerID="1e94df5034262ce6b8776dc6f009b8f77242c03c7e80a305f0d9fcbb1ea2c183" Jul 17 05:47:30.220218 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.220206 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e94df5034262ce6b8776dc6f009b8f77242c03c7e80a305f0d9fcbb1ea2c183"} err="failed to get container status \"1e94df5034262ce6b8776dc6f009b8f77242c03c7e80a305f0d9fcbb1ea2c183\": rpc error: code = NotFound desc = could not find container \"1e94df5034262ce6b8776dc6f009b8f77242c03c7e80a305f0d9fcbb1ea2c183\": container with ID starting with 1e94df5034262ce6b8776dc6f009b8f77242c03c7e80a305f0d9fcbb1ea2c183 not found: ID does not exist" Jul 17 05:47:30.220434 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.220226 2580 scope.go:117] "RemoveContainer" containerID="a7f9c9bacfb4c047b4fa13f703fc62aa8cf2885b6eecc52b26e776ee18315799" Jul 17 05:47:30.220434 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.220277 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 05:47:30.220553 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:47:30.220485 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f9c9bacfb4c047b4fa13f703fc62aa8cf2885b6eecc52b26e776ee18315799\": container with ID starting with a7f9c9bacfb4c047b4fa13f703fc62aa8cf2885b6eecc52b26e776ee18315799 not found: ID does not exist" containerID="a7f9c9bacfb4c047b4fa13f703fc62aa8cf2885b6eecc52b26e776ee18315799" Jul 17 05:47:30.220553 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.220513 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f9c9bacfb4c047b4fa13f703fc62aa8cf2885b6eecc52b26e776ee18315799"} err="failed to get container status \"a7f9c9bacfb4c047b4fa13f703fc62aa8cf2885b6eecc52b26e776ee18315799\": rpc error: code = NotFound desc = could not find container \"a7f9c9bacfb4c047b4fa13f703fc62aa8cf2885b6eecc52b26e776ee18315799\": container with ID starting with a7f9c9bacfb4c047b4fa13f703fc62aa8cf2885b6eecc52b26e776ee18315799 not found: ID does not exist" Jul 17 05:47:30.220553 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.220543 2580 scope.go:117] "RemoveContainer" containerID="33756dd6d61e120b1b88190b9a939e4fd47963fa7a47b6f2afdd3eba702c9e64" Jul 17 05:47:30.220808 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.220595 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mlkbz\"" Jul 17 05:47:30.220808 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.220615 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 05:47:30.220808 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.220670 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 05:47:30.220808 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.220712 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 05:47:30.221220 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.220843 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 05:47:30.221220 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.220919 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 05:47:30.221220 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.221059 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 05:47:30.221220 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.221192 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 05:47:30.221220 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.221214 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3svbn0ehar4bm\"" Jul 17 05:47:30.221607 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.221316 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 05:47:30.221607 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.221461 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 05:47:30.221607 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.221539 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 05:47:30.221607 ip-10-0-142-12 kubenswrapper[2580]: E0717 05:47:30.221546 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33756dd6d61e120b1b88190b9a939e4fd47963fa7a47b6f2afdd3eba702c9e64\": container with ID starting with 33756dd6d61e120b1b88190b9a939e4fd47963fa7a47b6f2afdd3eba702c9e64 not found: ID does not exist" containerID="33756dd6d61e120b1b88190b9a939e4fd47963fa7a47b6f2afdd3eba702c9e64" Jul 17 05:47:30.221607 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.221575 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33756dd6d61e120b1b88190b9a939e4fd47963fa7a47b6f2afdd3eba702c9e64"} err="failed to get container status \"33756dd6d61e120b1b88190b9a939e4fd47963fa7a47b6f2afdd3eba702c9e64\": rpc error: code = NotFound desc = could not find container \"33756dd6d61e120b1b88190b9a939e4fd47963fa7a47b6f2afdd3eba702c9e64\": container with ID starting with 33756dd6d61e120b1b88190b9a939e4fd47963fa7a47b6f2afdd3eba702c9e64 not found: ID does not exist" Jul 17 05:47:30.221607 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.221606 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 05:47:30.221970 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.221620 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 05:47:30.224395 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.224359 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 05:47:30.226924 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.226902 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 05:47:30.233324 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.233292 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 05:47:30.246870 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.246845 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-web-config\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.246995 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.246878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.246995 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.246903 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.246995 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.246927 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dgfc\" (UniqueName: \"kubernetes.io/projected/15cb2621-c194-4892-ae7b-dc9ee28f2db2-kube-api-access-4dgfc\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.246995 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.246948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.246995 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.246972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-config\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.247306 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.247003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.247306 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.247045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.247306 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.247076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.247306 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.247112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15cb2621-c194-4892-ae7b-dc9ee28f2db2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.247306 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.247137 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15cb2621-c194-4892-ae7b-dc9ee28f2db2-config-out\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.247306 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.247172 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.247306 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.247197 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.247306 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.247227 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.247306 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.247258 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.247306 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.247283 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.247780 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.247314 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.247780 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.247345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.247780 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.247370 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.247780 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.247424 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.323258 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.323224 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nsrf4"] Jul 17 05:47:30.326319 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:47:30.326292 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c472db7_71bf_46b2_9452_f936c8c53e67.slice/crio-676f786d55da0064b2a09de3d88c033e33dd4a15b43cc52fb2b26a24166947ff WatchSource:0}: Error finding container 676f786d55da0064b2a09de3d88c033e33dd4a15b43cc52fb2b26a24166947ff: Status 404 returned error can't find the container with id 676f786d55da0064b2a09de3d88c033e33dd4a15b43cc52fb2b26a24166947ff Jul 17 05:47:30.328184 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.328168 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 05:47:30.347948 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.347927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348033 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.347960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15cb2621-c194-4892-ae7b-dc9ee28f2db2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348033 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.347978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15cb2621-c194-4892-ae7b-dc9ee28f2db2-config-out\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348121 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348121 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348208 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348208 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348161 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348208 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348344 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348427 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348345 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348427 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348529 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348529 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348472 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-web-config\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348529 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348501 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348529 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348720 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dgfc\" (UniqueName: \"kubernetes.io/projected/15cb2621-c194-4892-ae7b-dc9ee28f2db2-kube-api-access-4dgfc\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348720 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348720 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348597 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348720 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-config\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.348720 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.348695 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.349414 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.349361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.350537 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.350512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.350630 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.350587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.350681 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.350639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.351306 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.351069 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-config\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.351306 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.351242 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15cb2621-c194-4892-ae7b-dc9ee28f2db2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.351481 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.351325 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.351552 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.351507 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.351627 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.351606 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.351912 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.351866 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.351999 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.351939 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.352215 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.352178 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15cb2621-c194-4892-ae7b-dc9ee28f2db2-config-out\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.352324 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.352303 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-web-config\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.352551 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.352535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.352864 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.352841 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.353061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.353045 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.353152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.353137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.353711 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.353688 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15cb2621-c194-4892-ae7b-dc9ee28f2db2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.354130 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.354112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/15cb2621-c194-4892-ae7b-dc9ee28f2db2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.357866 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.357849 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dgfc\" (UniqueName: \"kubernetes.io/projected/15cb2621-c194-4892-ae7b-dc9ee28f2db2-kube-api-access-4dgfc\") pod \"prometheus-k8s-0\" (UID: \"15cb2621-c194-4892-ae7b-dc9ee28f2db2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.529310 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.529221 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:30.658937 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.658914 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 05:47:30.661194 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:47:30.661167 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15cb2621_c194_4892_ae7b_dc9ee28f2db2.slice/crio-ab29af7982d71ae264dd31f977261dc8e45f3d2b62abd6ace7d0fcbf47fab27e WatchSource:0}: Error finding container ab29af7982d71ae264dd31f977261dc8e45f3d2b62abd6ace7d0fcbf47fab27e: Status 404 returned error can't find the container with id ab29af7982d71ae264dd31f977261dc8e45f3d2b62abd6ace7d0fcbf47fab27e Jul 17 05:47:30.890044 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:30.890006 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdfcf2fb-f8d5-4088-9833-df8c31c77385" path="/var/lib/kubelet/pods/bdfcf2fb-f8d5-4088-9833-df8c31c77385/volumes" Jul 17 05:47:31.170954 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:31.170913 2580 generic.go:356] "Generic (PLEG): container finished" podID="15cb2621-c194-4892-ae7b-dc9ee28f2db2" containerID="9732704ae09c3e4cc970b270e05887b13a17409269dd5e1b24ed83d47ef553a5" exitCode=0 Jul 17 05:47:31.171442 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:31.171001 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15cb2621-c194-4892-ae7b-dc9ee28f2db2","Type":"ContainerDied","Data":"9732704ae09c3e4cc970b270e05887b13a17409269dd5e1b24ed83d47ef553a5"} Jul 17 05:47:31.171442 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:31.171089 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15cb2621-c194-4892-ae7b-dc9ee28f2db2","Type":"ContainerStarted","Data":"ab29af7982d71ae264dd31f977261dc8e45f3d2b62abd6ace7d0fcbf47fab27e"} Jul 17 05:47:31.172447 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:31.172418 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nsrf4" event={"ID":"3c472db7-71bf-46b2-9452-f936c8c53e67","Type":"ContainerStarted","Data":"676f786d55da0064b2a09de3d88c033e33dd4a15b43cc52fb2b26a24166947ff"} Jul 17 05:47:32.179298 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:32.179263 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15cb2621-c194-4892-ae7b-dc9ee28f2db2","Type":"ContainerStarted","Data":"a38716be8b9ef66e3d2d334688681982c76b9f1add624020f1c820157064c7b0"} Jul 17 05:47:32.179781 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:32.179367 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15cb2621-c194-4892-ae7b-dc9ee28f2db2","Type":"ContainerStarted","Data":"6f02b0816da938e4416fd3cd109240ed61534185f7c86e9573613fe8828d6bd7"} Jul 17 05:47:32.179781 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:32.179423 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15cb2621-c194-4892-ae7b-dc9ee28f2db2","Type":"ContainerStarted","Data":"057f9418b9040f08b25fc36f1b3f0339198c08800a439fde9cb476085f366f98"} Jul 17 05:47:32.179781 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:32.179452 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15cb2621-c194-4892-ae7b-dc9ee28f2db2","Type":"ContainerStarted","Data":"019c7d1d8e796edfcf0476d02ad0b07d49b5bcbec9d7542179ac2e86245e8db8"} Jul 17 05:47:32.179781 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:32.179479 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15cb2621-c194-4892-ae7b-dc9ee28f2db2","Type":"ContainerStarted","Data":"05d562f3567f148e8c6256b51f62c611b16dec132af4c0cdabe089043bb70dcc"} Jul 17 05:47:32.179781 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:32.179506 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15cb2621-c194-4892-ae7b-dc9ee28f2db2","Type":"ContainerStarted","Data":"dbe76fca135f38a77b3ec90bdca0cf62ac2c7dc5b536fbba1d75a8d27c2c7c2c"} Jul 17 05:47:32.181053 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:32.181031 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nsrf4" event={"ID":"3c472db7-71bf-46b2-9452-f936c8c53e67","Type":"ContainerStarted","Data":"880a2da29db4ee34862015f0d037626de3811eef74c19a3229499b4ae8c3ae9a"} Jul 17 05:47:32.181130 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:32.181076 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nsrf4" event={"ID":"3c472db7-71bf-46b2-9452-f936c8c53e67","Type":"ContainerStarted","Data":"0a440c2de240e312bb5a4353d45d21cc6a2ccbf27b7276ba17a77f0b6b78f5f1"} Jul 17 05:47:32.206328 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:32.206273 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.187164831 podStartE2EDuration="2.206253549s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.088718ms" isStatefulPod=false podCreationTimestamp="2026-07-17 05:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:47:32.204205213 +0000 UTC m=+376.948884552" watchObservedRunningTime="2026-07-17 05:47:32.206253549 +0000 UTC m=+376.950932876" Jul 17 05:47:32.219830 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:32.219788 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nsrf4" podStartSLOduration=374.335970701 podStartE2EDuration="6m15.219777798s" totalImagesPullingTime="883.807097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:47:32.218055437 +0000 UTC m=+376.962734762" watchObservedRunningTime="2026-07-17 05:47:32.219777798 +0000 UTC m=+376.964457103" Jul 17 05:47:32.304128 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:32.304092 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:32.402787 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:32.402751 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:33.418876 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:33.418844 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:33.519939 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:33.519892 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:33.618466 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:33.618432 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:34.152597 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:34.152567 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:34.203140 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:34.203110 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:34.331969 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:34.331928 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:34.433325 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:34.433243 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:35.380722 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:35.380689 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:35.530156 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:35.530116 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:47:39.116323 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:39.116292 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:39.216781 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:39.216753 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:44.800136 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:44.800101 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:44.857941 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:44.857910 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:47:44.913696 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:47:44.913662 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:48:25.517885 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:25.517852 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:48:25.618422 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:25.618368 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:48:30.529894 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:30.529854 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:48:30.544913 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:30.544888 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:48:31.409592 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:31.409565 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 05:48:38.401133 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:38.401049 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:48:38.452258 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:38.452199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:48:42.172696 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:42.172650 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:48:50.327300 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:50.327263 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:48:50.428033 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:50.428002 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:48:53.029854 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:53.029820 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:48:53.133914 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:53.133878 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:48:53.235289 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:53.235261 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:48:54.126676 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:54.126643 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:48:54.225300 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:54.225271 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:48:54.399929 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:48:54.399847 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:20.656822 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:20.656783 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:20.750109 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:20.750075 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:41.517321 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:41.517277 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:41.618813 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:41.618777 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:42.227791 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:42.227751 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:42.996268 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:42.996219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:46.232232 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:46.232199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:46.284319 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:46.284286 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:47.540159 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:47.540122 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:47.657931 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:47.657887 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:48.430133 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:48.430094 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:48.512745 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:48.512716 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:50.646185 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:50.646154 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:50.826571 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:50.826536 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:52.962655 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:52.962618 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:53.063119 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:53.063086 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:53.163110 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:53.163081 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:55.889372 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:55.889337 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:55.990091 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:55.990060 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:58.838792 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:58.838753 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:49:58.939109 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:49:58.939073 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:05.060769 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:05.060730 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:05.161396 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:05.161349 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:07.250682 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:07.250601 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:14.043291 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:14.043252 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:24.043008 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:24.042963 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:24.156179 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:24.156139 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:24.655926 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:24.655886 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:24.845545 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:24.845510 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:24.958634 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:24.958498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:25.021373 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.021333 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn"] Jul 17 05:50:25.023962 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.023940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.026572 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.026552 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 05:50:25.026847 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.026830 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 05:50:25.026948 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.026848 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 05:50:25.027208 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.027193 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xnd5q\"" Jul 17 05:50:25.027251 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.027194 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 05:50:25.040464 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.040444 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn"] Jul 17 05:50:25.061223 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.061191 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4f0798af-769e-4f03-a93b-e020ef58711e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.061685 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.061237 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4f0798af-769e-4f03-a93b-e020ef58711e-manifests\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.061685 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.061265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8sl9\" (UniqueName: \"kubernetes.io/projected/4f0798af-769e-4f03-a93b-e020ef58711e-kube-api-access-q8sl9\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.061685 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.061311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f0798af-769e-4f03-a93b-e020ef58711e-tmp\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.061685 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.061341 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f0798af-769e-4f03-a93b-e020ef58711e-webhook-cert\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.162112 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.162074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f0798af-769e-4f03-a93b-e020ef58711e-tmp\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.162304 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.162123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f0798af-769e-4f03-a93b-e020ef58711e-webhook-cert\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.162304 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.162163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4f0798af-769e-4f03-a93b-e020ef58711e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.162304 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.162205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4f0798af-769e-4f03-a93b-e020ef58711e-manifests\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.162304 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.162220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8sl9\" (UniqueName: \"kubernetes.io/projected/4f0798af-769e-4f03-a93b-e020ef58711e-kube-api-access-q8sl9\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.162634 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.162608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4f0798af-769e-4f03-a93b-e020ef58711e-manifests\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.164560 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.164534 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f0798af-769e-4f03-a93b-e020ef58711e-tmp\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.164671 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.164641 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f0798af-769e-4f03-a93b-e020ef58711e-webhook-cert\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.164749 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.164730 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4f0798af-769e-4f03-a93b-e020ef58711e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.172810 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.172783 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8sl9\" (UniqueName: \"kubernetes.io/projected/4f0798af-769e-4f03-a93b-e020ef58711e-kube-api-access-q8sl9\") pod \"opendatahub-operator-controller-manager-554df894d5-bklwn\" (UID: \"4f0798af-769e-4f03-a93b-e020ef58711e\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.173105 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.173082 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:25.273887 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.273804 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:25.333905 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.333871 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:25.375491 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.375442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:25.454233 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.454202 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:25.464670 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.464643 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn"] Jul 17 05:50:25.471168 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:50:25.471137 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0798af_769e_4f03_a93b_e020ef58711e.slice/crio-41a585dea22f8ac841f94bf1cba543cd7626383a59d7c3b241cc5ba82d10f6ea WatchSource:0}: Error finding container 41a585dea22f8ac841f94bf1cba543cd7626383a59d7c3b241cc5ba82d10f6ea: Status 404 returned error can't find the container with id 41a585dea22f8ac841f94bf1cba543cd7626383a59d7c3b241cc5ba82d10f6ea Jul 17 05:50:25.516818 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.516784 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:25.618588 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.618553 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:25.718810 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.718778 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:25.807152 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.807118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" event={"ID":"4f0798af-769e-4f03-a93b-e020ef58711e","Type":"ContainerStarted","Data":"41a585dea22f8ac841f94bf1cba543cd7626383a59d7c3b241cc5ba82d10f6ea"} Jul 17 05:50:25.818708 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:25.818683 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:28.820728 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:28.820681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" event={"ID":"4f0798af-769e-4f03-a93b-e020ef58711e","Type":"ContainerStarted","Data":"e4529d6c1c1227376f2ac982ddd4ed4a9d7178039566f0bb48a57b73b19b8d17"} Jul 17 05:50:29.826070 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:29.825972 2580 generic.go:356] "Generic (PLEG): container finished" podID="4f0798af-769e-4f03-a93b-e020ef58711e" containerID="e4529d6c1c1227376f2ac982ddd4ed4a9d7178039566f0bb48a57b73b19b8d17" exitCode=0 Jul 17 05:50:29.826544 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:29.826066 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" event={"ID":"4f0798af-769e-4f03-a93b-e020ef58711e","Type":"ContainerDied","Data":"e4529d6c1c1227376f2ac982ddd4ed4a9d7178039566f0bb48a57b73b19b8d17"} Jul 17 05:50:30.831296 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:30.831264 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" event={"ID":"4f0798af-769e-4f03-a93b-e020ef58711e","Type":"ContainerStarted","Data":"4ea2fb0ec4cf7e5ff8645d8c8548dc4f3a348196b3b4c78b0ba38837412a4ae8"} Jul 17 05:50:30.831716 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:30.831364 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:30.854305 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:30.854253 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" podStartSLOduration=2.690932769 podStartE2EDuration="6.854236518s" totalImagesPullingTime="3.107641198s" totalInitContainerRuntime="1.055662551s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:50:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:50:30.853365864 +0000 UTC m=+555.598045192" watchObservedRunningTime="2026-07-17 05:50:30.854236518 +0000 UTC m=+555.598915845" Jul 17 05:50:32.241923 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:32.241891 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:32.363652 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:32.363608 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:37.051951 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:37.051916 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:37.152916 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:37.152883 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:41.837359 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:41.837325 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-bklwn" Jul 17 05:50:43.928777 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:43.928729 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:44.070583 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:44.070544 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:44.818029 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:44.817990 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:44.918948 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:44.918909 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:45.022052 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:45.022021 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:45.122077 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:45.122045 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:45.267024 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:45.266981 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:45.347328 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:45.347289 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:45.411109 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:45.411017 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:45.682428 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:45.682325 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:49.925762 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:49.925729 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:50.199948 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:50.199850 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:51.081489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:51.081448 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:51.196544 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:51.196507 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:52.148966 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.148924 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx"] Jul 17 05:50:52.155626 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.155590 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:52.158144 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.158114 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 05:50:52.158323 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.158151 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 05:50:52.158828 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.158810 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 05:50:52.159061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.159046 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xsjtl\"" Jul 17 05:50:52.159275 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.159253 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 05:50:52.159344 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.159282 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 05:50:52.162828 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.162798 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx"] Jul 17 05:50:52.289784 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.289745 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9010838-12a1-4913-bf38-cb360367770f-cert\") pod \"lws-controller-manager-d9bdd98d9-rt4lx\" (UID: \"f9010838-12a1-4913-bf38-cb360367770f\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:52.289976 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.289803 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f9010838-12a1-4913-bf38-cb360367770f-manager-config\") pod \"lws-controller-manager-d9bdd98d9-rt4lx\" (UID: \"f9010838-12a1-4913-bf38-cb360367770f\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:52.289976 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.289870 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9010838-12a1-4913-bf38-cb360367770f-metrics-cert\") pod \"lws-controller-manager-d9bdd98d9-rt4lx\" (UID: \"f9010838-12a1-4913-bf38-cb360367770f\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:52.289976 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.289910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rphf4\" (UniqueName: \"kubernetes.io/projected/f9010838-12a1-4913-bf38-cb360367770f-kube-api-access-rphf4\") pod \"lws-controller-manager-d9bdd98d9-rt4lx\" (UID: \"f9010838-12a1-4913-bf38-cb360367770f\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:52.390780 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.390738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9010838-12a1-4913-bf38-cb360367770f-cert\") pod \"lws-controller-manager-d9bdd98d9-rt4lx\" (UID: \"f9010838-12a1-4913-bf38-cb360367770f\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:52.391018 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.390802 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f9010838-12a1-4913-bf38-cb360367770f-manager-config\") pod \"lws-controller-manager-d9bdd98d9-rt4lx\" (UID: \"f9010838-12a1-4913-bf38-cb360367770f\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:52.391018 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.390831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9010838-12a1-4913-bf38-cb360367770f-metrics-cert\") pod \"lws-controller-manager-d9bdd98d9-rt4lx\" (UID: \"f9010838-12a1-4913-bf38-cb360367770f\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:52.391018 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.390850 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rphf4\" (UniqueName: \"kubernetes.io/projected/f9010838-12a1-4913-bf38-cb360367770f-kube-api-access-rphf4\") pod \"lws-controller-manager-d9bdd98d9-rt4lx\" (UID: \"f9010838-12a1-4913-bf38-cb360367770f\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:52.391629 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.391601 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f9010838-12a1-4913-bf38-cb360367770f-manager-config\") pod \"lws-controller-manager-d9bdd98d9-rt4lx\" (UID: \"f9010838-12a1-4913-bf38-cb360367770f\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:52.393620 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.393592 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9010838-12a1-4913-bf38-cb360367770f-cert\") pod \"lws-controller-manager-d9bdd98d9-rt4lx\" (UID: \"f9010838-12a1-4913-bf38-cb360367770f\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:52.393717 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.393686 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9010838-12a1-4913-bf38-cb360367770f-metrics-cert\") pod \"lws-controller-manager-d9bdd98d9-rt4lx\" (UID: \"f9010838-12a1-4913-bf38-cb360367770f\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:52.399509 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.399457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rphf4\" (UniqueName: \"kubernetes.io/projected/f9010838-12a1-4913-bf38-cb360367770f-kube-api-access-rphf4\") pod \"lws-controller-manager-d9bdd98d9-rt4lx\" (UID: \"f9010838-12a1-4913-bf38-cb360367770f\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:52.468460 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.468420 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:52.476508 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.474814 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2"] Jul 17 05:50:52.477778 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.477752 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.480316 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.480294 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 17 05:50:52.480645 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.480297 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 17 05:50:52.480753 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.480297 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 17 05:50:52.480753 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.480412 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fcvqb\"" Jul 17 05:50:52.480863 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.480462 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 17 05:50:52.480863 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.480465 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 05:50:52.481189 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.481174 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 05:50:52.489778 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.489754 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2"] Jul 17 05:50:52.527084 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.527054 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:52.592375 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.592340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a34069-bd84-4eeb-bce4-5469f2236571-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.592539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.592378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9a34069-bd84-4eeb-bce4-5469f2236571-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.592539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.592441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9a34069-bd84-4eeb-bce4-5469f2236571-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.592539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.592489 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c9a34069-bd84-4eeb-bce4-5469f2236571-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.592539 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.592519 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c9a34069-bd84-4eeb-bce4-5469f2236571-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.592670 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.592582 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c9a34069-bd84-4eeb-bce4-5469f2236571-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.592670 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.592627 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6f9b\" (UniqueName: \"kubernetes.io/projected/c9a34069-bd84-4eeb-bce4-5469f2236571-kube-api-access-x6f9b\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.599208 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.599181 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx"] Jul 17 05:50:52.602354 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:50:52.602329 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9010838_12a1_4913_bf38_cb360367770f.slice/crio-a200ea0a44f51d574653fe035865c1411f67f16bc5465fbdbd346f7aa6a9ac66 WatchSource:0}: Error finding container a200ea0a44f51d574653fe035865c1411f67f16bc5465fbdbd346f7aa6a9ac66: Status 404 returned error can't find the container with id a200ea0a44f51d574653fe035865c1411f67f16bc5465fbdbd346f7aa6a9ac66 Jul 17 05:50:52.693712 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.693596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9a34069-bd84-4eeb-bce4-5469f2236571-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.693712 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.693645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c9a34069-bd84-4eeb-bce4-5469f2236571-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.693712 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.693669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c9a34069-bd84-4eeb-bce4-5469f2236571-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.693712 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.693712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c9a34069-bd84-4eeb-bce4-5469f2236571-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.694038 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.693757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6f9b\" (UniqueName: \"kubernetes.io/projected/c9a34069-bd84-4eeb-bce4-5469f2236571-kube-api-access-x6f9b\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.694038 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.693810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a34069-bd84-4eeb-bce4-5469f2236571-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.694038 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.693839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9a34069-bd84-4eeb-bce4-5469f2236571-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.694592 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.694566 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c9a34069-bd84-4eeb-bce4-5469f2236571-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.696061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.696036 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a34069-bd84-4eeb-bce4-5469f2236571-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.696254 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.696233 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c9a34069-bd84-4eeb-bce4-5469f2236571-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.696292 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.696254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c9a34069-bd84-4eeb-bce4-5469f2236571-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.696327 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.696301 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9a34069-bd84-4eeb-bce4-5469f2236571-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.701499 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.701481 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9a34069-bd84-4eeb-bce4-5469f2236571-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.702003 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.701971 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6f9b\" (UniqueName: \"kubernetes.io/projected/c9a34069-bd84-4eeb-bce4-5469f2236571-kube-api-access-x6f9b\") pod \"istiod-openshift-gateway-94bb8fbfd-nmbk2\" (UID: \"c9a34069-bd84-4eeb-bce4-5469f2236571\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.800364 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.800328 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:52.917644 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.917613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" event={"ID":"f9010838-12a1-4913-bf38-cb360367770f","Type":"ContainerStarted","Data":"a200ea0a44f51d574653fe035865c1411f67f16bc5465fbdbd346f7aa6a9ac66"} Jul 17 05:50:52.943206 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:52.943162 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2"] Jul 17 05:50:53.926180 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:53.926092 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" event={"ID":"c9a34069-bd84-4eeb-bce4-5469f2236571","Type":"ContainerStarted","Data":"c9c947ed7bb5f9df8a595f55ef4499fa8599e7678718cd90a260647b1fe74a0c"} Jul 17 05:50:54.437489 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:54.437405 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:54.607744 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:54.607620 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:54.800469 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:54.800357 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:54.905852 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:54.905816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:54.960704 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:54.960669 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:55.220616 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:55.220577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:55.301263 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:55.301228 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:55.836313 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:55.836276 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:55.921350 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:55.921316 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:56.022727 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:56.022689 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:56.182399 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:56.182349 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:56.382866 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:56.382819 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 17 05:50:56.383005 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:56.382914 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 17 05:50:56.387829 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:56.387801 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:56.504510 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:56.504473 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:50:56.939135 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:56.939090 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" event={"ID":"f9010838-12a1-4913-bf38-cb360367770f","Type":"ContainerStarted","Data":"1508507ce138430add555294f820d9cdf4ce4e98b813c815f80f82762ccac00e"} Jul 17 05:50:56.939344 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:56.939211 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:50:56.940739 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:56.940714 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" event={"ID":"c9a34069-bd84-4eeb-bce4-5469f2236571","Type":"ContainerStarted","Data":"a2fece324dcd30294564bb2be6d54af996cb2df54b2436ca2ba64f11c5c8a067"} Jul 17 05:50:56.940869 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:56.940815 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:50:56.956287 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:56.956239 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" podStartSLOduration=1.182349047 podStartE2EDuration="4.956224129s" totalImagesPullingTime="3.773875082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:50:56.955276471 +0000 UTC m=+581.699955800" watchObservedRunningTime="2026-07-17 05:50:56.956224129 +0000 UTC m=+581.700903457" Jul 17 05:50:56.975721 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:56.975657 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" podStartSLOduration=1.537998397 podStartE2EDuration="4.97564102s" totalImagesPullingTime="3.437642623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:50:56.974263464 +0000 UTC m=+581.718942821" watchObservedRunningTime="2026-07-17 05:50:56.97564102 +0000 UTC m=+581.720320347" Jul 17 05:50:57.946981 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:50:57.946951 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nmbk2" Jul 17 05:51:07.947260 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:51:07.947229 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-rt4lx" Jul 17 05:51:45.454025 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:51:45.453932 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:51:46.612745 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:51:46.612705 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:14.380950 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:14.380910 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:17.773732 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:17.773690 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:18.569195 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:18.569158 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:21.619832 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:21.619798 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:21.716210 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:21.716171 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:21.823183 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:21.823144 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:24.241888 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:24.241854 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:26.018127 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:26.018087 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:27.192877 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:27.192845 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:29.237546 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:29.237513 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:30.800168 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:30.800126 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:30.887141 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:30.887092 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:34.771584 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:34.771545 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:34.871397 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:34.871359 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:35.171848 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:35.171798 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:35.274574 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:35.274536 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:35.658643 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:35.658613 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:35.941922 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:35.941841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:39.193111 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:39.193070 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:39.292736 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:39.292695 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:45.745146 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:45.745107 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:45.843263 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:45.843233 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:47.233572 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:47.233537 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:47.336521 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:47.336484 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:47.434253 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:47.434216 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:47.544299 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:47.544206 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:50.026109 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:50.026074 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:50.126207 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:50.126168 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:55.424350 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:55.424318 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:55.477182 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:55.477147 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:55.828368 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:55.828282 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:55.898813 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:55.898780 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:59.423432 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:59.423366 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:52:59.534403 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:52:59.534351 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:01.101216 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:01.101176 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:01.201841 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:01.201807 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:03.047919 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:03.047882 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:05.293913 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:05.293883 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:05.393133 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:05.393091 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:19.164299 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:19.164219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:19.213200 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:19.213161 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:26.066247 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:26.066215 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:26.407565 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:26.407531 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:41.961819 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:41.961784 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:42.067822 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.067754 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:42.171067 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.171033 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:42.349566 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.349477 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 05:53:42.352785 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.352762 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-lxk5j" Jul 17 05:53:42.355100 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.355080 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 17 05:53:42.355207 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.355135 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 05:53:42.355261 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.355228 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 05:53:42.355804 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.355784 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cksw5\"" Jul 17 05:53:42.360763 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.360740 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 05:53:42.371805 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.371769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q52zz\" (UniqueName: \"kubernetes.io/projected/77e7d293-2e85-4011-a049-7ae1d8cb7d9e-kube-api-access-q52zz\") pod \"limitador-limitador-54d6d988d5-lxk5j\" (UID: \"77e7d293-2e85-4011-a049-7ae1d8cb7d9e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lxk5j" Jul 17 05:53:42.371938 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.371867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/77e7d293-2e85-4011-a049-7ae1d8cb7d9e-config-file\") pod \"limitador-limitador-54d6d988d5-lxk5j\" (UID: \"77e7d293-2e85-4011-a049-7ae1d8cb7d9e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lxk5j" Jul 17 05:53:42.384687 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.384655 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 05:53:42.472514 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.472477 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/77e7d293-2e85-4011-a049-7ae1d8cb7d9e-config-file\") pod \"limitador-limitador-54d6d988d5-lxk5j\" (UID: \"77e7d293-2e85-4011-a049-7ae1d8cb7d9e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lxk5j" Jul 17 05:53:42.472705 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.472548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q52zz\" (UniqueName: \"kubernetes.io/projected/77e7d293-2e85-4011-a049-7ae1d8cb7d9e-kube-api-access-q52zz\") pod \"limitador-limitador-54d6d988d5-lxk5j\" (UID: \"77e7d293-2e85-4011-a049-7ae1d8cb7d9e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lxk5j" Jul 17 05:53:42.473185 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.473162 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/77e7d293-2e85-4011-a049-7ae1d8cb7d9e-config-file\") pod \"limitador-limitador-54d6d988d5-lxk5j\" (UID: \"77e7d293-2e85-4011-a049-7ae1d8cb7d9e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lxk5j" Jul 17 05:53:42.484336 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.484301 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q52zz\" (UniqueName: \"kubernetes.io/projected/77e7d293-2e85-4011-a049-7ae1d8cb7d9e-kube-api-access-q52zz\") pod \"limitador-limitador-54d6d988d5-lxk5j\" (UID: \"77e7d293-2e85-4011-a049-7ae1d8cb7d9e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lxk5j" Jul 17 05:53:42.663645 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.663599 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-lxk5j" Jul 17 05:53:42.795253 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.795205 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:42.795640 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.795608 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 05:53:42.798677 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:53:42.798644 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e7d293_2e85_4011_a049_7ae1d8cb7d9e.slice/crio-4890e061104b567dd26d994f063b544d2f3362d9cd2703f13360a206e3c082d0 WatchSource:0}: Error finding container 4890e061104b567dd26d994f063b544d2f3362d9cd2703f13360a206e3c082d0: Status 404 returned error can't find the container with id 4890e061104b567dd26d994f063b544d2f3362d9cd2703f13360a206e3c082d0 Jul 17 05:53:42.800634 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.800617 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 05:53:42.889408 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:42.889350 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:43.581822 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:43.581747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-lxk5j" event={"ID":"77e7d293-2e85-4011-a049-7ae1d8cb7d9e","Type":"ContainerStarted","Data":"4890e061104b567dd26d994f063b544d2f3362d9cd2703f13360a206e3c082d0"} Jul 17 05:53:45.592468 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:45.592432 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-lxk5j" event={"ID":"77e7d293-2e85-4011-a049-7ae1d8cb7d9e","Type":"ContainerStarted","Data":"6156c75ebf85a857c37faad41633332d7ed5e8110bfad71e03a7351f8ed1203d"} Jul 17 05:53:45.592895 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:45.592515 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-lxk5j" Jul 17 05:53:45.607228 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:45.607175 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-lxk5j" podStartSLOduration=1.040802787 podStartE2EDuration="3.607158888s" totalImagesPullingTime="2.566356101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:53:45.605993948 +0000 UTC m=+750.350673275" watchObservedRunningTime="2026-07-17 05:53:45.607158888 +0000 UTC m=+750.351838219" Jul 17 05:53:56.602908 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:56.602874 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-lxk5j" Jul 17 05:53:59.388428 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:59.388362 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:59.484610 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:59.484567 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:53:59.837059 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:53:59.836972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:54:11.841218 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:54:11.841178 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:54:11.987573 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:54:11.987546 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:54:18.780054 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:54:18.780018 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:54:18.837014 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:54:18.836972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:54:19.053694 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:54:19.053599 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:54:22.492618 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:54:22.492581 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:54:22.589647 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:54:22.589609 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:54:55.119445 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:54:55.119341 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:54:55.215966 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:54:55.215928 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:54:55.631175 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:54:55.631145 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:54:55.749679 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:54:55.749644 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:54:56.088408 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:54:56.088297 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:54:56.167176 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:54:56.167140 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:54:56.283349 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:54:56.283309 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 05:55:01.115807 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:01.115776 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:55:01.217763 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:01.217725 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:55:05.841051 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:05.841010 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:55:05.938731 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:05.938693 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:55:23.367424 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.367372 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj"] Jul 17 05:55:23.370598 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.370574 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.372975 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.372951 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 05:55:23.373104 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.372975 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 05:55:23.373104 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.372999 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 17 05:55:23.373104 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.373030 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8bkch\"" Jul 17 05:55:23.379670 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.379645 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj"] Jul 17 05:55:23.393198 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.393166 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:55:23.450680 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.450641 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8a37b28-4186-4b97-80fb-73dfd1342ebe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.450840 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.450687 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a37b28-4186-4b97-80fb-73dfd1342ebe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.450840 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.450707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a37b28-4186-4b97-80fb-73dfd1342ebe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.450951 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.450857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw68z\" (UniqueName: \"kubernetes.io/projected/f8a37b28-4186-4b97-80fb-73dfd1342ebe-kube-api-access-bw68z\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.450951 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.450921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a37b28-4186-4b97-80fb-73dfd1342ebe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.451048 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.450957 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8a37b28-4186-4b97-80fb-73dfd1342ebe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.552010 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.551954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw68z\" (UniqueName: \"kubernetes.io/projected/f8a37b28-4186-4b97-80fb-73dfd1342ebe-kube-api-access-bw68z\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.552010 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.552020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a37b28-4186-4b97-80fb-73dfd1342ebe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.552299 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.552047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8a37b28-4186-4b97-80fb-73dfd1342ebe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.552299 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.552078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8a37b28-4186-4b97-80fb-73dfd1342ebe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.552299 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.552101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a37b28-4186-4b97-80fb-73dfd1342ebe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.552299 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.552118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a37b28-4186-4b97-80fb-73dfd1342ebe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.552561 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.552535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a37b28-4186-4b97-80fb-73dfd1342ebe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.552625 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.552580 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8a37b28-4186-4b97-80fb-73dfd1342ebe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.552625 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.552599 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a37b28-4186-4b97-80fb-73dfd1342ebe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.554428 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.554411 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8a37b28-4186-4b97-80fb-73dfd1342ebe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.554731 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.554714 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a37b28-4186-4b97-80fb-73dfd1342ebe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.559061 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.559042 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw68z\" (UniqueName: \"kubernetes.io/projected/f8a37b28-4186-4b97-80fb-73dfd1342ebe-kube-api-access-bw68z\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj\" (UID: \"f8a37b28-4186-4b97-80fb-73dfd1342ebe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.681683 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.681587 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:23.772133 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.772099 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 05:55:23.814725 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:23.814696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj"] Jul 17 05:55:23.816924 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:55:23.816899 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a37b28_4186_4b97_80fb_73dfd1342ebe.slice/crio-6531e78c8e43ffc6f9afdcace2a0fe45621388abfadc7bae9dc4702c86d6dabf WatchSource:0}: Error finding container 6531e78c8e43ffc6f9afdcace2a0fe45621388abfadc7bae9dc4702c86d6dabf: Status 404 returned error can't find the container with id 6531e78c8e43ffc6f9afdcace2a0fe45621388abfadc7bae9dc4702c86d6dabf Jul 17 05:55:24.017379 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:24.017289 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" event={"ID":"f8a37b28-4186-4b97-80fb-73dfd1342ebe","Type":"ContainerStarted","Data":"6531e78c8e43ffc6f9afdcace2a0fe45621388abfadc7bae9dc4702c86d6dabf"} Jul 17 05:55:25.023445 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:25.023408 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" event={"ID":"f8a37b28-4186-4b97-80fb-73dfd1342ebe","Type":"ContainerStarted","Data":"6d306d07a7fdb3c350fc8317227f106949d38ded63938c67e5533bfc6f22b04a"} Jul 17 05:55:25.042547 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:25.042491 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" podStartSLOduration=0.939054993 podStartE2EDuration="2.042474611s" totalImagesPullingTime="1.103419618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:55:25.04084004 +0000 UTC m=+849.785519380" watchObservedRunningTime="2026-07-17 05:55:25.042474611 +0000 UTC m=+849.787153982" Jul 17 05:55:33.681970 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:33.681931 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:33.682617 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:33.682003 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:33.694591 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:33.694565 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:34.074180 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:34.074088 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj" Jul 17 05:55:42.906053 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:42.905998 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:55:43.780220 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:55:43.780180 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 05:56:05.309697 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:05.309655 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:56:06.677018 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:06.676980 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 05:56:17.803880 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:17.803841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:56:19.381282 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:19.381243 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 05:56:44.876901 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:44.876859 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g"] Jul 17 05:56:44.880250 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:44.880227 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:44.882687 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:44.882664 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 17 05:56:44.890000 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:44.889958 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g"] Jul 17 05:56:44.917004 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:44.916969 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:56:44.965182 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:44.965142 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1230e234-d26f-4d28-a830-9e850c2c1ed7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:44.965340 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:44.965186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z75z6\" (UniqueName: \"kubernetes.io/projected/1230e234-d26f-4d28-a830-9e850c2c1ed7-kube-api-access-z75z6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:44.965340 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:44.965220 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1230e234-d26f-4d28-a830-9e850c2c1ed7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:44.965340 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:44.965260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1230e234-d26f-4d28-a830-9e850c2c1ed7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:44.965340 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:44.965312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1230e234-d26f-4d28-a830-9e850c2c1ed7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:44.965503 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:44.965409 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1230e234-d26f-4d28-a830-9e850c2c1ed7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:45.066046 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.065999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1230e234-d26f-4d28-a830-9e850c2c1ed7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:45.066046 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.066051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z75z6\" (UniqueName: \"kubernetes.io/projected/1230e234-d26f-4d28-a830-9e850c2c1ed7-kube-api-access-z75z6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:45.066294 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.066076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1230e234-d26f-4d28-a830-9e850c2c1ed7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:45.066294 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.066107 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1230e234-d26f-4d28-a830-9e850c2c1ed7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:45.066294 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.066128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1230e234-d26f-4d28-a830-9e850c2c1ed7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:45.066294 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.066156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1230e234-d26f-4d28-a830-9e850c2c1ed7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:45.066534 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.066462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1230e234-d26f-4d28-a830-9e850c2c1ed7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:45.066597 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.066568 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1230e234-d26f-4d28-a830-9e850c2c1ed7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:45.066651 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.066614 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1230e234-d26f-4d28-a830-9e850c2c1ed7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:45.068568 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.068541 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1230e234-d26f-4d28-a830-9e850c2c1ed7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:45.068800 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.068782 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1230e234-d26f-4d28-a830-9e850c2c1ed7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:45.073288 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.073268 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z75z6\" (UniqueName: \"kubernetes.io/projected/1230e234-d26f-4d28-a830-9e850c2c1ed7-kube-api-access-z75z6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xmx2g\" (UID: \"1230e234-d26f-4d28-a830-9e850c2c1ed7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:45.191469 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.191332 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:45.324403 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.324361 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g"] Jul 17 05:56:45.327750 ip-10-0-142-12 kubenswrapper[2580]: W0717 05:56:45.327717 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1230e234_d26f_4d28_a830_9e850c2c1ed7.slice/crio-b5ac6cf28e73afff731f05da4667f1eb304574eea83707d4d6bec41adf820a40 WatchSource:0}: Error finding container b5ac6cf28e73afff731f05da4667f1eb304574eea83707d4d6bec41adf820a40: Status 404 returned error can't find the container with id b5ac6cf28e73afff731f05da4667f1eb304574eea83707d4d6bec41adf820a40 Jul 17 05:56:45.374154 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:45.374114 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" event={"ID":"1230e234-d26f-4d28-a830-9e850c2c1ed7","Type":"ContainerStarted","Data":"b5ac6cf28e73afff731f05da4667f1eb304574eea83707d4d6bec41adf820a40"} Jul 17 05:56:46.380840 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:46.380804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" event={"ID":"1230e234-d26f-4d28-a830-9e850c2c1ed7","Type":"ContainerStarted","Data":"626dbfdd3817c28fa28b12a42a67624706e780e8864e4eea329bf1b64ec533bf"} Jul 17 05:56:46.397479 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:46.397425 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" podStartSLOduration=2.177051484 podStartE2EDuration="2.397407356s" totalImagesPullingTime="220.355872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:56:46.395754966 +0000 UTC m=+931.140434313" watchObservedRunningTime="2026-07-17 05:56:46.397407356 +0000 UTC m=+931.142086675" Jul 17 05:56:46.783144 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:46.783042 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 05:56:49.803858 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:49.803809 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:56:55.191647 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:55.191609 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:55.192156 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:55.191713 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:55.204289 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:55.204258 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:55.432718 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:55.432682 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xmx2g" Jul 17 05:56:57.577661 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:56:57.577627 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 05:57:06.961508 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:57:06.961462 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:57:07.664881 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:57:07.664846 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:58:05.395001 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:58:05.394962 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:58:05.447073 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:58:05.447039 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:58:09.848431 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:58:09.848375 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:58:09.916235 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:58:09.916199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:58:10.897580 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:58:10.897548 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:58:10.997012 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:58:10.996957 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:58:11.521515 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:58:11.521479 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:58:38.073897 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:58:38.073851 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:58:38.082537 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:58:38.082505 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 05:58:47.276474 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:58:47.276431 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 05:58:58.082758 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:58:58.082715 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 05:59:09.146703 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:59:09.146628 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 05:59:09.246740 ip-10-0-142-12 kubenswrapper[2580]: I0717 05:59:09.246711 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:00:14.289629 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:00:14.289597 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:00:15.561424 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:00:15.561369 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:00:38.273112 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:00:38.273033 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:00:47.713090 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:00:47.713053 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:00:49.014263 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:00:49.014231 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:00:50.270873 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:00:50.270836 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:00:54.517477 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:00:54.517442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:00:54.617243 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:00:54.617209 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:00:54.748295 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:00:54.748259 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:01:00.426124 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:01:00.426083 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:01:00.526522 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:01:00.526475 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:01:00.626436 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:01:00.626362 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:01:00.770831 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:01:00.770732 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:01:06.886184 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:01:06.886136 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:01:06.937615 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:01:06.937579 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:01:22.781744 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:01:22.781706 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:01:29.775588 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:01:29.775553 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:01:40.274032 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:01:40.273997 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:01:45.979433 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:01:45.979398 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:01:56.445670 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:01:56.445630 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:01:56.680544 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:01:56.680509 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:02:05.575593 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:02:05.575561 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:02:17.379583 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:02:17.379497 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:02:24.569724 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:02:24.569687 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:02:31.580010 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:02:31.579969 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:02:41.382659 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:02:41.382617 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:02:59.078528 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:02:59.078483 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:03:09.670870 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:03:09.670831 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:03:32.081082 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:03:32.081039 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:03:43.785168 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:03:43.785091 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:04:00.084347 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:04:00.084309 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:04:09.678828 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:04:09.678791 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:04:28.576029 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:04:28.575992 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:04:39.875471 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:04:39.875430 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:06:10.207045 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:06:10.207010 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:06:10.291490 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:06:10.291460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:06:10.463117 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:06:10.463036 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:06:27.385766 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:06:27.385727 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:06:36.081312 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:06:36.081276 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:06:48.074914 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:06:48.074868 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:06:57.480873 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:06:57.480833 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:07:06.784662 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:07:06.784606 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:07:15.479976 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:07:15.479937 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:07:25.782366 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:07:25.782327 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:07:26.516300 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:07:26.516270 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:07:37.381189 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:07:37.381145 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:07:46.376695 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:07:46.376655 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:07:56.472053 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:07:56.471990 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:08:08.680156 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:08:08.680070 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:08:17.787499 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:08:17.787465 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:08:34.480879 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:08:34.480845 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:08:43.687204 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:08:43.687159 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:08:53.272695 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:08:53.272657 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:09:03.680651 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:09:03.680609 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:09:13.580604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:09:13.580558 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:09:25.372656 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:09:25.372614 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:09:34.280702 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:09:34.280664 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:09:43.676816 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:09:43.676729 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:09:55.379434 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:09:55.379380 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:10:03.978021 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:10:03.977981 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:10:13.379219 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:10:13.379181 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:10:26.086904 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:10:26.086861 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:10:34.983604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:10:34.983570 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:10:45.678758 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:10:45.678721 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:10:55.975276 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:10:55.975235 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:11:06.677707 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:11:06.677665 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:11:13.384775 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:11:13.384746 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:11:31.110334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:11:31.110300 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:11:42.590692 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:11:42.590658 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:11:42.659439 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:11:42.659405 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:11:43.290673 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:11:43.290635 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:11:57.814044 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:11:57.814007 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:12:01.085665 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:01.085629 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:12:01.248665 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:01.248629 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:12:01.797346 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:01.797296 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:12:08.728025 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:08.727989 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:12:08.833033 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:08.832999 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:12:09.012874 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:09.012793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:12:09.108941 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:09.108888 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:12:10.262829 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.262782 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:12:10.404028 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.403991 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq"] Jul 17 06:12:10.407914 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.407890 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:10.413943 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.413918 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpsw\"" Jul 17 06:12:10.414115 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.413947 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 06:12:10.414115 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.413917 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0c567a84-serving-cert\"" Jul 17 06:12:10.415029 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.414716 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 06:12:10.448971 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.448917 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:12:10.450348 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.450323 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq"] Jul 17 06:12:10.493149 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.493112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f4gp\" (UniqueName: \"kubernetes.io/projected/4e99ee18-cda8-4f07-8450-747f2e0367a6-kube-api-access-7f4gp\") pod \"maas-api-e2e-ait-0c567a84-9966bb597-zmbfq\" (UID: \"4e99ee18-cda8-4f07-8450-747f2e0367a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:10.493361 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.493209 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e99ee18-cda8-4f07-8450-747f2e0367a6-maas-api-tls\") pod \"maas-api-e2e-ait-0c567a84-9966bb597-zmbfq\" (UID: \"4e99ee18-cda8-4f07-8450-747f2e0367a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:10.493361 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.493272 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e99ee18-cda8-4f07-8450-747f2e0367a6-openshift-service-ca\") pod \"maas-api-e2e-ait-0c567a84-9966bb597-zmbfq\" (UID: \"4e99ee18-cda8-4f07-8450-747f2e0367a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:10.594126 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.594031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e99ee18-cda8-4f07-8450-747f2e0367a6-maas-api-tls\") pod \"maas-api-e2e-ait-0c567a84-9966bb597-zmbfq\" (UID: \"4e99ee18-cda8-4f07-8450-747f2e0367a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:10.594126 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.594073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e99ee18-cda8-4f07-8450-747f2e0367a6-openshift-service-ca\") pod \"maas-api-e2e-ait-0c567a84-9966bb597-zmbfq\" (UID: \"4e99ee18-cda8-4f07-8450-747f2e0367a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:10.594126 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.594117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f4gp\" (UniqueName: \"kubernetes.io/projected/4e99ee18-cda8-4f07-8450-747f2e0367a6-kube-api-access-7f4gp\") pod \"maas-api-e2e-ait-0c567a84-9966bb597-zmbfq\" (UID: \"4e99ee18-cda8-4f07-8450-747f2e0367a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:10.594456 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:12:10.594196 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-serving-cert: secret "maas-api-e2e-ait-0c567a84-serving-cert" not found Jul 17 06:12:10.594456 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:12:10.594278 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e99ee18-cda8-4f07-8450-747f2e0367a6-maas-api-tls podName:4e99ee18-cda8-4f07-8450-747f2e0367a6 nodeName:}" failed. No retries permitted until 2026-07-17 06:12:11.094259829 +0000 UTC m=+1855.838939135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e99ee18-cda8-4f07-8450-747f2e0367a6-maas-api-tls") pod "maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" (UID: "4e99ee18-cda8-4f07-8450-747f2e0367a6") : secret "maas-api-e2e-ait-0c567a84-serving-cert" not found Jul 17 06:12:10.594889 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.594863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e99ee18-cda8-4f07-8450-747f2e0367a6-openshift-service-ca\") pod \"maas-api-e2e-ait-0c567a84-9966bb597-zmbfq\" (UID: \"4e99ee18-cda8-4f07-8450-747f2e0367a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:10.612056 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.612028 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f4gp\" (UniqueName: \"kubernetes.io/projected/4e99ee18-cda8-4f07-8450-747f2e0367a6-kube-api-access-7f4gp\") pod \"maas-api-e2e-ait-0c567a84-9966bb597-zmbfq\" (UID: \"4e99ee18-cda8-4f07-8450-747f2e0367a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:10.648806 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:10.648761 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:12:11.099325 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:11.099282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e99ee18-cda8-4f07-8450-747f2e0367a6-maas-api-tls\") pod \"maas-api-e2e-ait-0c567a84-9966bb597-zmbfq\" (UID: \"4e99ee18-cda8-4f07-8450-747f2e0367a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:11.102315 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:11.102287 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e99ee18-cda8-4f07-8450-747f2e0367a6-maas-api-tls\") pod \"maas-api-e2e-ait-0c567a84-9966bb597-zmbfq\" (UID: \"4e99ee18-cda8-4f07-8450-747f2e0367a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:11.155516 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:11.155477 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:12:11.318580 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:11.318535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:11.482357 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:11.482320 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq"] Jul 17 06:12:11.486828 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:12:11.486800 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e99ee18_cda8_4f07_8450_747f2e0367a6.slice/crio-5dd5bc80664fa9191712f28659572c189af67fc018b8df5f3b6d41af0acea26c WatchSource:0}: Error finding container 5dd5bc80664fa9191712f28659572c189af67fc018b8df5f3b6d41af0acea26c: Status 404 returned error can't find the container with id 5dd5bc80664fa9191712f28659572c189af67fc018b8df5f3b6d41af0acea26c Jul 17 06:12:11.488599 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:11.488579 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 06:12:11.647656 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:11.647571 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:12:11.829491 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:11.829461 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:12:12.411123 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:12.411067 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" event={"ID":"4e99ee18-cda8-4f07-8450-747f2e0367a6","Type":"ContainerStarted","Data":"5dd5bc80664fa9191712f28659572c189af67fc018b8df5f3b6d41af0acea26c"} Jul 17 06:12:13.113165 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:13.113122 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:12:14.422968 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:14.422933 2580 generic.go:356] "Generic (PLEG): container finished" podID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerID="5b0e253f45ab56a6ed9d318d3d2a779497e6c2c3cc7fc59318ae7943a2b6acda" exitCode=1 Jul 17 06:12:14.423456 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:14.423007 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" event={"ID":"4e99ee18-cda8-4f07-8450-747f2e0367a6","Type":"ContainerDied","Data":"5b0e253f45ab56a6ed9d318d3d2a779497e6c2c3cc7fc59318ae7943a2b6acda"} Jul 17 06:12:14.423456 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:14.423327 2580 scope.go:117] "RemoveContainer" containerID="5b0e253f45ab56a6ed9d318d3d2a779497e6c2c3cc7fc59318ae7943a2b6acda" Jul 17 06:12:15.429082 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:15.428985 2580 generic.go:356] "Generic (PLEG): container finished" podID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerID="b316e5a0abb279f316d63f9000ddd65148f1999021c3f4f0ad43606067c19f56" exitCode=1 Jul 17 06:12:15.429082 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:15.429066 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" event={"ID":"4e99ee18-cda8-4f07-8450-747f2e0367a6","Type":"ContainerDied","Data":"b316e5a0abb279f316d63f9000ddd65148f1999021c3f4f0ad43606067c19f56"} Jul 17 06:12:15.429894 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:15.429127 2580 scope.go:117] "RemoveContainer" containerID="5b0e253f45ab56a6ed9d318d3d2a779497e6c2c3cc7fc59318ae7943a2b6acda" Jul 17 06:12:15.429894 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:15.429436 2580 scope.go:117] "RemoveContainer" containerID="b316e5a0abb279f316d63f9000ddd65148f1999021c3f4f0ad43606067c19f56" Jul 17 06:12:15.429894 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:12:15.429644 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:12:16.318920 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:16.318885 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:16.435074 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:16.435049 2580 scope.go:117] "RemoveContainer" containerID="b316e5a0abb279f316d63f9000ddd65148f1999021c3f4f0ad43606067c19f56" Jul 17 06:12:16.435463 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:12:16.435221 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:12:17.439890 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:17.439861 2580 scope.go:117] "RemoveContainer" containerID="b316e5a0abb279f316d63f9000ddd65148f1999021c3f4f0ad43606067c19f56" Jul 17 06:12:17.440275 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:12:17.440086 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:12:17.794886 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:17.794801 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:12:21.319033 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:21.318987 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:21.319549 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:21.319367 2580 scope.go:117] "RemoveContainer" containerID="b316e5a0abb279f316d63f9000ddd65148f1999021c3f4f0ad43606067c19f56" Jul 17 06:12:21.319593 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:12:21.319576 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:12:31.883846 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:31.883815 2580 scope.go:117] "RemoveContainer" containerID="b316e5a0abb279f316d63f9000ddd65148f1999021c3f4f0ad43606067c19f56" Jul 17 06:12:32.507229 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:32.507139 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" event={"ID":"4e99ee18-cda8-4f07-8450-747f2e0367a6","Type":"ContainerStarted","Data":"1eb8573cde1e007d1707b19ba8a301986f00cd5327923a7555c48bd0b68f2657"} Jul 17 06:12:32.507413 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:32.507379 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:32.531154 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:32.531102 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podStartSLOduration=19.328417025 podStartE2EDuration="22.531087699s" totalImagesPullingTime="3.202670674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:12:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:12:32.530598404 +0000 UTC m=+1877.275277750" watchObservedRunningTime="2026-07-17 06:12:32.531087699 +0000 UTC m=+1877.275767005" Jul 17 06:12:33.512786 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:33.512750 2580 generic.go:356] "Generic (PLEG): container finished" podID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerID="1eb8573cde1e007d1707b19ba8a301986f00cd5327923a7555c48bd0b68f2657" exitCode=1 Jul 17 06:12:33.513226 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:33.512835 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" event={"ID":"4e99ee18-cda8-4f07-8450-747f2e0367a6","Type":"ContainerDied","Data":"1eb8573cde1e007d1707b19ba8a301986f00cd5327923a7555c48bd0b68f2657"} Jul 17 06:12:33.513226 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:33.512897 2580 scope.go:117] "RemoveContainer" containerID="b316e5a0abb279f316d63f9000ddd65148f1999021c3f4f0ad43606067c19f56" Jul 17 06:12:33.513226 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:33.513084 2580 scope.go:117] "RemoveContainer" containerID="1eb8573cde1e007d1707b19ba8a301986f00cd5327923a7555c48bd0b68f2657" Jul 17 06:12:33.513339 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:12:33.513287 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:12:34.519234 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:34.519205 2580 scope.go:117] "RemoveContainer" containerID="1eb8573cde1e007d1707b19ba8a301986f00cd5327923a7555c48bd0b68f2657" Jul 17 06:12:34.519658 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:12:34.519427 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:12:41.319279 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:41.319190 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:12:41.319682 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:41.319630 2580 scope.go:117] "RemoveContainer" containerID="1eb8573cde1e007d1707b19ba8a301986f00cd5327923a7555c48bd0b68f2657" Jul 17 06:12:41.319836 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:12:41.319814 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:12:56.886293 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:56.886258 2580 scope.go:117] "RemoveContainer" containerID="1eb8573cde1e007d1707b19ba8a301986f00cd5327923a7555c48bd0b68f2657" Jul 17 06:12:57.624210 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:57.624177 2580 generic.go:356] "Generic (PLEG): container finished" podID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerID="782bb89c1348a400348757722d30e36f497183214a64f8abcf883ea4484e5fe3" exitCode=1 Jul 17 06:12:57.624373 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:57.624228 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" event={"ID":"4e99ee18-cda8-4f07-8450-747f2e0367a6","Type":"ContainerDied","Data":"782bb89c1348a400348757722d30e36f497183214a64f8abcf883ea4484e5fe3"} Jul 17 06:12:57.624373 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:57.624308 2580 scope.go:117] "RemoveContainer" containerID="1eb8573cde1e007d1707b19ba8a301986f00cd5327923a7555c48bd0b68f2657" Jul 17 06:12:57.624722 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:12:57.624700 2580 scope.go:117] "RemoveContainer" containerID="782bb89c1348a400348757722d30e36f497183214a64f8abcf883ea4484e5fe3" Jul 17 06:12:57.624942 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:12:57.624914 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:13:01.319601 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:01.319557 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:13:01.320045 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:01.319975 2580 scope.go:117] "RemoveContainer" containerID="782bb89c1348a400348757722d30e36f497183214a64f8abcf883ea4484e5fe3" Jul 17 06:13:01.320158 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:13:01.320138 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:13:02.508117 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:02.508081 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:13:02.508549 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:02.508516 2580 scope.go:117] "RemoveContainer" containerID="782bb89c1348a400348757722d30e36f497183214a64f8abcf883ea4484e5fe3" Jul 17 06:13:02.508725 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:13:02.508703 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:13:14.889121 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:14.889085 2580 scope.go:117] "RemoveContainer" containerID="782bb89c1348a400348757722d30e36f497183214a64f8abcf883ea4484e5fe3" Jul 17 06:13:14.889543 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:13:14.889249 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:13:28.884805 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:28.884773 2580 scope.go:117] "RemoveContainer" containerID="782bb89c1348a400348757722d30e36f497183214a64f8abcf883ea4484e5fe3" Jul 17 06:13:28.885306 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:13:28.884967 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:13:42.884144 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:42.884116 2580 scope.go:117] "RemoveContainer" containerID="782bb89c1348a400348757722d30e36f497183214a64f8abcf883ea4484e5fe3" Jul 17 06:13:43.831487 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:43.831455 2580 generic.go:356] "Generic (PLEG): container finished" podID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerID="2c01b0a96016898c47da161185d77e4fc3898d89e07d210155bcc32d7f8ce665" exitCode=1 Jul 17 06:13:43.831676 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:43.831507 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" event={"ID":"4e99ee18-cda8-4f07-8450-747f2e0367a6","Type":"ContainerDied","Data":"2c01b0a96016898c47da161185d77e4fc3898d89e07d210155bcc32d7f8ce665"} Jul 17 06:13:43.831676 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:43.831556 2580 scope.go:117] "RemoveContainer" containerID="782bb89c1348a400348757722d30e36f497183214a64f8abcf883ea4484e5fe3" Jul 17 06:13:43.831927 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:43.831900 2580 scope.go:117] "RemoveContainer" containerID="2c01b0a96016898c47da161185d77e4fc3898d89e07d210155bcc32d7f8ce665" Jul 17 06:13:43.832140 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:13:43.832120 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:13:47.508242 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:47.508200 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:13:47.508709 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:47.508656 2580 scope.go:117] "RemoveContainer" containerID="2c01b0a96016898c47da161185d77e4fc3898d89e07d210155bcc32d7f8ce665" Jul 17 06:13:47.508870 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:13:47.508848 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:13:51.319425 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:51.319371 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:13:51.319823 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:13:51.319796 2580 scope.go:117] "RemoveContainer" containerID="2c01b0a96016898c47da161185d77e4fc3898d89e07d210155bcc32d7f8ce665" Jul 17 06:13:51.319979 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:13:51.319959 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:14:02.884687 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:14:02.884653 2580 scope.go:117] "RemoveContainer" containerID="2c01b0a96016898c47da161185d77e4fc3898d89e07d210155bcc32d7f8ce665" Jul 17 06:14:02.885172 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:14:02.884819 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:14:14.884159 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:14:14.884041 2580 scope.go:117] "RemoveContainer" containerID="2c01b0a96016898c47da161185d77e4fc3898d89e07d210155bcc32d7f8ce665" Jul 17 06:14:14.884750 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:14:14.884301 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:14:25.884515 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:14:25.884464 2580 scope.go:117] "RemoveContainer" containerID="2c01b0a96016898c47da161185d77e4fc3898d89e07d210155bcc32d7f8ce665" Jul 17 06:14:25.884915 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:14:25.884691 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:14:37.883991 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:14:37.883951 2580 scope.go:117] "RemoveContainer" containerID="2c01b0a96016898c47da161185d77e4fc3898d89e07d210155bcc32d7f8ce665" Jul 17 06:14:37.884516 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:14:37.884156 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:14:50.890029 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:14:50.889998 2580 scope.go:117] "RemoveContainer" containerID="2c01b0a96016898c47da161185d77e4fc3898d89e07d210155bcc32d7f8ce665" Jul 17 06:14:50.890429 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:14:50.890195 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:15:04.884104 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:04.884059 2580 scope.go:117] "RemoveContainer" containerID="2c01b0a96016898c47da161185d77e4fc3898d89e07d210155bcc32d7f8ce665" Jul 17 06:15:06.191308 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:06.191278 2580 generic.go:356] "Generic (PLEG): container finished" podID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerID="9e64ea3186671967fe47ae8560cda4205851083effd98d7cc56a9b85af6e9021" exitCode=1 Jul 17 06:15:06.191780 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:06.191358 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" event={"ID":"4e99ee18-cda8-4f07-8450-747f2e0367a6","Type":"ContainerDied","Data":"9e64ea3186671967fe47ae8560cda4205851083effd98d7cc56a9b85af6e9021"} Jul 17 06:15:06.191780 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:06.191438 2580 scope.go:117] "RemoveContainer" containerID="2c01b0a96016898c47da161185d77e4fc3898d89e07d210155bcc32d7f8ce665" Jul 17 06:15:06.191780 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:06.191726 2580 scope.go:117] "RemoveContainer" containerID="9e64ea3186671967fe47ae8560cda4205851083effd98d7cc56a9b85af6e9021" Jul 17 06:15:06.191982 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:15:06.191958 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:15:07.508056 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:07.507998 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:15:07.508606 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:07.508581 2580 scope.go:117] "RemoveContainer" containerID="9e64ea3186671967fe47ae8560cda4205851083effd98d7cc56a9b85af6e9021" Jul 17 06:15:07.508814 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:15:07.508788 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:15:11.319218 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:11.319181 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:15:11.319774 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:11.319583 2580 scope.go:117] "RemoveContainer" containerID="9e64ea3186671967fe47ae8560cda4205851083effd98d7cc56a9b85af6e9021" Jul 17 06:15:11.319825 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:15:11.319767 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-0c567a84-9966bb597-zmbfq_odh-ai-gateway-infra(4e99ee18-cda8-4f07-8450-747f2e0367a6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" Jul 17 06:15:15.356711 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:15.356673 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:17.446273 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:17.446234 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:17.876348 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:17.876312 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:15:20.815491 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:20.815455 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq"] Jul 17 06:15:20.870699 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:20.870668 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:20.945908 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:20.945885 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:15:21.105315 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.105286 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f4gp\" (UniqueName: \"kubernetes.io/projected/4e99ee18-cda8-4f07-8450-747f2e0367a6-kube-api-access-7f4gp\") pod \"4e99ee18-cda8-4f07-8450-747f2e0367a6\" (UID: \"4e99ee18-cda8-4f07-8450-747f2e0367a6\") " Jul 17 06:15:21.105515 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.105495 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e99ee18-cda8-4f07-8450-747f2e0367a6-openshift-service-ca\") pod \"4e99ee18-cda8-4f07-8450-747f2e0367a6\" (UID: \"4e99ee18-cda8-4f07-8450-747f2e0367a6\") " Jul 17 06:15:21.105577 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.105562 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e99ee18-cda8-4f07-8450-747f2e0367a6-maas-api-tls\") pod \"4e99ee18-cda8-4f07-8450-747f2e0367a6\" (UID: \"4e99ee18-cda8-4f07-8450-747f2e0367a6\") " Jul 17 06:15:21.105826 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.105796 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e99ee18-cda8-4f07-8450-747f2e0367a6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e99ee18-cda8-4f07-8450-747f2e0367a6" (UID: "4e99ee18-cda8-4f07-8450-747f2e0367a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:15:21.107437 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.107410 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e99ee18-cda8-4f07-8450-747f2e0367a6-kube-api-access-7f4gp" (OuterVolumeSpecName: "kube-api-access-7f4gp") pod "4e99ee18-cda8-4f07-8450-747f2e0367a6" (UID: "4e99ee18-cda8-4f07-8450-747f2e0367a6"). InnerVolumeSpecName "kube-api-access-7f4gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:15:21.107536 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.107488 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e99ee18-cda8-4f07-8450-747f2e0367a6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e99ee18-cda8-4f07-8450-747f2e0367a6" (UID: "4e99ee18-cda8-4f07-8450-747f2e0367a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:15:21.207116 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.207028 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e99ee18-cda8-4f07-8450-747f2e0367a6-maas-api-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:15:21.207116 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.207058 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f4gp\" (UniqueName: \"kubernetes.io/projected/4e99ee18-cda8-4f07-8450-747f2e0367a6-kube-api-access-7f4gp\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:15:21.207116 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.207073 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e99ee18-cda8-4f07-8450-747f2e0367a6-openshift-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:15:21.259108 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.259078 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" Jul 17 06:15:21.259287 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.259098 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq" event={"ID":"4e99ee18-cda8-4f07-8450-747f2e0367a6","Type":"ContainerDied","Data":"5dd5bc80664fa9191712f28659572c189af67fc018b8df5f3b6d41af0acea26c"} Jul 17 06:15:21.259287 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.259173 2580 scope.go:117] "RemoveContainer" containerID="9e64ea3186671967fe47ae8560cda4205851083effd98d7cc56a9b85af6e9021" Jul 17 06:15:21.281285 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.281261 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq"] Jul 17 06:15:21.287495 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.287468 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0c567a84-9966bb597-zmbfq"] Jul 17 06:15:21.459811 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:21.459721 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:22.889326 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:22.889289 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" path="/var/lib/kubelet/pods/4e99ee18-cda8-4f07-8450-747f2e0367a6/volumes" Jul 17 06:15:26.984145 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:26.984111 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:27.042273 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:27.042244 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:27.240311 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:27.240199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:27.339666 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:27.339632 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:27.439597 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:27.439553 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:28.752985 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.752951 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:28.865471 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.865432 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th"] Jul 17 06:15:28.865933 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.865919 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.865976 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.865936 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.865976 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.865951 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.865976 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.865959 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.866077 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.865978 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.866077 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.865987 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.866077 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.865999 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.866077 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.866006 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.866077 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.866015 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.866077 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.866024 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.866253 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.866093 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.866253 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.866103 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.866253 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.866113 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.866253 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.866121 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.866253 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.866133 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:15:28.868824 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.868805 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:28.872586 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.872556 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpsw\"" Jul 17 06:15:28.872722 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.872566 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 06:15:28.872722 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.872612 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a9646996-serving-cert\"" Jul 17 06:15:28.872722 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.872568 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 06:15:28.882737 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.882714 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th"] Jul 17 06:15:28.913588 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.913554 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:28.975619 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.975578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-maas-api-tls\") pod \"maas-api-e2e-migrate-a9646996-7449579b88-zn2th\" (UID: \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:28.975619 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.975621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsl6t\" (UniqueName: \"kubernetes.io/projected/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-kube-api-access-gsl6t\") pod \"maas-api-e2e-migrate-a9646996-7449579b88-zn2th\" (UID: \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:28.975936 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:28.975909 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-openshift-service-ca\") pod \"maas-api-e2e-migrate-a9646996-7449579b88-zn2th\" (UID: \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:29.077128 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:29.077002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-openshift-service-ca\") pod \"maas-api-e2e-migrate-a9646996-7449579b88-zn2th\" (UID: \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:29.077361 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:29.077152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-maas-api-tls\") pod \"maas-api-e2e-migrate-a9646996-7449579b88-zn2th\" (UID: \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:29.077361 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:29.077186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsl6t\" (UniqueName: \"kubernetes.io/projected/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-kube-api-access-gsl6t\") pod \"maas-api-e2e-migrate-a9646996-7449579b88-zn2th\" (UID: \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:29.077504 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:15:29.077354 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-serving-cert: secret "maas-api-e2e-migrate-a9646996-serving-cert" not found Jul 17 06:15:29.077504 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:15:29.077456 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-maas-api-tls podName:9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2 nodeName:}" failed. No retries permitted until 2026-07-17 06:15:29.57743492 +0000 UTC m=+2054.322114226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-maas-api-tls") pod "maas-api-e2e-migrate-a9646996-7449579b88-zn2th" (UID: "9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2") : secret "maas-api-e2e-migrate-a9646996-serving-cert" not found Jul 17 06:15:29.077884 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:29.077863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-openshift-service-ca\") pod \"maas-api-e2e-migrate-a9646996-7449579b88-zn2th\" (UID: \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:29.087929 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:29.087892 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsl6t\" (UniqueName: \"kubernetes.io/projected/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-kube-api-access-gsl6t\") pod \"maas-api-e2e-migrate-a9646996-7449579b88-zn2th\" (UID: \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:29.368422 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:29.368368 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:29.467176 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:29.467139 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:29.581678 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:29.581640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-maas-api-tls\") pod \"maas-api-e2e-migrate-a9646996-7449579b88-zn2th\" (UID: \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:29.584104 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:29.584082 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-maas-api-tls\") pod \"maas-api-e2e-migrate-a9646996-7449579b88-zn2th\" (UID: \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:29.779885 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:29.779780 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:29.913155 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:29.913123 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th"] Jul 17 06:15:29.916791 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:15:29.916760 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a18db76_fcc0_4e64_a1bb_cf8dbc5fccc2.slice/crio-c4cf5ac97d147885f21b0304da2ebc86ddbe90a8a3cb2adc02c7c1b246c85008 WatchSource:0}: Error finding container c4cf5ac97d147885f21b0304da2ebc86ddbe90a8a3cb2adc02c7c1b246c85008: Status 404 returned error can't find the container with id c4cf5ac97d147885f21b0304da2ebc86ddbe90a8a3cb2adc02c7c1b246c85008 Jul 17 06:15:29.979428 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:29.979374 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:30.298767 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:30.298731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" event={"ID":"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2","Type":"ContainerStarted","Data":"c4cf5ac97d147885f21b0304da2ebc86ddbe90a8a3cb2adc02c7c1b246c85008"} Jul 17 06:15:31.279925 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:31.279886 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:15:31.304850 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:31.304818 2580 generic.go:356] "Generic (PLEG): container finished" podID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerID="dd05f64dd4e585846618ba74f58987eaab211d425ebf2503f279f2c4909885e9" exitCode=1 Jul 17 06:15:31.305027 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:31.304896 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" event={"ID":"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2","Type":"ContainerDied","Data":"dd05f64dd4e585846618ba74f58987eaab211d425ebf2503f279f2c4909885e9"} Jul 17 06:15:31.305114 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:31.305096 2580 scope.go:117] "RemoveContainer" containerID="dd05f64dd4e585846618ba74f58987eaab211d425ebf2503f279f2c4909885e9" Jul 17 06:15:31.953529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:31.953479 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:32.016792 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:32.016750 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:32.312015 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:32.311931 2580 generic.go:356] "Generic (PLEG): container finished" podID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerID="cf71fa1b19dd05148e0e5f29b2c224331b9f379e4a3bdf8e32e6514097406001" exitCode=1 Jul 17 06:15:32.312015 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:32.312012 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" event={"ID":"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2","Type":"ContainerDied","Data":"cf71fa1b19dd05148e0e5f29b2c224331b9f379e4a3bdf8e32e6514097406001"} Jul 17 06:15:32.312513 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:32.312052 2580 scope.go:117] "RemoveContainer" containerID="dd05f64dd4e585846618ba74f58987eaab211d425ebf2503f279f2c4909885e9" Jul 17 06:15:32.312513 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:32.312346 2580 scope.go:117] "RemoveContainer" containerID="cf71fa1b19dd05148e0e5f29b2c224331b9f379e4a3bdf8e32e6514097406001" Jul 17 06:15:32.312611 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:15:32.312590 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-a9646996-7449579b88-zn2th_odh-ai-gateway-infra(9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" Jul 17 06:15:33.318362 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:33.318331 2580 scope.go:117] "RemoveContainer" containerID="cf71fa1b19dd05148e0e5f29b2c224331b9f379e4a3bdf8e32e6514097406001" Jul 17 06:15:33.318791 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:15:33.318514 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-a9646996-7449579b88-zn2th_odh-ai-gateway-infra(9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" Jul 17 06:15:34.780026 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:34.779974 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:34.780477 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:34.780372 2580 scope.go:117] "RemoveContainer" containerID="cf71fa1b19dd05148e0e5f29b2c224331b9f379e4a3bdf8e32e6514097406001" Jul 17 06:15:34.780608 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:15:34.780586 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-a9646996-7449579b88-zn2th_odh-ai-gateway-infra(9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" Jul 17 06:15:35.410212 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:35.410178 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:36.325435 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:36.325377 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:36.426539 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:36.426504 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:36.885170 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:36.885127 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:36.978648 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:36.978618 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:39.780594 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:39.780557 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:39.781005 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:39.780984 2580 scope.go:117] "RemoveContainer" containerID="cf71fa1b19dd05148e0e5f29b2c224331b9f379e4a3bdf8e32e6514097406001" Jul 17 06:15:39.781187 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:15:39.781166 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-a9646996-7449579b88-zn2th_odh-ai-gateway-infra(9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" Jul 17 06:15:50.267268 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:50.267234 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:50.350897 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:50.350863 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:15:50.883829 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:50.883797 2580 scope.go:117] "RemoveContainer" containerID="cf71fa1b19dd05148e0e5f29b2c224331b9f379e4a3bdf8e32e6514097406001" Jul 17 06:15:51.401008 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:51.400965 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" event={"ID":"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2","Type":"ContainerStarted","Data":"e027873c88e9d662a97eed107b2a5fbff89fa8dcda4fda9e6d05d70814fd2909"} Jul 17 06:15:51.401406 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:51.401256 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:51.419537 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:51.419469 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" podStartSLOduration=22.269290841 podStartE2EDuration="23.419453504s" totalImagesPullingTime="1.150162663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:15:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:15:51.417835619 +0000 UTC m=+2076.162514958" watchObservedRunningTime="2026-07-17 06:15:51.419453504 +0000 UTC m=+2076.164132832" Jul 17 06:15:52.407712 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:52.407669 2580 generic.go:356] "Generic (PLEG): container finished" podID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerID="e027873c88e9d662a97eed107b2a5fbff89fa8dcda4fda9e6d05d70814fd2909" exitCode=1 Jul 17 06:15:52.408221 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:52.407758 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" event={"ID":"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2","Type":"ContainerDied","Data":"e027873c88e9d662a97eed107b2a5fbff89fa8dcda4fda9e6d05d70814fd2909"} Jul 17 06:15:52.408221 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:52.407817 2580 scope.go:117] "RemoveContainer" containerID="cf71fa1b19dd05148e0e5f29b2c224331b9f379e4a3bdf8e32e6514097406001" Jul 17 06:15:52.408221 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:52.408116 2580 scope.go:117] "RemoveContainer" containerID="e027873c88e9d662a97eed107b2a5fbff89fa8dcda4fda9e6d05d70814fd2909" Jul 17 06:15:52.408407 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:15:52.408336 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-a9646996-7449579b88-zn2th_odh-ai-gateway-infra(9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" Jul 17 06:15:53.414167 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:53.414138 2580 scope.go:117] "RemoveContainer" containerID="e027873c88e9d662a97eed107b2a5fbff89fa8dcda4fda9e6d05d70814fd2909" Jul 17 06:15:53.414571 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:15:53.414326 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-a9646996-7449579b88-zn2th_odh-ai-gateway-infra(9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" Jul 17 06:15:59.780649 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:59.780607 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:15:59.781065 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:15:59.780958 2580 scope.go:117] "RemoveContainer" containerID="e027873c88e9d662a97eed107b2a5fbff89fa8dcda4fda9e6d05d70814fd2909" Jul 17 06:15:59.781171 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:15:59.781150 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-a9646996-7449579b88-zn2th_odh-ai-gateway-infra(9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" Jul 17 06:16:12.884349 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:12.884315 2580 scope.go:117] "RemoveContainer" containerID="e027873c88e9d662a97eed107b2a5fbff89fa8dcda4fda9e6d05d70814fd2909" Jul 17 06:16:13.508063 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:13.508031 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" event={"ID":"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2","Type":"ContainerStarted","Data":"21c0bbba9262dcddf3d123850c56f4614eb7c0a1586cbda2b199913c4308d283"} Jul 17 06:16:13.508267 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:13.508248 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:16:14.514560 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:14.514524 2580 generic.go:356] "Generic (PLEG): container finished" podID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerID="21c0bbba9262dcddf3d123850c56f4614eb7c0a1586cbda2b199913c4308d283" exitCode=1 Jul 17 06:16:14.515049 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:14.514614 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" event={"ID":"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2","Type":"ContainerDied","Data":"21c0bbba9262dcddf3d123850c56f4614eb7c0a1586cbda2b199913c4308d283"} Jul 17 06:16:14.515049 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:14.514690 2580 scope.go:117] "RemoveContainer" containerID="e027873c88e9d662a97eed107b2a5fbff89fa8dcda4fda9e6d05d70814fd2909" Jul 17 06:16:14.515049 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:14.514906 2580 scope.go:117] "RemoveContainer" containerID="21c0bbba9262dcddf3d123850c56f4614eb7c0a1586cbda2b199913c4308d283" Jul 17 06:16:14.515178 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:16:14.515135 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-a9646996-7449579b88-zn2th_odh-ai-gateway-infra(9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" Jul 17 06:16:15.520444 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:15.520415 2580 scope.go:117] "RemoveContainer" containerID="21c0bbba9262dcddf3d123850c56f4614eb7c0a1586cbda2b199913c4308d283" Jul 17 06:16:15.520818 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:16:15.520597 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-a9646996-7449579b88-zn2th_odh-ai-gateway-infra(9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" Jul 17 06:16:19.780218 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:19.780159 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:16:19.780678 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:19.780638 2580 scope.go:117] "RemoveContainer" containerID="21c0bbba9262dcddf3d123850c56f4614eb7c0a1586cbda2b199913c4308d283" Jul 17 06:16:19.780840 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:16:19.780819 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-a9646996-7449579b88-zn2th_odh-ai-gateway-infra(9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" Jul 17 06:16:30.583972 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:30.583939 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:32.884840 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:32.884805 2580 scope.go:117] "RemoveContainer" containerID="21c0bbba9262dcddf3d123850c56f4614eb7c0a1586cbda2b199913c4308d283" Jul 17 06:16:32.885231 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:16:32.884982 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-a9646996-7449579b88-zn2th_odh-ai-gateway-infra(9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" Jul 17 06:16:33.336201 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:33.336100 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:33.438802 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:33.438770 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:35.890461 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:35.890429 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:16:38.461936 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.461890 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th"] Jul 17 06:16:38.532210 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.532177 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:38.597340 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.597310 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:16:38.624753 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.624712 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" event={"ID":"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2","Type":"ContainerDied","Data":"c4cf5ac97d147885f21b0304da2ebc86ddbe90a8a3cb2adc02c7c1b246c85008"} Jul 17 06:16:38.624938 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.624788 2580 scope.go:117] "RemoveContainer" containerID="21c0bbba9262dcddf3d123850c56f4614eb7c0a1586cbda2b199913c4308d283" Jul 17 06:16:38.624938 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.624732 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th" Jul 17 06:16:38.701075 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.701043 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-maas-api-tls\") pod \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\" (UID: \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\") " Jul 17 06:16:38.701235 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.701116 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsl6t\" (UniqueName: \"kubernetes.io/projected/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-kube-api-access-gsl6t\") pod \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\" (UID: \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\") " Jul 17 06:16:38.701235 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.701163 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-openshift-service-ca\") pod \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\" (UID: \"9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2\") " Jul 17 06:16:38.701628 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.701597 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" (UID: "9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:16:38.703156 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.703134 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" (UID: "9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:16:38.703259 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.703237 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-kube-api-access-gsl6t" (OuterVolumeSpecName: "kube-api-access-gsl6t") pod "9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" (UID: "9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2"). InnerVolumeSpecName "kube-api-access-gsl6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:16:38.802201 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.802102 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-maas-api-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:16:38.802201 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.802142 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsl6t\" (UniqueName: \"kubernetes.io/projected/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-kube-api-access-gsl6t\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:16:38.802201 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.802153 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2-openshift-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:16:38.940834 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.940802 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th"] Jul 17 06:16:38.943254 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:38.943231 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a9646996-7449579b88-zn2th"] Jul 17 06:16:39.292505 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:39.292468 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:40.004742 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:40.004703 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:40.889635 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:40.889601 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" path="/var/lib/kubelet/pods/9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2/volumes" Jul 17 06:16:45.042549 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.042380 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:45.141428 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.141360 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:45.263874 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.263837 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:45.318571 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.318486 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:45.385777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.385736 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s"] Jul 17 06:16:45.386215 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.386192 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerName="maas-api" Jul 17 06:16:45.386264 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.386222 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerName="maas-api" Jul 17 06:16:45.386264 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.386242 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:16:45.386264 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.386253 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:16:45.386357 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.386267 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerName="maas-api" Jul 17 06:16:45.386357 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.386275 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerName="maas-api" Jul 17 06:16:45.386357 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.386286 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerName="maas-api" Jul 17 06:16:45.386357 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.386294 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerName="maas-api" Jul 17 06:16:45.386518 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.386419 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerName="maas-api" Jul 17 06:16:45.386518 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.386436 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerName="maas-api" Jul 17 06:16:45.386518 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.386447 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e99ee18-cda8-4f07-8450-747f2e0367a6" containerName="maas-api" Jul 17 06:16:45.391082 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.391063 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" Jul 17 06:16:45.393457 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.393428 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 06:16:45.393638 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.393435 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:16:45.397105 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.397078 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s"] Jul 17 06:16:45.465838 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.465798 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58584594-7121-4bcc-bb4e-00668e8257ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s\" (UID: \"58584594-7121-4bcc-bb4e-00668e8257ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" Jul 17 06:16:45.566637 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.566606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58584594-7121-4bcc-bb4e-00668e8257ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s\" (UID: \"58584594-7121-4bcc-bb4e-00668e8257ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" Jul 17 06:16:45.567410 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.567357 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58584594-7121-4bcc-bb4e-00668e8257ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s\" (UID: \"58584594-7121-4bcc-bb4e-00668e8257ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" Jul 17 06:16:45.702310 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.702269 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" Jul 17 06:16:45.833244 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:45.833218 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s"] Jul 17 06:16:45.835484 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:16:45.835457 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58584594_7121_4bcc_bb4e_00668e8257ad.slice/crio-657c5927c168987f06624c5742e79f82dfede30640a1c4cfadaafbdc5556b986 WatchSource:0}: Error finding container 657c5927c168987f06624c5742e79f82dfede30640a1c4cfadaafbdc5556b986: Status 404 returned error can't find the container with id 657c5927c168987f06624c5742e79f82dfede30640a1c4cfadaafbdc5556b986 Jul 17 06:16:46.384577 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.384540 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:46.485642 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.485610 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm"] Jul 17 06:16:46.485955 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.485940 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerName="maas-api" Jul 17 06:16:46.485955 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.485953 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerName="maas-api" Jul 17 06:16:46.486104 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.486004 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerName="maas-api" Jul 17 06:16:46.489400 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.489356 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:46.490188 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.490168 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:16:46.492600 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.492583 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpsw\"" Jul 17 06:16:46.492961 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.492942 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-572ab047-serving-cert\"" Jul 17 06:16:46.493275 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.493259 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 06:16:46.502597 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.502575 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm"] Jul 17 06:16:46.562850 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.562812 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:46.576097 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.576064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65b4941-22fa-4d95-8c9d-43fc877c071e-maas-api-tls\") pod \"maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm\" (UID: \"a65b4941-22fa-4d95-8c9d-43fc877c071e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:16:46.576250 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.576099 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65b4941-22fa-4d95-8c9d-43fc877c071e-openshift-service-ca\") pod \"maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm\" (UID: \"a65b4941-22fa-4d95-8c9d-43fc877c071e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:16:46.576250 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.576196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nldwp\" (UniqueName: \"kubernetes.io/projected/a65b4941-22fa-4d95-8c9d-43fc877c071e-kube-api-access-nldwp\") pod \"maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm\" (UID: \"a65b4941-22fa-4d95-8c9d-43fc877c071e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:16:46.661891 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.661801 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" event={"ID":"58584594-7121-4bcc-bb4e-00668e8257ad","Type":"ContainerStarted","Data":"657c5927c168987f06624c5742e79f82dfede30640a1c4cfadaafbdc5556b986"} Jul 17 06:16:46.677238 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.677213 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nldwp\" (UniqueName: \"kubernetes.io/projected/a65b4941-22fa-4d95-8c9d-43fc877c071e-kube-api-access-nldwp\") pod \"maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm\" (UID: \"a65b4941-22fa-4d95-8c9d-43fc877c071e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:16:46.677318 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.677280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65b4941-22fa-4d95-8c9d-43fc877c071e-maas-api-tls\") pod \"maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm\" (UID: \"a65b4941-22fa-4d95-8c9d-43fc877c071e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:16:46.677318 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.677296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65b4941-22fa-4d95-8c9d-43fc877c071e-openshift-service-ca\") pod \"maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm\" (UID: \"a65b4941-22fa-4d95-8c9d-43fc877c071e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:16:46.677906 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.677884 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65b4941-22fa-4d95-8c9d-43fc877c071e-openshift-service-ca\") pod \"maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm\" (UID: \"a65b4941-22fa-4d95-8c9d-43fc877c071e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:16:46.679780 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.679760 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65b4941-22fa-4d95-8c9d-43fc877c071e-maas-api-tls\") pod \"maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm\" (UID: \"a65b4941-22fa-4d95-8c9d-43fc877c071e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:16:46.687233 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.687210 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nldwp\" (UniqueName: \"kubernetes.io/projected/a65b4941-22fa-4d95-8c9d-43fc877c071e-kube-api-access-nldwp\") pod \"maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm\" (UID: \"a65b4941-22fa-4d95-8c9d-43fc877c071e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:16:46.714604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.714577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:46.801898 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.801859 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:16:46.961793 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:46.961710 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm"] Jul 17 06:16:46.965739 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:16:46.965704 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65b4941_22fa_4d95_8c9d_43fc877c071e.slice/crio-c68c5f474215838367c7c46bd0144f325bc744444faf2650e73cb16ee3b6b0b4 WatchSource:0}: Error finding container c68c5f474215838367c7c46bd0144f325bc744444faf2650e73cb16ee3b6b0b4: Status 404 returned error can't find the container with id c68c5f474215838367c7c46bd0144f325bc744444faf2650e73cb16ee3b6b0b4 Jul 17 06:16:47.034750 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:47.034698 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:47.135330 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:47.135285 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:47.669905 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:47.669819 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" event={"ID":"a65b4941-22fa-4d95-8c9d-43fc877c071e","Type":"ContainerStarted","Data":"f3fd2a5905ee36e66c34006c1ef125798b90b4a79cd37b6512ff4d59cc9bf44a"} Jul 17 06:16:47.669905 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:47.669881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" event={"ID":"a65b4941-22fa-4d95-8c9d-43fc877c071e","Type":"ContainerStarted","Data":"c68c5f474215838367c7c46bd0144f325bc744444faf2650e73cb16ee3b6b0b4"} Jul 17 06:16:47.670459 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:47.669940 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:16:47.687602 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:47.687547 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podStartSLOduration=1.2900476140000001 podStartE2EDuration="1.687531482s" totalImagesPullingTime="397.483868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:16:47.685536189 +0000 UTC m=+2132.430215530" watchObservedRunningTime="2026-07-17 06:16:47.687531482 +0000 UTC m=+2132.432210812" Jul 17 06:16:48.675821 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:48.675719 2580 generic.go:356] "Generic (PLEG): container finished" podID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerID="f3fd2a5905ee36e66c34006c1ef125798b90b4a79cd37b6512ff4d59cc9bf44a" exitCode=1 Jul 17 06:16:48.675821 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:48.675806 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" event={"ID":"a65b4941-22fa-4d95-8c9d-43fc877c071e","Type":"ContainerDied","Data":"f3fd2a5905ee36e66c34006c1ef125798b90b4a79cd37b6512ff4d59cc9bf44a"} Jul 17 06:16:48.676322 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:48.676106 2580 scope.go:117] "RemoveContainer" containerID="f3fd2a5905ee36e66c34006c1ef125798b90b4a79cd37b6512ff4d59cc9bf44a" Jul 17 06:16:48.677580 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:48.677557 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" event={"ID":"58584594-7121-4bcc-bb4e-00668e8257ad","Type":"ContainerStarted","Data":"60b0f7893a3fd560dcf78e6947fc3d9b38bff3c333098f0a055e71cc707c72d3"} Jul 17 06:16:48.713259 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:48.713199 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" podStartSLOduration=1.398482684 podStartE2EDuration="3.71318166s" totalImagesPullingTime="2.314698976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:16:48.712346554 +0000 UTC m=+2133.457025883" watchObservedRunningTime="2026-07-17 06:16:48.71318166 +0000 UTC m=+2133.457861033" Jul 17 06:16:49.068787 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:49.068745 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:16:49.084303 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:49.084270 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:16:49.683752 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:49.683714 2580 generic.go:356] "Generic (PLEG): container finished" podID="58584594-7121-4bcc-bb4e-00668e8257ad" containerID="60b0f7893a3fd560dcf78e6947fc3d9b38bff3c333098f0a055e71cc707c72d3" exitCode=6 Jul 17 06:16:49.684230 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:49.683803 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" event={"ID":"58584594-7121-4bcc-bb4e-00668e8257ad","Type":"ContainerDied","Data":"60b0f7893a3fd560dcf78e6947fc3d9b38bff3c333098f0a055e71cc707c72d3"} Jul 17 06:16:49.684230 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:49.684021 2580 scope.go:117] "RemoveContainer" containerID="60b0f7893a3fd560dcf78e6947fc3d9b38bff3c333098f0a055e71cc707c72d3" Jul 17 06:16:49.685872 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:49.685845 2580 generic.go:356] "Generic (PLEG): container finished" podID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerID="135a0dc174cd9a9206e228416b55ce71debca86d46111bca2483d6a2c78089ed" exitCode=1 Jul 17 06:16:49.685978 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:49.685881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" event={"ID":"a65b4941-22fa-4d95-8c9d-43fc877c071e","Type":"ContainerDied","Data":"135a0dc174cd9a9206e228416b55ce71debca86d46111bca2483d6a2c78089ed"} Jul 17 06:16:49.685978 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:49.685932 2580 scope.go:117] "RemoveContainer" containerID="f3fd2a5905ee36e66c34006c1ef125798b90b4a79cd37b6512ff4d59cc9bf44a" Jul 17 06:16:49.686167 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:49.686152 2580 scope.go:117] "RemoveContainer" containerID="135a0dc174cd9a9206e228416b55ce71debca86d46111bca2483d6a2c78089ed" Jul 17 06:16:49.686359 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:16:49.686336 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:16:50.691893 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:50.691860 2580 scope.go:117] "RemoveContainer" containerID="135a0dc174cd9a9206e228416b55ce71debca86d46111bca2483d6a2c78089ed" Jul 17 06:16:50.692309 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:16:50.692094 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:16:50.693503 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:50.693473 2580 generic.go:356] "Generic (PLEG): container finished" podID="58584594-7121-4bcc-bb4e-00668e8257ad" containerID="740209447cda4864ffafcb5d2324895a8d5d1b865715c66fa807cfe281d0e413" exitCode=6 Jul 17 06:16:50.693629 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:50.693513 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" event={"ID":"58584594-7121-4bcc-bb4e-00668e8257ad","Type":"ContainerDied","Data":"740209447cda4864ffafcb5d2324895a8d5d1b865715c66fa807cfe281d0e413"} Jul 17 06:16:50.693629 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:50.693566 2580 scope.go:117] "RemoveContainer" containerID="60b0f7893a3fd560dcf78e6947fc3d9b38bff3c333098f0a055e71cc707c72d3" Jul 17 06:16:50.693754 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:50.693738 2580 scope.go:117] "RemoveContainer" containerID="740209447cda4864ffafcb5d2324895a8d5d1b865715c66fa807cfe281d0e413" Jul 17 06:16:50.693950 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:16:50.693932 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s_odh-ai-gateway-infra(58584594-7121-4bcc-bb4e-00668e8257ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" podUID="58584594-7121-4bcc-bb4e-00668e8257ad" Jul 17 06:16:51.701630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:51.701603 2580 scope.go:117] "RemoveContainer" containerID="740209447cda4864ffafcb5d2324895a8d5d1b865715c66fa807cfe281d0e413" Jul 17 06:16:51.702112 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:16:51.701854 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s_odh-ai-gateway-infra(58584594-7121-4bcc-bb4e-00668e8257ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" podUID="58584594-7121-4bcc-bb4e-00668e8257ad" Jul 17 06:16:52.670397 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:52.670362 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:16:52.670799 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:52.670781 2580 scope.go:117] "RemoveContainer" containerID="135a0dc174cd9a9206e228416b55ce71debca86d46111bca2483d6a2c78089ed" Jul 17 06:16:52.671003 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:16:52.670979 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:16:56.802827 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:56.802781 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:16:56.803236 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:16:56.803210 2580 scope.go:117] "RemoveContainer" containerID="135a0dc174cd9a9206e228416b55ce71debca86d46111bca2483d6a2c78089ed" Jul 17 06:16:56.803430 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:16:56.803369 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:17:04.884323 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:04.884282 2580 scope.go:117] "RemoveContainer" containerID="740209447cda4864ffafcb5d2324895a8d5d1b865715c66fa807cfe281d0e413" Jul 17 06:17:05.762162 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:05.762078 2580 generic.go:356] "Generic (PLEG): container finished" podID="58584594-7121-4bcc-bb4e-00668e8257ad" containerID="58e6f05c4bbb0b932d3029869d4acab205df58ec9c6c229c3ee59810481d80f5" exitCode=6 Jul 17 06:17:05.762309 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:05.762151 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" event={"ID":"58584594-7121-4bcc-bb4e-00668e8257ad","Type":"ContainerDied","Data":"58e6f05c4bbb0b932d3029869d4acab205df58ec9c6c229c3ee59810481d80f5"} Jul 17 06:17:05.762309 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:05.762238 2580 scope.go:117] "RemoveContainer" containerID="740209447cda4864ffafcb5d2324895a8d5d1b865715c66fa807cfe281d0e413" Jul 17 06:17:05.762531 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:05.762516 2580 scope.go:117] "RemoveContainer" containerID="58e6f05c4bbb0b932d3029869d4acab205df58ec9c6c229c3ee59810481d80f5" Jul 17 06:17:05.762769 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:05.762749 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s_odh-ai-gateway-infra(58584594-7121-4bcc-bb4e-00668e8257ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" podUID="58584594-7121-4bcc-bb4e-00668e8257ad" Jul 17 06:17:06.785975 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:06.785936 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s"] Jul 17 06:17:06.926150 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:06.926106 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" Jul 17 06:17:07.063146 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:07.063044 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58584594-7121-4bcc-bb4e-00668e8257ad-maas-api-service-ca\") pod \"58584594-7121-4bcc-bb4e-00668e8257ad\" (UID: \"58584594-7121-4bcc-bb4e-00668e8257ad\") " Jul 17 06:17:07.063527 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:07.063498 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58584594-7121-4bcc-bb4e-00668e8257ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "58584594-7121-4bcc-bb4e-00668e8257ad" (UID: "58584594-7121-4bcc-bb4e-00668e8257ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:17:07.164459 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:07.164350 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58584594-7121-4bcc-bb4e-00668e8257ad-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:17:07.774265 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:07.774229 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" event={"ID":"58584594-7121-4bcc-bb4e-00668e8257ad","Type":"ContainerDied","Data":"657c5927c168987f06624c5742e79f82dfede30640a1c4cfadaafbdc5556b986"} Jul 17 06:17:07.774497 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:07.774293 2580 scope.go:117] "RemoveContainer" containerID="58e6f05c4bbb0b932d3029869d4acab205df58ec9c6c229c3ee59810481d80f5" Jul 17 06:17:07.774497 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:07.774243 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s" Jul 17 06:17:07.797018 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:07.796982 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s"] Jul 17 06:17:07.802030 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:07.801999 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-mbz2s"] Jul 17 06:17:07.884668 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:07.884636 2580 scope.go:117] "RemoveContainer" containerID="135a0dc174cd9a9206e228416b55ce71debca86d46111bca2483d6a2c78089ed" Jul 17 06:17:08.781917 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:08.781881 2580 generic.go:356] "Generic (PLEG): container finished" podID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerID="ab8f6dce8cd62182c46604e13b9b96e0cc717f1e0dd16134e507888103e07d91" exitCode=1 Jul 17 06:17:08.782114 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:08.781959 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" event={"ID":"a65b4941-22fa-4d95-8c9d-43fc877c071e","Type":"ContainerDied","Data":"ab8f6dce8cd62182c46604e13b9b96e0cc717f1e0dd16134e507888103e07d91"} Jul 17 06:17:08.782114 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:08.782031 2580 scope.go:117] "RemoveContainer" containerID="135a0dc174cd9a9206e228416b55ce71debca86d46111bca2483d6a2c78089ed" Jul 17 06:17:08.782418 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:08.782378 2580 scope.go:117] "RemoveContainer" containerID="ab8f6dce8cd62182c46604e13b9b96e0cc717f1e0dd16134e507888103e07d91" Jul 17 06:17:08.782673 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:08.782652 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:17:08.889223 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:08.889191 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58584594-7121-4bcc-bb4e-00668e8257ad" path="/var/lib/kubelet/pods/58584594-7121-4bcc-bb4e-00668e8257ad/volumes" Jul 17 06:17:12.670180 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:12.670140 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:17:12.670590 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:12.670564 2580 scope.go:117] "RemoveContainer" containerID="ab8f6dce8cd62182c46604e13b9b96e0cc717f1e0dd16134e507888103e07d91" Jul 17 06:17:12.670769 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:12.670746 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:17:16.802126 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:16.802029 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:17:16.802542 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:16.802486 2580 scope.go:117] "RemoveContainer" containerID="ab8f6dce8cd62182c46604e13b9b96e0cc717f1e0dd16134e507888103e07d91" Jul 17 06:17:16.802707 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:16.802686 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:17:30.883907 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:30.883871 2580 scope.go:117] "RemoveContainer" containerID="ab8f6dce8cd62182c46604e13b9b96e0cc717f1e0dd16134e507888103e07d91" Jul 17 06:17:30.884446 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:30.884168 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 06:17:31.888887 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:31.888853 2580 generic.go:356] "Generic (PLEG): container finished" podID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerID="dbbd382df0c915e2a42ba14b7371e1f46d5d4c1f92d986fc9afb541e3d7214b4" exitCode=1 Jul 17 06:17:31.889454 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:31.888931 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" event={"ID":"a65b4941-22fa-4d95-8c9d-43fc877c071e","Type":"ContainerDied","Data":"dbbd382df0c915e2a42ba14b7371e1f46d5d4c1f92d986fc9afb541e3d7214b4"} Jul 17 06:17:31.889454 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:31.888997 2580 scope.go:117] "RemoveContainer" containerID="ab8f6dce8cd62182c46604e13b9b96e0cc717f1e0dd16134e507888103e07d91" Jul 17 06:17:31.889454 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:31.889366 2580 scope.go:117] "RemoveContainer" containerID="dbbd382df0c915e2a42ba14b7371e1f46d5d4c1f92d986fc9afb541e3d7214b4" Jul 17 06:17:31.889659 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:31.889635 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:17:32.670270 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:32.670225 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:17:32.894209 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:32.894184 2580 scope.go:117] "RemoveContainer" containerID="dbbd382df0c915e2a42ba14b7371e1f46d5d4c1f92d986fc9afb541e3d7214b4" Jul 17 06:17:32.894606 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:32.894353 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:17:33.299513 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.299479 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c"] Jul 17 06:17:33.299847 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.299833 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58584594-7121-4bcc-bb4e-00668e8257ad" containerName="revoke-keys" Jul 17 06:17:33.299896 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.299848 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="58584594-7121-4bcc-bb4e-00668e8257ad" containerName="revoke-keys" Jul 17 06:17:33.299896 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.299866 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58584594-7121-4bcc-bb4e-00668e8257ad" containerName="revoke-keys" Jul 17 06:17:33.299896 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.299871 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="58584594-7121-4bcc-bb4e-00668e8257ad" containerName="revoke-keys" Jul 17 06:17:33.299995 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.299930 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="58584594-7121-4bcc-bb4e-00668e8257ad" containerName="revoke-keys" Jul 17 06:17:33.299995 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.299938 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="58584594-7121-4bcc-bb4e-00668e8257ad" containerName="revoke-keys" Jul 17 06:17:33.299995 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.299944 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a18db76-fcc0-4e64-a1bb-cf8dbc5fccc2" containerName="maas-api" Jul 17 06:17:33.299995 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.299954 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="58584594-7121-4bcc-bb4e-00668e8257ad" containerName="revoke-keys" Jul 17 06:17:33.304003 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.303985 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" Jul 17 06:17:33.306266 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.306242 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:17:33.311215 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.311193 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c"] Jul 17 06:17:33.381486 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.381448 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b875f2-cb27-4a53-909c-1aca057a3022-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-zs98c\" (UID: \"01b875f2-cb27-4a53-909c-1aca057a3022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" Jul 17 06:17:33.482485 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.482439 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b875f2-cb27-4a53-909c-1aca057a3022-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-zs98c\" (UID: \"01b875f2-cb27-4a53-909c-1aca057a3022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" Jul 17 06:17:33.483103 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.483081 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b875f2-cb27-4a53-909c-1aca057a3022-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-zs98c\" (UID: \"01b875f2-cb27-4a53-909c-1aca057a3022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" Jul 17 06:17:33.615367 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.615319 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" Jul 17 06:17:33.752521 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.752482 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c"] Jul 17 06:17:33.755627 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:17:33.755595 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b875f2_cb27_4a53_909c_1aca057a3022.slice/crio-65a5acd7d8edecf89926a97f8b7a0c840d88402f58eafa596ce7baa8e060d12a WatchSource:0}: Error finding container 65a5acd7d8edecf89926a97f8b7a0c840d88402f58eafa596ce7baa8e060d12a: Status 404 returned error can't find the container with id 65a5acd7d8edecf89926a97f8b7a0c840d88402f58eafa596ce7baa8e060d12a Jul 17 06:17:33.899623 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:33.899529 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" event={"ID":"01b875f2-cb27-4a53-909c-1aca057a3022","Type":"ContainerStarted","Data":"65a5acd7d8edecf89926a97f8b7a0c840d88402f58eafa596ce7baa8e060d12a"} Jul 17 06:17:34.905581 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:34.905538 2580 generic.go:356] "Generic (PLEG): container finished" podID="01b875f2-cb27-4a53-909c-1aca057a3022" containerID="ac7222bbc4e8e9833d1a9b7f350792c58310600c6f52cb2343015fbf2282756a" exitCode=6 Jul 17 06:17:34.905976 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:34.905615 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" event={"ID":"01b875f2-cb27-4a53-909c-1aca057a3022","Type":"ContainerDied","Data":"ac7222bbc4e8e9833d1a9b7f350792c58310600c6f52cb2343015fbf2282756a"} Jul 17 06:17:34.905976 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:34.905802 2580 scope.go:117] "RemoveContainer" containerID="ac7222bbc4e8e9833d1a9b7f350792c58310600c6f52cb2343015fbf2282756a" Jul 17 06:17:35.912213 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:35.912167 2580 generic.go:356] "Generic (PLEG): container finished" podID="01b875f2-cb27-4a53-909c-1aca057a3022" containerID="7771c74cf3364de291863aafe90e562a06b934ace469537a6648f85d479a4022" exitCode=6 Jul 17 06:17:35.912673 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:35.912247 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" event={"ID":"01b875f2-cb27-4a53-909c-1aca057a3022","Type":"ContainerDied","Data":"7771c74cf3364de291863aafe90e562a06b934ace469537a6648f85d479a4022"} Jul 17 06:17:35.912673 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:35.912306 2580 scope.go:117] "RemoveContainer" containerID="ac7222bbc4e8e9833d1a9b7f350792c58310600c6f52cb2343015fbf2282756a" Jul 17 06:17:35.912673 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:35.912500 2580 scope.go:117] "RemoveContainer" containerID="7771c74cf3364de291863aafe90e562a06b934ace469537a6648f85d479a4022" Jul 17 06:17:35.912790 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:35.912715 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-zs98c_odh-ai-gateway-infra(01b875f2-cb27-4a53-909c-1aca057a3022)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" podUID="01b875f2-cb27-4a53-909c-1aca057a3022" Jul 17 06:17:36.802640 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:36.802602 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:17:36.802997 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:36.802982 2580 scope.go:117] "RemoveContainer" containerID="dbbd382df0c915e2a42ba14b7371e1f46d5d4c1f92d986fc9afb541e3d7214b4" Jul 17 06:17:36.803179 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:36.803161 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:17:36.918019 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:36.917985 2580 scope.go:117] "RemoveContainer" containerID="7771c74cf3364de291863aafe90e562a06b934ace469537a6648f85d479a4022" Jul 17 06:17:36.918443 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:36.918192 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-zs98c_odh-ai-gateway-infra(01b875f2-cb27-4a53-909c-1aca057a3022)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" podUID="01b875f2-cb27-4a53-909c-1aca057a3022" Jul 17 06:17:45.433424 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:45.433375 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h"] Jul 17 06:17:45.433838 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:45.433729 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58584594-7121-4bcc-bb4e-00668e8257ad" containerName="revoke-keys" Jul 17 06:17:45.433838 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:45.433741 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="58584594-7121-4bcc-bb4e-00668e8257ad" containerName="revoke-keys" Jul 17 06:17:45.438017 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:45.438001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" Jul 17 06:17:45.443661 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:45.443635 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h"] Jul 17 06:17:45.494454 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:45.494418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a8bcb0-f5bb-4396-b060-61fa9305f08b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h\" (UID: \"b6a8bcb0-f5bb-4396-b060-61fa9305f08b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" Jul 17 06:17:45.595902 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:45.595865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a8bcb0-f5bb-4396-b060-61fa9305f08b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h\" (UID: \"b6a8bcb0-f5bb-4396-b060-61fa9305f08b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" Jul 17 06:17:45.596558 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:45.596534 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a8bcb0-f5bb-4396-b060-61fa9305f08b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h\" (UID: \"b6a8bcb0-f5bb-4396-b060-61fa9305f08b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" Jul 17 06:17:45.749664 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:45.749577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" Jul 17 06:17:45.880919 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:45.880885 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h"] Jul 17 06:17:45.884047 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:17:45.884015 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a8bcb0_f5bb_4396_b060_61fa9305f08b.slice/crio-4ac1661ea12beed94c533bc590aced5cb164dae23fb5c2b575fe1eb208fd1048 WatchSource:0}: Error finding container 4ac1661ea12beed94c533bc590aced5cb164dae23fb5c2b575fe1eb208fd1048: Status 404 returned error can't find the container with id 4ac1661ea12beed94c533bc590aced5cb164dae23fb5c2b575fe1eb208fd1048 Jul 17 06:17:45.963653 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:45.963618 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" event={"ID":"b6a8bcb0-f5bb-4396-b060-61fa9305f08b","Type":"ContainerStarted","Data":"4ac1661ea12beed94c533bc590aced5cb164dae23fb5c2b575fe1eb208fd1048"} Jul 17 06:17:46.969566 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:46.969525 2580 generic.go:356] "Generic (PLEG): container finished" podID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" containerID="b35cfafe5dc22f37bd4bfc00f8dcd3b037e686330ea5569c3871b57861caee53" exitCode=6 Jul 17 06:17:46.969976 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:46.969578 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" event={"ID":"b6a8bcb0-f5bb-4396-b060-61fa9305f08b","Type":"ContainerDied","Data":"b35cfafe5dc22f37bd4bfc00f8dcd3b037e686330ea5569c3871b57861caee53"} Jul 17 06:17:46.969976 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:46.969786 2580 scope.go:117] "RemoveContainer" containerID="b35cfafe5dc22f37bd4bfc00f8dcd3b037e686330ea5569c3871b57861caee53" Jul 17 06:17:47.975354 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:47.975318 2580 generic.go:356] "Generic (PLEG): container finished" podID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" containerID="13cfaf8cb66dac65c8f6a41edcf08fcef503be8cc848676fb296fabb7cae3476" exitCode=6 Jul 17 06:17:47.975848 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:47.975375 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" event={"ID":"b6a8bcb0-f5bb-4396-b060-61fa9305f08b","Type":"ContainerDied","Data":"13cfaf8cb66dac65c8f6a41edcf08fcef503be8cc848676fb296fabb7cae3476"} Jul 17 06:17:47.975848 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:47.975437 2580 scope.go:117] "RemoveContainer" containerID="b35cfafe5dc22f37bd4bfc00f8dcd3b037e686330ea5569c3871b57861caee53" Jul 17 06:17:47.975848 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:47.975627 2580 scope.go:117] "RemoveContainer" containerID="13cfaf8cb66dac65c8f6a41edcf08fcef503be8cc848676fb296fabb7cae3476" Jul 17 06:17:47.976011 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:47.975875 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h_odh-ai-gateway-infra(b6a8bcb0-f5bb-4396-b060-61fa9305f08b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" podUID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" Jul 17 06:17:48.981261 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:48.981229 2580 scope.go:117] "RemoveContainer" containerID="13cfaf8cb66dac65c8f6a41edcf08fcef503be8cc848676fb296fabb7cae3476" Jul 17 06:17:48.981771 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:48.981502 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h_odh-ai-gateway-infra(b6a8bcb0-f5bb-4396-b060-61fa9305f08b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" podUID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" Jul 17 06:17:49.884507 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:49.884469 2580 scope.go:117] "RemoveContainer" containerID="dbbd382df0c915e2a42ba14b7371e1f46d5d4c1f92d986fc9afb541e3d7214b4" Jul 17 06:17:49.884710 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:49.884660 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:17:51.775852 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:51.775818 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn"] Jul 17 06:17:51.779376 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:51.779356 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" Jul 17 06:17:51.788487 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:51.788454 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn"] Jul 17 06:17:51.852262 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:51.852227 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b612e3-6fe7-4e83-98f9-0ac30a4208dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-k4tnn\" (UID: \"d8b612e3-6fe7-4e83-98f9-0ac30a4208dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" Jul 17 06:17:51.884586 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:51.884558 2580 scope.go:117] "RemoveContainer" containerID="7771c74cf3364de291863aafe90e562a06b934ace469537a6648f85d479a4022" Jul 17 06:17:51.953222 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:51.953186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b612e3-6fe7-4e83-98f9-0ac30a4208dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-k4tnn\" (UID: \"d8b612e3-6fe7-4e83-98f9-0ac30a4208dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" Jul 17 06:17:51.953829 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:51.953808 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b612e3-6fe7-4e83-98f9-0ac30a4208dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-k4tnn\" (UID: \"d8b612e3-6fe7-4e83-98f9-0ac30a4208dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" Jul 17 06:17:52.090269 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:52.090231 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" Jul 17 06:17:52.223291 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:52.223268 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn"] Jul 17 06:17:52.225807 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:17:52.225776 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b612e3_6fe7_4e83_98f9_0ac30a4208dc.slice/crio-7ed9a8b60f71b624324fb9e215f6312f737205bb9ad19882220bc0d7025f4a16 WatchSource:0}: Error finding container 7ed9a8b60f71b624324fb9e215f6312f737205bb9ad19882220bc0d7025f4a16: Status 404 returned error can't find the container with id 7ed9a8b60f71b624324fb9e215f6312f737205bb9ad19882220bc0d7025f4a16 Jul 17 06:17:52.999146 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:52.999104 2580 generic.go:356] "Generic (PLEG): container finished" podID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" containerID="90438d651135642a45dd9ee6cb2abd614500303ccefdfa4f36299a42fc7dfdf9" exitCode=7 Jul 17 06:17:52.999625 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:52.999174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" event={"ID":"d8b612e3-6fe7-4e83-98f9-0ac30a4208dc","Type":"ContainerDied","Data":"90438d651135642a45dd9ee6cb2abd614500303ccefdfa4f36299a42fc7dfdf9"} Jul 17 06:17:52.999625 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:52.999233 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" event={"ID":"d8b612e3-6fe7-4e83-98f9-0ac30a4208dc","Type":"ContainerStarted","Data":"7ed9a8b60f71b624324fb9e215f6312f737205bb9ad19882220bc0d7025f4a16"} Jul 17 06:17:52.999625 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:52.999478 2580 scope.go:117] "RemoveContainer" containerID="90438d651135642a45dd9ee6cb2abd614500303ccefdfa4f36299a42fc7dfdf9" Jul 17 06:17:53.001163 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:53.001141 2580 generic.go:356] "Generic (PLEG): container finished" podID="01b875f2-cb27-4a53-909c-1aca057a3022" containerID="d1b9e5d6c5a945f636ada42745796bd98968167710fda74c291a8b7ba193d2e6" exitCode=6 Jul 17 06:17:53.001257 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:53.001170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" event={"ID":"01b875f2-cb27-4a53-909c-1aca057a3022","Type":"ContainerDied","Data":"d1b9e5d6c5a945f636ada42745796bd98968167710fda74c291a8b7ba193d2e6"} Jul 17 06:17:53.001257 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:53.001210 2580 scope.go:117] "RemoveContainer" containerID="7771c74cf3364de291863aafe90e562a06b934ace469537a6648f85d479a4022" Jul 17 06:17:53.001549 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:53.001535 2580 scope.go:117] "RemoveContainer" containerID="d1b9e5d6c5a945f636ada42745796bd98968167710fda74c291a8b7ba193d2e6" Jul 17 06:17:53.001837 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:53.001735 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-zs98c_odh-ai-gateway-infra(01b875f2-cb27-4a53-909c-1aca057a3022)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" podUID="01b875f2-cb27-4a53-909c-1aca057a3022" Jul 17 06:17:54.008877 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:54.008793 2580 generic.go:356] "Generic (PLEG): container finished" podID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" containerID="369ba2ac6f0e6dfb942b77fce4ca098f0bd37cd7f8e2f08379fad22f16d41974" exitCode=7 Jul 17 06:17:54.009289 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:54.008884 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" event={"ID":"d8b612e3-6fe7-4e83-98f9-0ac30a4208dc","Type":"ContainerDied","Data":"369ba2ac6f0e6dfb942b77fce4ca098f0bd37cd7f8e2f08379fad22f16d41974"} Jul 17 06:17:54.009289 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:54.008942 2580 scope.go:117] "RemoveContainer" containerID="90438d651135642a45dd9ee6cb2abd614500303ccefdfa4f36299a42fc7dfdf9" Jul 17 06:17:54.009289 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:54.009073 2580 scope.go:117] "RemoveContainer" containerID="369ba2ac6f0e6dfb942b77fce4ca098f0bd37cd7f8e2f08379fad22f16d41974" Jul 17 06:17:54.009493 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:54.009331 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-k4tnn_odh-ai-gateway-infra(d8b612e3-6fe7-4e83-98f9-0ac30a4208dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" podUID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" Jul 17 06:17:54.056379 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:54.056340 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c"] Jul 17 06:17:54.191394 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:54.191366 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" Jul 17 06:17:54.274546 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:54.274444 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b875f2-cb27-4a53-909c-1aca057a3022-maas-api-service-ca\") pod \"01b875f2-cb27-4a53-909c-1aca057a3022\" (UID: \"01b875f2-cb27-4a53-909c-1aca057a3022\") " Jul 17 06:17:54.274839 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:54.274816 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b875f2-cb27-4a53-909c-1aca057a3022-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "01b875f2-cb27-4a53-909c-1aca057a3022" (UID: "01b875f2-cb27-4a53-909c-1aca057a3022"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:17:54.375735 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:54.375686 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b875f2-cb27-4a53-909c-1aca057a3022-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:17:55.016969 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:55.016935 2580 scope.go:117] "RemoveContainer" containerID="369ba2ac6f0e6dfb942b77fce4ca098f0bd37cd7f8e2f08379fad22f16d41974" Jul 17 06:17:55.017353 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:17:55.017164 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-k4tnn_odh-ai-gateway-infra(d8b612e3-6fe7-4e83-98f9-0ac30a4208dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" podUID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" Jul 17 06:17:55.018133 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:55.018114 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" Jul 17 06:17:55.018246 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:55.018135 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c" event={"ID":"01b875f2-cb27-4a53-909c-1aca057a3022","Type":"ContainerDied","Data":"65a5acd7d8edecf89926a97f8b7a0c840d88402f58eafa596ce7baa8e060d12a"} Jul 17 06:17:55.018246 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:55.018206 2580 scope.go:117] "RemoveContainer" containerID="d1b9e5d6c5a945f636ada42745796bd98968167710fda74c291a8b7ba193d2e6" Jul 17 06:17:55.044043 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:55.044013 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c"] Jul 17 06:17:55.047954 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:55.047932 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-zs98c"] Jul 17 06:17:56.889862 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:17:56.889819 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01b875f2-cb27-4a53-909c-1aca057a3022" path="/var/lib/kubelet/pods/01b875f2-cb27-4a53-909c-1aca057a3022/volumes" Jul 17 06:18:02.884410 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:02.884361 2580 scope.go:117] "RemoveContainer" containerID="13cfaf8cb66dac65c8f6a41edcf08fcef503be8cc848676fb296fabb7cae3476" Jul 17 06:18:04.060261 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:04.060226 2580 generic.go:356] "Generic (PLEG): container finished" podID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" containerID="f5bf7896e7863c4bb4ade07e9a115b7b221fbffd9a22aceed6f080463fc7bbf3" exitCode=6 Jul 17 06:18:04.060698 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:04.060291 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" event={"ID":"b6a8bcb0-f5bb-4396-b060-61fa9305f08b","Type":"ContainerDied","Data":"f5bf7896e7863c4bb4ade07e9a115b7b221fbffd9a22aceed6f080463fc7bbf3"} Jul 17 06:18:04.060698 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:04.060355 2580 scope.go:117] "RemoveContainer" containerID="13cfaf8cb66dac65c8f6a41edcf08fcef503be8cc848676fb296fabb7cae3476" Jul 17 06:18:04.060698 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:04.060680 2580 scope.go:117] "RemoveContainer" containerID="f5bf7896e7863c4bb4ade07e9a115b7b221fbffd9a22aceed6f080463fc7bbf3" Jul 17 06:18:04.060901 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:18:04.060881 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h_odh-ai-gateway-infra(b6a8bcb0-f5bb-4396-b060-61fa9305f08b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" podUID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" Jul 17 06:18:04.883919 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:04.883881 2580 scope.go:117] "RemoveContainer" containerID="dbbd382df0c915e2a42ba14b7371e1f46d5d4c1f92d986fc9afb541e3d7214b4" Jul 17 06:18:04.884135 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:18:04.884076 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:18:05.087972 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:05.087933 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h"] Jul 17 06:18:05.219320 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:05.219291 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" Jul 17 06:18:05.269623 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:05.269586 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a8bcb0-f5bb-4396-b060-61fa9305f08b-maas-api-service-ca\") pod \"b6a8bcb0-f5bb-4396-b060-61fa9305f08b\" (UID: \"b6a8bcb0-f5bb-4396-b060-61fa9305f08b\") " Jul 17 06:18:05.269933 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:05.269911 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a8bcb0-f5bb-4396-b060-61fa9305f08b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b6a8bcb0-f5bb-4396-b060-61fa9305f08b" (UID: "b6a8bcb0-f5bb-4396-b060-61fa9305f08b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:18:05.371069 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:05.371031 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a8bcb0-f5bb-4396-b060-61fa9305f08b-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:18:06.072032 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:06.072000 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" Jul 17 06:18:06.072032 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:06.072009 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h" event={"ID":"b6a8bcb0-f5bb-4396-b060-61fa9305f08b","Type":"ContainerDied","Data":"4ac1661ea12beed94c533bc590aced5cb164dae23fb5c2b575fe1eb208fd1048"} Jul 17 06:18:06.072259 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:06.072079 2580 scope.go:117] "RemoveContainer" containerID="f5bf7896e7863c4bb4ade07e9a115b7b221fbffd9a22aceed6f080463fc7bbf3" Jul 17 06:18:06.112213 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:06.112175 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h"] Jul 17 06:18:06.120403 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:06.120357 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-zrj2h"] Jul 17 06:18:06.889695 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:06.889661 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" path="/var/lib/kubelet/pods/b6a8bcb0-f5bb-4396-b060-61fa9305f08b/volumes" Jul 17 06:18:07.884427 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:07.884368 2580 scope.go:117] "RemoveContainer" containerID="369ba2ac6f0e6dfb942b77fce4ca098f0bd37cd7f8e2f08379fad22f16d41974" Jul 17 06:18:09.088028 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:09.087993 2580 generic.go:356] "Generic (PLEG): container finished" podID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" containerID="de77d5ef19bb30c0a08190cda59abff48e5f9f53e7f854bca528ed8d4cefb533" exitCode=7 Jul 17 06:18:09.088514 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:09.088052 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" event={"ID":"d8b612e3-6fe7-4e83-98f9-0ac30a4208dc","Type":"ContainerDied","Data":"de77d5ef19bb30c0a08190cda59abff48e5f9f53e7f854bca528ed8d4cefb533"} Jul 17 06:18:09.088514 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:09.088125 2580 scope.go:117] "RemoveContainer" containerID="369ba2ac6f0e6dfb942b77fce4ca098f0bd37cd7f8e2f08379fad22f16d41974" Jul 17 06:18:09.088514 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:09.088496 2580 scope.go:117] "RemoveContainer" containerID="de77d5ef19bb30c0a08190cda59abff48e5f9f53e7f854bca528ed8d4cefb533" Jul 17 06:18:09.088743 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:18:09.088722 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-k4tnn_odh-ai-gateway-infra(d8b612e3-6fe7-4e83-98f9-0ac30a4208dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" podUID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" Jul 17 06:18:10.115536 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:10.115499 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn"] Jul 17 06:18:10.256327 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:10.256295 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" Jul 17 06:18:10.311273 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:10.311239 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b612e3-6fe7-4e83-98f9-0ac30a4208dc-maas-api-service-ca\") pod \"d8b612e3-6fe7-4e83-98f9-0ac30a4208dc\" (UID: \"d8b612e3-6fe7-4e83-98f9-0ac30a4208dc\") " Jul 17 06:18:10.311668 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:10.311646 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b612e3-6fe7-4e83-98f9-0ac30a4208dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" (UID: "d8b612e3-6fe7-4e83-98f9-0ac30a4208dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:18:10.412346 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:10.412272 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b612e3-6fe7-4e83-98f9-0ac30a4208dc-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:18:11.099464 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:11.099424 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" event={"ID":"d8b612e3-6fe7-4e83-98f9-0ac30a4208dc","Type":"ContainerDied","Data":"7ed9a8b60f71b624324fb9e215f6312f737205bb9ad19882220bc0d7025f4a16"} Jul 17 06:18:11.099464 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:11.099451 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn" Jul 17 06:18:11.099713 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:11.099501 2580 scope.go:117] "RemoveContainer" containerID="de77d5ef19bb30c0a08190cda59abff48e5f9f53e7f854bca528ed8d4cefb533" Jul 17 06:18:11.119492 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:11.119460 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn"] Jul 17 06:18:11.125792 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:11.125767 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-k4tnn"] Jul 17 06:18:12.889378 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:12.889343 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" path="/var/lib/kubelet/pods/d8b612e3-6fe7-4e83-98f9-0ac30a4208dc/volumes" Jul 17 06:18:15.884848 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:15.884812 2580 scope.go:117] "RemoveContainer" containerID="dbbd382df0c915e2a42ba14b7371e1f46d5d4c1f92d986fc9afb541e3d7214b4" Jul 17 06:18:17.129329 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:17.129290 2580 generic.go:356] "Generic (PLEG): container finished" podID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerID="ec7bf73ead1c8c72925e817b8bdc6c4e9ee43f053bb3303d46362110a245081e" exitCode=1 Jul 17 06:18:17.129770 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:17.129361 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" event={"ID":"a65b4941-22fa-4d95-8c9d-43fc877c071e","Type":"ContainerDied","Data":"ec7bf73ead1c8c72925e817b8bdc6c4e9ee43f053bb3303d46362110a245081e"} Jul 17 06:18:17.129770 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:17.129443 2580 scope.go:117] "RemoveContainer" containerID="dbbd382df0c915e2a42ba14b7371e1f46d5d4c1f92d986fc9afb541e3d7214b4" Jul 17 06:18:17.129770 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:17.129761 2580 scope.go:117] "RemoveContainer" containerID="ec7bf73ead1c8c72925e817b8bdc6c4e9ee43f053bb3303d46362110a245081e" Jul 17 06:18:17.130010 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:18:17.129985 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:18:17.670557 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:17.670517 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:18:18.135838 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:18.135810 2580 scope.go:117] "RemoveContainer" containerID="ec7bf73ead1c8c72925e817b8bdc6c4e9ee43f053bb3303d46362110a245081e" Jul 17 06:18:18.136237 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:18:18.135981 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:18:26.802715 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:26.802662 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:18:26.803247 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:26.803228 2580 scope.go:117] "RemoveContainer" containerID="ec7bf73ead1c8c72925e817b8bdc6c4e9ee43f053bb3303d46362110a245081e" Jul 17 06:18:26.803562 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:18:26.803532 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:18:33.358319 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358279 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl"] Jul 17 06:18:33.358742 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358663 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b875f2-cb27-4a53-909c-1aca057a3022" containerName="revoke-keys" Jul 17 06:18:33.358742 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358675 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b875f2-cb27-4a53-909c-1aca057a3022" containerName="revoke-keys" Jul 17 06:18:33.358742 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358684 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" containerName="revoke-keys" Jul 17 06:18:33.358742 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358690 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" containerName="revoke-keys" Jul 17 06:18:33.358742 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358696 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" containerName="revoke-keys" Jul 17 06:18:33.358742 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358702 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" containerName="revoke-keys" Jul 17 06:18:33.358742 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358710 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" containerName="revoke-keys" Jul 17 06:18:33.358742 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358715 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" containerName="revoke-keys" Jul 17 06:18:33.358742 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358726 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" containerName="revoke-keys" Jul 17 06:18:33.358742 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358731 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" containerName="revoke-keys" Jul 17 06:18:33.358742 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358736 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b875f2-cb27-4a53-909c-1aca057a3022" containerName="revoke-keys" Jul 17 06:18:33.358742 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358742 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b875f2-cb27-4a53-909c-1aca057a3022" containerName="revoke-keys" Jul 17 06:18:33.359108 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358798 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" containerName="revoke-keys" Jul 17 06:18:33.359108 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358807 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" containerName="revoke-keys" Jul 17 06:18:33.359108 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358813 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b875f2-cb27-4a53-909c-1aca057a3022" containerName="revoke-keys" Jul 17 06:18:33.359108 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358819 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" containerName="revoke-keys" Jul 17 06:18:33.359108 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358825 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b875f2-cb27-4a53-909c-1aca057a3022" containerName="revoke-keys" Jul 17 06:18:33.359108 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358831 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b875f2-cb27-4a53-909c-1aca057a3022" containerName="revoke-keys" Jul 17 06:18:33.359108 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.358837 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" containerName="revoke-keys" Jul 17 06:18:33.361740 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.361721 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" Jul 17 06:18:33.363981 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.363957 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:18:33.369794 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.369773 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl"] Jul 17 06:18:33.423205 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.423159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f01faf-76c4-4a43-a0b1-fd412bbd120b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-9sltl\" (UID: \"b3f01faf-76c4-4a43-a0b1-fd412bbd120b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" Jul 17 06:18:33.524171 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.524122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f01faf-76c4-4a43-a0b1-fd412bbd120b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-9sltl\" (UID: \"b3f01faf-76c4-4a43-a0b1-fd412bbd120b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" Jul 17 06:18:33.524846 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.524818 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f01faf-76c4-4a43-a0b1-fd412bbd120b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-9sltl\" (UID: \"b3f01faf-76c4-4a43-a0b1-fd412bbd120b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" Jul 17 06:18:33.672677 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.672573 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" Jul 17 06:18:33.804701 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:33.804665 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl"] Jul 17 06:18:33.807564 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:18:33.807540 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f01faf_76c4_4a43_a0b1_fd412bbd120b.slice/crio-d544e5b1ab2d7274b1a3c355aaacbacdf3057f4af19446f034cdd57c0d2ee0f7 WatchSource:0}: Error finding container d544e5b1ab2d7274b1a3c355aaacbacdf3057f4af19446f034cdd57c0d2ee0f7: Status 404 returned error can't find the container with id d544e5b1ab2d7274b1a3c355aaacbacdf3057f4af19446f034cdd57c0d2ee0f7 Jul 17 06:18:34.208210 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:34.208171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" event={"ID":"b3f01faf-76c4-4a43-a0b1-fd412bbd120b","Type":"ContainerStarted","Data":"3802b1023cfaa2120019e9aadbc93a4f2d98dbf071363517dfe081b439980b41"} Jul 17 06:18:34.208428 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:34.208223 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" event={"ID":"b3f01faf-76c4-4a43-a0b1-fd412bbd120b","Type":"ContainerStarted","Data":"d544e5b1ab2d7274b1a3c355aaacbacdf3057f4af19446f034cdd57c0d2ee0f7"} Jul 17 06:18:34.227769 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:34.227698 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" podStartSLOduration=1.2276837170000001 podStartE2EDuration="1.227683717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:18:34.227482819 +0000 UTC m=+2238.972162148" watchObservedRunningTime="2026-07-17 06:18:34.227683717 +0000 UTC m=+2238.972363022" Jul 17 06:18:35.214445 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:35.214411 2580 generic.go:356] "Generic (PLEG): container finished" podID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" containerID="3802b1023cfaa2120019e9aadbc93a4f2d98dbf071363517dfe081b439980b41" exitCode=6 Jul 17 06:18:35.214913 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:35.214482 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" event={"ID":"b3f01faf-76c4-4a43-a0b1-fd412bbd120b","Type":"ContainerDied","Data":"3802b1023cfaa2120019e9aadbc93a4f2d98dbf071363517dfe081b439980b41"} Jul 17 06:18:35.214913 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:35.214698 2580 scope.go:117] "RemoveContainer" containerID="3802b1023cfaa2120019e9aadbc93a4f2d98dbf071363517dfe081b439980b41" Jul 17 06:18:36.221128 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:36.221027 2580 generic.go:356] "Generic (PLEG): container finished" podID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" containerID="f7a14d2bb62765973520df4921e4360f2367382d4af3bf983f6f9d1a35ad07be" exitCode=6 Jul 17 06:18:36.221128 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:36.221119 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" event={"ID":"b3f01faf-76c4-4a43-a0b1-fd412bbd120b","Type":"ContainerDied","Data":"f7a14d2bb62765973520df4921e4360f2367382d4af3bf983f6f9d1a35ad07be"} Jul 17 06:18:36.221690 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:36.221173 2580 scope.go:117] "RemoveContainer" containerID="3802b1023cfaa2120019e9aadbc93a4f2d98dbf071363517dfe081b439980b41" Jul 17 06:18:36.221690 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:36.221348 2580 scope.go:117] "RemoveContainer" containerID="f7a14d2bb62765973520df4921e4360f2367382d4af3bf983f6f9d1a35ad07be" Jul 17 06:18:36.221690 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:18:36.221620 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-9sltl_odh-ai-gateway-infra(b3f01faf-76c4-4a43-a0b1-fd412bbd120b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" podUID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" Jul 17 06:18:37.226907 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:37.226830 2580 scope.go:117] "RemoveContainer" containerID="f7a14d2bb62765973520df4921e4360f2367382d4af3bf983f6f9d1a35ad07be" Jul 17 06:18:37.227269 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:18:37.227041 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-9sltl_odh-ai-gateway-infra(b3f01faf-76c4-4a43-a0b1-fd412bbd120b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" podUID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" Jul 17 06:18:40.889815 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:40.889770 2580 scope.go:117] "RemoveContainer" containerID="ec7bf73ead1c8c72925e817b8bdc6c4e9ee43f053bb3303d46362110a245081e" Jul 17 06:18:40.890476 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:18:40.890029 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:18:51.884378 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:51.884343 2580 scope.go:117] "RemoveContainer" containerID="f7a14d2bb62765973520df4921e4360f2367382d4af3bf983f6f9d1a35ad07be" Jul 17 06:18:52.294104 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:52.294011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" event={"ID":"b3f01faf-76c4-4a43-a0b1-fd412bbd120b","Type":"ContainerStarted","Data":"fd437d72301fdc841a6d7f10d26770ee8f579966e25e3a7a2b43dee2c9b35783"} Jul 17 06:18:53.300306 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:53.300268 2580 generic.go:356] "Generic (PLEG): container finished" podID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" containerID="fd437d72301fdc841a6d7f10d26770ee8f579966e25e3a7a2b43dee2c9b35783" exitCode=6 Jul 17 06:18:53.300758 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:53.300327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" event={"ID":"b3f01faf-76c4-4a43-a0b1-fd412bbd120b","Type":"ContainerDied","Data":"fd437d72301fdc841a6d7f10d26770ee8f579966e25e3a7a2b43dee2c9b35783"} Jul 17 06:18:53.300758 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:53.300376 2580 scope.go:117] "RemoveContainer" containerID="f7a14d2bb62765973520df4921e4360f2367382d4af3bf983f6f9d1a35ad07be" Jul 17 06:18:53.300758 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:53.300707 2580 scope.go:117] "RemoveContainer" containerID="fd437d72301fdc841a6d7f10d26770ee8f579966e25e3a7a2b43dee2c9b35783" Jul 17 06:18:53.300957 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:18:53.300937 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-9sltl_odh-ai-gateway-infra(b3f01faf-76c4-4a43-a0b1-fd412bbd120b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" podUID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" Jul 17 06:18:53.338899 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:53.338864 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl"] Jul 17 06:18:54.434326 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:54.434296 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" Jul 17 06:18:54.515376 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:54.515344 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f01faf-76c4-4a43-a0b1-fd412bbd120b-maas-api-service-ca\") pod \"b3f01faf-76c4-4a43-a0b1-fd412bbd120b\" (UID: \"b3f01faf-76c4-4a43-a0b1-fd412bbd120b\") " Jul 17 06:18:54.515736 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:54.515711 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f01faf-76c4-4a43-a0b1-fd412bbd120b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3f01faf-76c4-4a43-a0b1-fd412bbd120b" (UID: "b3f01faf-76c4-4a43-a0b1-fd412bbd120b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:18:54.616944 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:54.616911 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f01faf-76c4-4a43-a0b1-fd412bbd120b-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:18:54.884507 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:54.884410 2580 scope.go:117] "RemoveContainer" containerID="ec7bf73ead1c8c72925e817b8bdc6c4e9ee43f053bb3303d46362110a245081e" Jul 17 06:18:54.884688 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:18:54.884619 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:18:55.311378 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:55.311292 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" Jul 17 06:18:55.311378 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:55.311301 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl" event={"ID":"b3f01faf-76c4-4a43-a0b1-fd412bbd120b","Type":"ContainerDied","Data":"d544e5b1ab2d7274b1a3c355aaacbacdf3057f4af19446f034cdd57c0d2ee0f7"} Jul 17 06:18:55.311378 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:55.311359 2580 scope.go:117] "RemoveContainer" containerID="fd437d72301fdc841a6d7f10d26770ee8f579966e25e3a7a2b43dee2c9b35783" Jul 17 06:18:55.330990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:55.330956 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl"] Jul 17 06:18:55.337498 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:55.337469 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-9sltl"] Jul 17 06:18:56.888944 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:18:56.888911 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" path="/var/lib/kubelet/pods/b3f01faf-76c4-4a43-a0b1-fd412bbd120b/volumes" Jul 17 06:19:08.884018 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:08.883978 2580 scope.go:117] "RemoveContainer" containerID="ec7bf73ead1c8c72925e817b8bdc6c4e9ee43f053bb3303d46362110a245081e" Jul 17 06:19:08.884572 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:19:08.884213 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:19:19.884172 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:19.884139 2580 scope.go:117] "RemoveContainer" containerID="ec7bf73ead1c8c72925e817b8bdc6c4e9ee43f053bb3303d46362110a245081e" Jul 17 06:19:19.884607 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:19:19.884329 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:19:21.024538 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:21.024510 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:19:21.354633 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:21.354595 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:19:30.884791 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:30.884755 2580 scope.go:117] "RemoveContainer" containerID="ec7bf73ead1c8c72925e817b8bdc6c4e9ee43f053bb3303d46362110a245081e" Jul 17 06:19:30.885297 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:19:30.884966 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:19:33.461691 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.461651 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l"] Jul 17 06:19:33.462087 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.462049 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" containerName="revoke-keys" Jul 17 06:19:33.462087 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.462063 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" containerName="revoke-keys" Jul 17 06:19:33.462087 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.462084 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b875f2-cb27-4a53-909c-1aca057a3022" containerName="revoke-keys" Jul 17 06:19:33.462193 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.462090 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b875f2-cb27-4a53-909c-1aca057a3022" containerName="revoke-keys" Jul 17 06:19:33.462193 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.462100 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" containerName="revoke-keys" Jul 17 06:19:33.462193 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.462106 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" containerName="revoke-keys" Jul 17 06:19:33.462193 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.462114 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" containerName="revoke-keys" Jul 17 06:19:33.462193 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.462123 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" containerName="revoke-keys" Jul 17 06:19:33.462193 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.462136 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" containerName="revoke-keys" Jul 17 06:19:33.462193 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.462142 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" containerName="revoke-keys" Jul 17 06:19:33.462193 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.462188 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a8bcb0-f5bb-4396-b060-61fa9305f08b" containerName="revoke-keys" Jul 17 06:19:33.462466 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.462198 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b612e3-6fe7-4e83-98f9-0ac30a4208dc" containerName="revoke-keys" Jul 17 06:19:33.462466 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.462210 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" containerName="revoke-keys" Jul 17 06:19:33.462466 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.462221 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" containerName="revoke-keys" Jul 17 06:19:33.465135 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.465117 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" Jul 17 06:19:33.468054 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.468028 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:19:33.483034 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.482993 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l"] Jul 17 06:19:33.561296 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.561220 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab79bb8a-d52e-4470-887e-397898788798-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l\" (UID: \"ab79bb8a-d52e-4470-887e-397898788798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" Jul 17 06:19:33.662522 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.662484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab79bb8a-d52e-4470-887e-397898788798-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l\" (UID: \"ab79bb8a-d52e-4470-887e-397898788798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" Jul 17 06:19:33.663113 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.663094 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab79bb8a-d52e-4470-887e-397898788798-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l\" (UID: \"ab79bb8a-d52e-4470-887e-397898788798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" Jul 17 06:19:33.774833 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.774727 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" Jul 17 06:19:33.913511 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:33.913480 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l"] Jul 17 06:19:33.917366 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:19:33.917341 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab79bb8a_d52e_4470_887e_397898788798.slice/crio-ab330cc3b6efb71a21ec18f8f8392f911c2b1a4df2d9b5ef51f2f7b3f4f7c014 WatchSource:0}: Error finding container ab330cc3b6efb71a21ec18f8f8392f911c2b1a4df2d9b5ef51f2f7b3f4f7c014: Status 404 returned error can't find the container with id ab330cc3b6efb71a21ec18f8f8392f911c2b1a4df2d9b5ef51f2f7b3f4f7c014 Jul 17 06:19:34.487269 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:34.487183 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" event={"ID":"ab79bb8a-d52e-4470-887e-397898788798","Type":"ContainerStarted","Data":"0f7e64c92dde7f5e84110f63534631176609db37e61deefd3fb88bb37ecb11fe"} Jul 17 06:19:34.487269 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:34.487232 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" event={"ID":"ab79bb8a-d52e-4470-887e-397898788798","Type":"ContainerStarted","Data":"ab330cc3b6efb71a21ec18f8f8392f911c2b1a4df2d9b5ef51f2f7b3f4f7c014"} Jul 17 06:19:34.519036 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:34.518982 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" podStartSLOduration=1.518966259 podStartE2EDuration="1.518966259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:19:34.518249126 +0000 UTC m=+2299.262928458" watchObservedRunningTime="2026-07-17 06:19:34.518966259 +0000 UTC m=+2299.263645616" Jul 17 06:19:35.492829 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:35.492794 2580 generic.go:356] "Generic (PLEG): container finished" podID="ab79bb8a-d52e-4470-887e-397898788798" containerID="0f7e64c92dde7f5e84110f63534631176609db37e61deefd3fb88bb37ecb11fe" exitCode=6 Jul 17 06:19:35.493218 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:35.492879 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" event={"ID":"ab79bb8a-d52e-4470-887e-397898788798","Type":"ContainerDied","Data":"0f7e64c92dde7f5e84110f63534631176609db37e61deefd3fb88bb37ecb11fe"} Jul 17 06:19:35.493218 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:35.493075 2580 scope.go:117] "RemoveContainer" containerID="0f7e64c92dde7f5e84110f63534631176609db37e61deefd3fb88bb37ecb11fe" Jul 17 06:19:36.498817 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:36.498723 2580 generic.go:356] "Generic (PLEG): container finished" podID="ab79bb8a-d52e-4470-887e-397898788798" containerID="6cad555ea9ca78aaea30f7d7f261d7682a065085ee3a655fbe6c8c50a9e69fb4" exitCode=6 Jul 17 06:19:36.498817 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:36.498787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" event={"ID":"ab79bb8a-d52e-4470-887e-397898788798","Type":"ContainerDied","Data":"6cad555ea9ca78aaea30f7d7f261d7682a065085ee3a655fbe6c8c50a9e69fb4"} Jul 17 06:19:36.499300 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:36.498848 2580 scope.go:117] "RemoveContainer" containerID="0f7e64c92dde7f5e84110f63534631176609db37e61deefd3fb88bb37ecb11fe" Jul 17 06:19:36.499300 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:36.499037 2580 scope.go:117] "RemoveContainer" containerID="6cad555ea9ca78aaea30f7d7f261d7682a065085ee3a655fbe6c8c50a9e69fb4" Jul 17 06:19:36.499300 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:19:36.499267 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l_odh-ai-gateway-infra(ab79bb8a-d52e-4470-887e-397898788798)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" podUID="ab79bb8a-d52e-4470-887e-397898788798" Jul 17 06:19:37.505062 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:37.505036 2580 scope.go:117] "RemoveContainer" containerID="6cad555ea9ca78aaea30f7d7f261d7682a065085ee3a655fbe6c8c50a9e69fb4" Jul 17 06:19:37.505501 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:19:37.505233 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l_odh-ai-gateway-infra(ab79bb8a-d52e-4470-887e-397898788798)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" podUID="ab79bb8a-d52e-4470-887e-397898788798" Jul 17 06:19:41.882153 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:41.882105 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb"] Jul 17 06:19:41.882714 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:41.882482 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" containerName="revoke-keys" Jul 17 06:19:41.882714 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:41.882493 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" containerName="revoke-keys" Jul 17 06:19:41.882714 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:41.882554 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f01faf-76c4-4a43-a0b1-fd412bbd120b" containerName="revoke-keys" Jul 17 06:19:41.886728 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:41.886702 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" Jul 17 06:19:41.897203 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:41.897173 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb"] Jul 17 06:19:41.948086 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:41.948048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575565fd-e8ae-4788-8111-eb9b15ded6f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-r22tb\" (UID: \"575565fd-e8ae-4788-8111-eb9b15ded6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" Jul 17 06:19:42.049053 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:42.048994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575565fd-e8ae-4788-8111-eb9b15ded6f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-r22tb\" (UID: \"575565fd-e8ae-4788-8111-eb9b15ded6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" Jul 17 06:19:42.049671 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:42.049648 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575565fd-e8ae-4788-8111-eb9b15ded6f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-r22tb\" (UID: \"575565fd-e8ae-4788-8111-eb9b15ded6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" Jul 17 06:19:42.198834 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:42.198746 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" Jul 17 06:19:42.356952 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:42.356906 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb"] Jul 17 06:19:42.362685 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:19:42.362643 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575565fd_e8ae_4788_8111_eb9b15ded6f1.slice/crio-c0d887b86de57727eaf435945f84ee10262a69960c84746f8c96fec9fa37d684 WatchSource:0}: Error finding container c0d887b86de57727eaf435945f84ee10262a69960c84746f8c96fec9fa37d684: Status 404 returned error can't find the container with id c0d887b86de57727eaf435945f84ee10262a69960c84746f8c96fec9fa37d684 Jul 17 06:19:42.530632 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:42.530541 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" event={"ID":"575565fd-e8ae-4788-8111-eb9b15ded6f1","Type":"ContainerStarted","Data":"c0d887b86de57727eaf435945f84ee10262a69960c84746f8c96fec9fa37d684"} Jul 17 06:19:43.537634 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:43.537596 2580 generic.go:356] "Generic (PLEG): container finished" podID="575565fd-e8ae-4788-8111-eb9b15ded6f1" containerID="67157b3909982d8d67e45fcf7102deaec2afd26060b5b11d9846af2be7c2195a" exitCode=7 Jul 17 06:19:43.538050 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:43.537683 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" event={"ID":"575565fd-e8ae-4788-8111-eb9b15ded6f1","Type":"ContainerDied","Data":"67157b3909982d8d67e45fcf7102deaec2afd26060b5b11d9846af2be7c2195a"} Jul 17 06:19:43.538050 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:43.537853 2580 scope.go:117] "RemoveContainer" containerID="67157b3909982d8d67e45fcf7102deaec2afd26060b5b11d9846af2be7c2195a" Jul 17 06:19:44.543987 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:44.543888 2580 generic.go:356] "Generic (PLEG): container finished" podID="575565fd-e8ae-4788-8111-eb9b15ded6f1" containerID="02aadefc97b0f4d2964c6989a3fd7521a73106d9ca607a1c7bf48a848a3b0847" exitCode=7 Jul 17 06:19:44.543987 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:44.543931 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" event={"ID":"575565fd-e8ae-4788-8111-eb9b15ded6f1","Type":"ContainerDied","Data":"02aadefc97b0f4d2964c6989a3fd7521a73106d9ca607a1c7bf48a848a3b0847"} Jul 17 06:19:44.544484 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:44.544024 2580 scope.go:117] "RemoveContainer" containerID="67157b3909982d8d67e45fcf7102deaec2afd26060b5b11d9846af2be7c2195a" Jul 17 06:19:44.544484 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:44.544221 2580 scope.go:117] "RemoveContainer" containerID="02aadefc97b0f4d2964c6989a3fd7521a73106d9ca607a1c7bf48a848a3b0847" Jul 17 06:19:44.544590 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:19:44.544511 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-r22tb_odh-ai-gateway-infra(575565fd-e8ae-4788-8111-eb9b15ded6f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" podUID="575565fd-e8ae-4788-8111-eb9b15ded6f1" Jul 17 06:19:45.550455 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:45.550426 2580 scope.go:117] "RemoveContainer" containerID="02aadefc97b0f4d2964c6989a3fd7521a73106d9ca607a1c7bf48a848a3b0847" Jul 17 06:19:45.550894 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:19:45.550669 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-r22tb_odh-ai-gateway-infra(575565fd-e8ae-4788-8111-eb9b15ded6f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" podUID="575565fd-e8ae-4788-8111-eb9b15ded6f1" Jul 17 06:19:45.884706 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:45.884679 2580 scope.go:117] "RemoveContainer" containerID="ec7bf73ead1c8c72925e817b8bdc6c4e9ee43f053bb3303d46362110a245081e" Jul 17 06:19:46.556375 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:46.556343 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" event={"ID":"a65b4941-22fa-4d95-8c9d-43fc877c071e","Type":"ContainerStarted","Data":"6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f"} Jul 17 06:19:46.556765 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:46.556575 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:19:47.563020 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:47.562979 2580 generic.go:356] "Generic (PLEG): container finished" podID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" exitCode=1 Jul 17 06:19:47.563496 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:47.563069 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" event={"ID":"a65b4941-22fa-4d95-8c9d-43fc877c071e","Type":"ContainerDied","Data":"6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f"} Jul 17 06:19:47.563496 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:47.563141 2580 scope.go:117] "RemoveContainer" containerID="ec7bf73ead1c8c72925e817b8bdc6c4e9ee43f053bb3303d46362110a245081e" Jul 17 06:19:47.563496 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:47.563320 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:19:47.563623 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:19:47.563572 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:19:48.568795 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:48.568763 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:19:48.569280 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:19:48.568936 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:19:48.884179 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:48.884143 2580 scope.go:117] "RemoveContainer" containerID="6cad555ea9ca78aaea30f7d7f261d7682a065085ee3a655fbe6c8c50a9e69fb4" Jul 17 06:19:49.575488 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:49.575455 2580 generic.go:356] "Generic (PLEG): container finished" podID="ab79bb8a-d52e-4470-887e-397898788798" containerID="9e5d5368bb96c8b3183e452e17a0c056eaf9e36ab3b5fd4ce26efda65e1c0443" exitCode=6 Jul 17 06:19:49.575908 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:49.575526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" event={"ID":"ab79bb8a-d52e-4470-887e-397898788798","Type":"ContainerDied","Data":"9e5d5368bb96c8b3183e452e17a0c056eaf9e36ab3b5fd4ce26efda65e1c0443"} Jul 17 06:19:49.575908 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:49.575587 2580 scope.go:117] "RemoveContainer" containerID="6cad555ea9ca78aaea30f7d7f261d7682a065085ee3a655fbe6c8c50a9e69fb4" Jul 17 06:19:49.575908 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:49.575884 2580 scope.go:117] "RemoveContainer" containerID="9e5d5368bb96c8b3183e452e17a0c056eaf9e36ab3b5fd4ce26efda65e1c0443" Jul 17 06:19:49.576080 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:19:49.576055 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l_odh-ai-gateway-infra(ab79bb8a-d52e-4470-887e-397898788798)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" podUID="ab79bb8a-d52e-4470-887e-397898788798" Jul 17 06:19:50.617350 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:50.617315 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l"] Jul 17 06:19:50.749407 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:50.749369 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" Jul 17 06:19:50.827503 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:50.827461 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab79bb8a-d52e-4470-887e-397898788798-maas-api-service-ca\") pod \"ab79bb8a-d52e-4470-887e-397898788798\" (UID: \"ab79bb8a-d52e-4470-887e-397898788798\") " Jul 17 06:19:50.827901 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:50.827870 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab79bb8a-d52e-4470-887e-397898788798-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ab79bb8a-d52e-4470-887e-397898788798" (UID: "ab79bb8a-d52e-4470-887e-397898788798"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:19:50.928931 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:50.928893 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab79bb8a-d52e-4470-887e-397898788798-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:19:51.586263 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:51.586234 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" Jul 17 06:19:51.586263 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:51.586256 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l" event={"ID":"ab79bb8a-d52e-4470-887e-397898788798","Type":"ContainerDied","Data":"ab330cc3b6efb71a21ec18f8f8392f911c2b1a4df2d9b5ef51f2f7b3f4f7c014"} Jul 17 06:19:51.586516 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:51.586316 2580 scope.go:117] "RemoveContainer" containerID="9e5d5368bb96c8b3183e452e17a0c056eaf9e36ab3b5fd4ce26efda65e1c0443" Jul 17 06:19:51.608249 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:51.608225 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l"] Jul 17 06:19:51.613445 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:51.613413 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hwf5l"] Jul 17 06:19:52.889347 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:52.889313 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab79bb8a-d52e-4470-887e-397898788798" path="/var/lib/kubelet/pods/ab79bb8a-d52e-4470-887e-397898788798/volumes" Jul 17 06:19:56.802579 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:56.802544 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:19:56.803022 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:19:56.803004 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:19:56.803217 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:19:56.803195 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:20:00.884090 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:00.884054 2580 scope.go:117] "RemoveContainer" containerID="02aadefc97b0f4d2964c6989a3fd7521a73106d9ca607a1c7bf48a848a3b0847" Jul 17 06:20:01.634079 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:01.634040 2580 generic.go:356] "Generic (PLEG): container finished" podID="575565fd-e8ae-4788-8111-eb9b15ded6f1" containerID="66e5c0a83fcb817f5df15d2ddf1119661014a0c53df84e146038c9a2dd8df0fd" exitCode=7 Jul 17 06:20:01.634262 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:01.634113 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" event={"ID":"575565fd-e8ae-4788-8111-eb9b15ded6f1","Type":"ContainerDied","Data":"66e5c0a83fcb817f5df15d2ddf1119661014a0c53df84e146038c9a2dd8df0fd"} Jul 17 06:20:01.634262 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:01.634202 2580 scope.go:117] "RemoveContainer" containerID="02aadefc97b0f4d2964c6989a3fd7521a73106d9ca607a1c7bf48a848a3b0847" Jul 17 06:20:01.634602 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:01.634580 2580 scope.go:117] "RemoveContainer" containerID="66e5c0a83fcb817f5df15d2ddf1119661014a0c53df84e146038c9a2dd8df0fd" Jul 17 06:20:01.634865 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:20:01.634839 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-r22tb_odh-ai-gateway-infra(575565fd-e8ae-4788-8111-eb9b15ded6f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" podUID="575565fd-e8ae-4788-8111-eb9b15ded6f1" Jul 17 06:20:02.673492 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:02.673452 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb"] Jul 17 06:20:02.808257 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:02.808229 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" Jul 17 06:20:02.937802 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:02.937698 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575565fd-e8ae-4788-8111-eb9b15ded6f1-maas-api-service-ca\") pod \"575565fd-e8ae-4788-8111-eb9b15ded6f1\" (UID: \"575565fd-e8ae-4788-8111-eb9b15ded6f1\") " Jul 17 06:20:02.938074 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:02.938049 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575565fd-e8ae-4788-8111-eb9b15ded6f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "575565fd-e8ae-4788-8111-eb9b15ded6f1" (UID: "575565fd-e8ae-4788-8111-eb9b15ded6f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:20:03.038755 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:03.038715 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/575565fd-e8ae-4788-8111-eb9b15ded6f1-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:20:03.645569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:03.645527 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" event={"ID":"575565fd-e8ae-4788-8111-eb9b15ded6f1","Type":"ContainerDied","Data":"c0d887b86de57727eaf435945f84ee10262a69960c84746f8c96fec9fa37d684"} Jul 17 06:20:03.645569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:03.645561 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb" Jul 17 06:20:03.645819 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:03.645608 2580 scope.go:117] "RemoveContainer" containerID="66e5c0a83fcb817f5df15d2ddf1119661014a0c53df84e146038c9a2dd8df0fd" Jul 17 06:20:03.668624 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:03.668594 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb"] Jul 17 06:20:03.673828 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:03.673801 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r22tb"] Jul 17 06:20:04.889296 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:04.889247 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="575565fd-e8ae-4788-8111-eb9b15ded6f1" path="/var/lib/kubelet/pods/575565fd-e8ae-4788-8111-eb9b15ded6f1/volumes" Jul 17 06:20:09.884701 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:09.884618 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:20:09.885098 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:20:09.884820 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:20:21.884582 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:21.884547 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:20:21.885059 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:20:21.884740 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:20:23.500177 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500139 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz"] Jul 17 06:20:23.500574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500492 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575565fd-e8ae-4788-8111-eb9b15ded6f1" containerName="revoke-keys" Jul 17 06:20:23.500574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500503 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="575565fd-e8ae-4788-8111-eb9b15ded6f1" containerName="revoke-keys" Jul 17 06:20:23.500574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500513 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab79bb8a-d52e-4470-887e-397898788798" containerName="revoke-keys" Jul 17 06:20:23.500574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500518 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab79bb8a-d52e-4470-887e-397898788798" containerName="revoke-keys" Jul 17 06:20:23.500574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500525 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab79bb8a-d52e-4470-887e-397898788798" containerName="revoke-keys" Jul 17 06:20:23.500574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500531 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab79bb8a-d52e-4470-887e-397898788798" containerName="revoke-keys" Jul 17 06:20:23.500574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500538 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab79bb8a-d52e-4470-887e-397898788798" containerName="revoke-keys" Jul 17 06:20:23.500574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500543 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab79bb8a-d52e-4470-887e-397898788798" containerName="revoke-keys" Jul 17 06:20:23.500574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500549 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575565fd-e8ae-4788-8111-eb9b15ded6f1" containerName="revoke-keys" Jul 17 06:20:23.500574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500554 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="575565fd-e8ae-4788-8111-eb9b15ded6f1" containerName="revoke-keys" Jul 17 06:20:23.500884 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500599 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="575565fd-e8ae-4788-8111-eb9b15ded6f1" containerName="revoke-keys" Jul 17 06:20:23.500884 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500607 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab79bb8a-d52e-4470-887e-397898788798" containerName="revoke-keys" Jul 17 06:20:23.500884 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500614 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab79bb8a-d52e-4470-887e-397898788798" containerName="revoke-keys" Jul 17 06:20:23.500884 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500620 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab79bb8a-d52e-4470-887e-397898788798" containerName="revoke-keys" Jul 17 06:20:23.500884 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500625 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="575565fd-e8ae-4788-8111-eb9b15ded6f1" containerName="revoke-keys" Jul 17 06:20:23.500884 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.500631 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="575565fd-e8ae-4788-8111-eb9b15ded6f1" containerName="revoke-keys" Jul 17 06:20:23.503458 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.503442 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" Jul 17 06:20:23.505752 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.505734 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:20:23.511273 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.511240 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz"] Jul 17 06:20:23.530570 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.530527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3a43fd-4186-4f29-be3c-9ef35b723422-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz\" (UID: \"3c3a43fd-4186-4f29-be3c-9ef35b723422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" Jul 17 06:20:23.631612 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.631572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3a43fd-4186-4f29-be3c-9ef35b723422-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz\" (UID: \"3c3a43fd-4186-4f29-be3c-9ef35b723422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" Jul 17 06:20:23.632237 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.632215 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3a43fd-4186-4f29-be3c-9ef35b723422-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz\" (UID: \"3c3a43fd-4186-4f29-be3c-9ef35b723422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" Jul 17 06:20:23.815813 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.815702 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" Jul 17 06:20:23.947659 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:23.947622 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz"] Jul 17 06:20:23.950403 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:20:23.950356 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3a43fd_4186_4f29_be3c_9ef35b723422.slice/crio-68a169d630874da6c44f0541bb957d88129d94771159c24f65249ee422e2130e WatchSource:0}: Error finding container 68a169d630874da6c44f0541bb957d88129d94771159c24f65249ee422e2130e: Status 404 returned error can't find the container with id 68a169d630874da6c44f0541bb957d88129d94771159c24f65249ee422e2130e Jul 17 06:20:24.742646 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:24.742616 2580 generic.go:356] "Generic (PLEG): container finished" podID="3c3a43fd-4186-4f29-be3c-9ef35b723422" containerID="91aa31c202e4c0c716a26a3352cc74d0aef31af8d26dce78a30f2790f2f4d5b8" exitCode=6 Jul 17 06:20:24.743046 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:24.742682 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" event={"ID":"3c3a43fd-4186-4f29-be3c-9ef35b723422","Type":"ContainerDied","Data":"91aa31c202e4c0c716a26a3352cc74d0aef31af8d26dce78a30f2790f2f4d5b8"} Jul 17 06:20:24.743046 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:24.742723 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" event={"ID":"3c3a43fd-4186-4f29-be3c-9ef35b723422","Type":"ContainerStarted","Data":"68a169d630874da6c44f0541bb957d88129d94771159c24f65249ee422e2130e"} Jul 17 06:20:24.743046 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:24.742874 2580 scope.go:117] "RemoveContainer" containerID="91aa31c202e4c0c716a26a3352cc74d0aef31af8d26dce78a30f2790f2f4d5b8" Jul 17 06:20:25.748912 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:25.748878 2580 generic.go:356] "Generic (PLEG): container finished" podID="3c3a43fd-4186-4f29-be3c-9ef35b723422" containerID="a7a45ab63d0b255319e52db9a42eafb9ee93b47d5cb4a8c7731cbbf7f5bdf5e6" exitCode=6 Jul 17 06:20:25.749441 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:25.748969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" event={"ID":"3c3a43fd-4186-4f29-be3c-9ef35b723422","Type":"ContainerDied","Data":"a7a45ab63d0b255319e52db9a42eafb9ee93b47d5cb4a8c7731cbbf7f5bdf5e6"} Jul 17 06:20:25.749441 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:25.749041 2580 scope.go:117] "RemoveContainer" containerID="91aa31c202e4c0c716a26a3352cc74d0aef31af8d26dce78a30f2790f2f4d5b8" Jul 17 06:20:25.749441 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:25.749157 2580 scope.go:117] "RemoveContainer" containerID="a7a45ab63d0b255319e52db9a42eafb9ee93b47d5cb4a8c7731cbbf7f5bdf5e6" Jul 17 06:20:25.749441 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:20:25.749419 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz_odh-ai-gateway-infra(3c3a43fd-4186-4f29-be3c-9ef35b723422)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" podUID="3c3a43fd-4186-4f29-be3c-9ef35b723422" Jul 17 06:20:26.755324 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:26.755287 2580 scope.go:117] "RemoveContainer" containerID="a7a45ab63d0b255319e52db9a42eafb9ee93b47d5cb4a8c7731cbbf7f5bdf5e6" Jul 17 06:20:26.755739 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:20:26.755533 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz_odh-ai-gateway-infra(3c3a43fd-4186-4f29-be3c-9ef35b723422)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" podUID="3c3a43fd-4186-4f29-be3c-9ef35b723422" Jul 17 06:20:36.885818 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:36.885790 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:20:36.886219 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:20:36.885964 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:20:37.228166 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:37.228049 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:20:38.425004 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:38.424964 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:20:40.884682 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:40.884651 2580 scope.go:117] "RemoveContainer" containerID="a7a45ab63d0b255319e52db9a42eafb9ee93b47d5cb4a8c7731cbbf7f5bdf5e6" Jul 17 06:20:41.825482 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:41.825372 2580 generic.go:356] "Generic (PLEG): container finished" podID="3c3a43fd-4186-4f29-be3c-9ef35b723422" containerID="a484b6a16a9bfab448734a59d2903c209a07e7e51380234ae1d3237f1eb513f8" exitCode=6 Jul 17 06:20:41.825482 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:41.825444 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" event={"ID":"3c3a43fd-4186-4f29-be3c-9ef35b723422","Type":"ContainerDied","Data":"a484b6a16a9bfab448734a59d2903c209a07e7e51380234ae1d3237f1eb513f8"} Jul 17 06:20:41.825674 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:41.825503 2580 scope.go:117] "RemoveContainer" containerID="a7a45ab63d0b255319e52db9a42eafb9ee93b47d5cb4a8c7731cbbf7f5bdf5e6" Jul 17 06:20:41.825768 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:41.825755 2580 scope.go:117] "RemoveContainer" containerID="a484b6a16a9bfab448734a59d2903c209a07e7e51380234ae1d3237f1eb513f8" Jul 17 06:20:41.825992 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:20:41.825971 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz_odh-ai-gateway-infra(3c3a43fd-4186-4f29-be3c-9ef35b723422)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" podUID="3c3a43fd-4186-4f29-be3c-9ef35b723422" Jul 17 06:20:42.616092 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:42.616051 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:20:42.726031 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:42.725995 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:20:42.871229 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:42.871139 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz"] Jul 17 06:20:43.002432 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:43.002401 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" Jul 17 06:20:43.116237 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:43.116207 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3a43fd-4186-4f29-be3c-9ef35b723422-maas-api-service-ca\") pod \"3c3a43fd-4186-4f29-be3c-9ef35b723422\" (UID: \"3c3a43fd-4186-4f29-be3c-9ef35b723422\") " Jul 17 06:20:43.116569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:43.116543 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3a43fd-4186-4f29-be3c-9ef35b723422-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c3a43fd-4186-4f29-be3c-9ef35b723422" (UID: "3c3a43fd-4186-4f29-be3c-9ef35b723422"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:20:43.216859 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:43.216781 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3a43fd-4186-4f29-be3c-9ef35b723422-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:20:43.837729 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:43.837698 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" Jul 17 06:20:43.838203 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:43.837722 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz" event={"ID":"3c3a43fd-4186-4f29-be3c-9ef35b723422","Type":"ContainerDied","Data":"68a169d630874da6c44f0541bb957d88129d94771159c24f65249ee422e2130e"} Jul 17 06:20:43.838203 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:43.837809 2580 scope.go:117] "RemoveContainer" containerID="a484b6a16a9bfab448734a59d2903c209a07e7e51380234ae1d3237f1eb513f8" Jul 17 06:20:43.865697 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:43.865669 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz"] Jul 17 06:20:43.872607 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:43.872585 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-g7lqz"] Jul 17 06:20:44.888792 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:44.888756 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c3a43fd-4186-4f29-be3c-9ef35b723422" path="/var/lib/kubelet/pods/3c3a43fd-4186-4f29-be3c-9ef35b723422/volumes" Jul 17 06:20:51.884063 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:20:51.884026 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:20:51.884505 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:20:51.884193 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:21:03.884687 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:21:03.884654 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:21:03.885145 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:21:03.884862 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:21:16.892418 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:21:16.891401 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:21:16.892418 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:21:16.891785 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:21:31.884313 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:21:31.884279 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:21:31.884764 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:21:31.884484 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:21:45.883922 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:21:45.883838 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:21:45.884309 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:21:45.884059 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:21:56.886302 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:21:56.886269 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:21:56.886813 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:21:56.886462 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:22:07.884024 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:07.883991 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:22:07.886512 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:22:07.884156 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:22:20.884373 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:20.884330 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:22:20.884887 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:22:20.884619 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:22:23.636336 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.636296 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546"] Jul 17 06:22:23.636848 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.636640 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3a43fd-4186-4f29-be3c-9ef35b723422" containerName="revoke-keys" Jul 17 06:22:23.636848 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.636652 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3a43fd-4186-4f29-be3c-9ef35b723422" containerName="revoke-keys" Jul 17 06:22:23.636848 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.636659 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3a43fd-4186-4f29-be3c-9ef35b723422" containerName="revoke-keys" Jul 17 06:22:23.636848 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.636665 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3a43fd-4186-4f29-be3c-9ef35b723422" containerName="revoke-keys" Jul 17 06:22:23.636848 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.636672 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575565fd-e8ae-4788-8111-eb9b15ded6f1" containerName="revoke-keys" Jul 17 06:22:23.636848 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.636678 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="575565fd-e8ae-4788-8111-eb9b15ded6f1" containerName="revoke-keys" Jul 17 06:22:23.636848 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.636741 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3a43fd-4186-4f29-be3c-9ef35b723422" containerName="revoke-keys" Jul 17 06:22:23.636848 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.636748 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3a43fd-4186-4f29-be3c-9ef35b723422" containerName="revoke-keys" Jul 17 06:22:23.639441 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.639419 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" Jul 17 06:22:23.643079 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.643052 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:22:23.652468 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.652436 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546"] Jul 17 06:22:23.723163 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.723118 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-hw546\" (UID: \"673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" Jul 17 06:22:23.823563 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.823524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-hw546\" (UID: \"673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" Jul 17 06:22:23.824230 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.824211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-hw546\" (UID: \"673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" Jul 17 06:22:23.949138 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:23.949039 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" Jul 17 06:22:24.081120 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:24.081084 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546"] Jul 17 06:22:24.084947 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:22:24.084916 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673b9685_5c3a_4c7f_a6dd_cb8b2a6021bd.slice/crio-c27fc62869f19a406b3c1150108cd73a545afce46f4e88e26766ed75dad3ba1e WatchSource:0}: Error finding container c27fc62869f19a406b3c1150108cd73a545afce46f4e88e26766ed75dad3ba1e: Status 404 returned error can't find the container with id c27fc62869f19a406b3c1150108cd73a545afce46f4e88e26766ed75dad3ba1e Jul 17 06:22:24.281081 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:24.281044 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" event={"ID":"673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd","Type":"ContainerStarted","Data":"c27fc62869f19a406b3c1150108cd73a545afce46f4e88e26766ed75dad3ba1e"} Jul 17 06:22:25.287215 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:25.287177 2580 generic.go:356] "Generic (PLEG): container finished" podID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" containerID="b42dcf8fa9e80d5bcebfd0fb5d0aa7ac9f91aabdfbbae0cf836e76e3a291331e" exitCode=6 Jul 17 06:22:25.287737 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:25.287243 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" event={"ID":"673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd","Type":"ContainerDied","Data":"b42dcf8fa9e80d5bcebfd0fb5d0aa7ac9f91aabdfbbae0cf836e76e3a291331e"} Jul 17 06:22:25.287737 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:25.287560 2580 scope.go:117] "RemoveContainer" containerID="b42dcf8fa9e80d5bcebfd0fb5d0aa7ac9f91aabdfbbae0cf836e76e3a291331e" Jul 17 06:22:26.293720 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:26.293688 2580 generic.go:356] "Generic (PLEG): container finished" podID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" containerID="eaa78435f1d1d45c1f1b2ae437660dffe8319db96f45ad6a30280126bec2ecc1" exitCode=6 Jul 17 06:22:26.294159 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:26.293742 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" event={"ID":"673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd","Type":"ContainerDied","Data":"eaa78435f1d1d45c1f1b2ae437660dffe8319db96f45ad6a30280126bec2ecc1"} Jul 17 06:22:26.294159 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:26.293799 2580 scope.go:117] "RemoveContainer" containerID="b42dcf8fa9e80d5bcebfd0fb5d0aa7ac9f91aabdfbbae0cf836e76e3a291331e" Jul 17 06:22:26.294159 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:26.293987 2580 scope.go:117] "RemoveContainer" containerID="eaa78435f1d1d45c1f1b2ae437660dffe8319db96f45ad6a30280126bec2ecc1" Jul 17 06:22:26.294291 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:22:26.294194 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-hw546_odh-ai-gateway-infra(673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" podUID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" Jul 17 06:22:27.299865 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:27.299838 2580 scope.go:117] "RemoveContainer" containerID="eaa78435f1d1d45c1f1b2ae437660dffe8319db96f45ad6a30280126bec2ecc1" Jul 17 06:22:27.300270 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:22:27.300044 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-hw546_odh-ai-gateway-infra(673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" podUID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" Jul 17 06:22:33.884781 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:33.884746 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:22:33.885378 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:33.885153 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 06:22:35.340930 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:35.340897 2580 generic.go:356] "Generic (PLEG): container finished" podID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerID="a20c99b95b89af91445845ba1f2ac1b6efef7960b55cf918d551701d507906ca" exitCode=1 Jul 17 06:22:35.340930 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:35.340932 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" event={"ID":"a65b4941-22fa-4d95-8c9d-43fc877c071e","Type":"ContainerDied","Data":"a20c99b95b89af91445845ba1f2ac1b6efef7960b55cf918d551701d507906ca"} Jul 17 06:22:35.341454 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:35.340982 2580 scope.go:117] "RemoveContainer" containerID="6d5f2def7a7d815b3ebb6dfd539de44d1623e0e0554047cde8a9e06ed02de51f" Jul 17 06:22:35.341454 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:35.341269 2580 scope.go:117] "RemoveContainer" containerID="a20c99b95b89af91445845ba1f2ac1b6efef7960b55cf918d551701d507906ca" Jul 17 06:22:35.341545 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:22:35.341468 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:22:36.556895 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:36.556841 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:22:36.557328 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:36.557215 2580 scope.go:117] "RemoveContainer" containerID="a20c99b95b89af91445845ba1f2ac1b6efef7960b55cf918d551701d507906ca" Jul 17 06:22:36.557440 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:22:36.557419 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:22:36.802246 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:36.802197 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:22:37.351701 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:37.351658 2580 scope.go:117] "RemoveContainer" containerID="a20c99b95b89af91445845ba1f2ac1b6efef7960b55cf918d551701d507906ca" Jul 17 06:22:37.351922 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:22:37.351896 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:22:41.883743 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:41.883709 2580 scope.go:117] "RemoveContainer" containerID="eaa78435f1d1d45c1f1b2ae437660dffe8319db96f45ad6a30280126bec2ecc1" Jul 17 06:22:42.374327 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:42.374294 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" event={"ID":"673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd","Type":"ContainerStarted","Data":"bcf8513a26b8ac535b48fb9c872b284d1d6a1fa55f84d6f39e21ed47dc9d8409"} Jul 17 06:22:42.391170 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:42.391063 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" podStartSLOduration=19.391046266 podStartE2EDuration="19.391046266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:22:42.389615482 +0000 UTC m=+2487.134294852" watchObservedRunningTime="2026-07-17 06:22:42.391046266 +0000 UTC m=+2487.135725623" Jul 17 06:22:43.380689 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:43.380651 2580 generic.go:356] "Generic (PLEG): container finished" podID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" containerID="bcf8513a26b8ac535b48fb9c872b284d1d6a1fa55f84d6f39e21ed47dc9d8409" exitCode=6 Jul 17 06:22:43.380689 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:43.380690 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" event={"ID":"673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd","Type":"ContainerDied","Data":"bcf8513a26b8ac535b48fb9c872b284d1d6a1fa55f84d6f39e21ed47dc9d8409"} Jul 17 06:22:43.381189 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:43.380739 2580 scope.go:117] "RemoveContainer" containerID="eaa78435f1d1d45c1f1b2ae437660dffe8319db96f45ad6a30280126bec2ecc1" Jul 17 06:22:43.381189 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:43.381088 2580 scope.go:117] "RemoveContainer" containerID="bcf8513a26b8ac535b48fb9c872b284d1d6a1fa55f84d6f39e21ed47dc9d8409" Jul 17 06:22:43.381326 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:22:43.381304 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-hw546_odh-ai-gateway-infra(673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" podUID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" Jul 17 06:22:43.405299 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:43.405272 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546"] Jul 17 06:22:44.518158 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:44.518133 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" Jul 17 06:22:44.604894 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:44.604853 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd-maas-api-service-ca\") pod \"673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd\" (UID: \"673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd\") " Jul 17 06:22:44.605244 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:44.605217 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" (UID: "673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:22:44.706050 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:44.705964 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:22:45.392086 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.392047 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" event={"ID":"673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd","Type":"ContainerDied","Data":"c27fc62869f19a406b3c1150108cd73a545afce46f4e88e26766ed75dad3ba1e"} Jul 17 06:22:45.392086 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.392072 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546" Jul 17 06:22:45.392338 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.392122 2580 scope.go:117] "RemoveContainer" containerID="bcf8513a26b8ac535b48fb9c872b284d1d6a1fa55f84d6f39e21ed47dc9d8409" Jul 17 06:22:45.407841 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.407814 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546"] Jul 17 06:22:45.409989 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.409967 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-hw546"] Jul 17 06:22:45.792744 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.792663 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68"] Jul 17 06:22:45.793134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.793063 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" containerName="revoke-keys" Jul 17 06:22:45.793134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.793077 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" containerName="revoke-keys" Jul 17 06:22:45.793134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.793088 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" containerName="revoke-keys" Jul 17 06:22:45.793134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.793095 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" containerName="revoke-keys" Jul 17 06:22:45.793134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.793103 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3a43fd-4186-4f29-be3c-9ef35b723422" containerName="revoke-keys" Jul 17 06:22:45.793134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.793109 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3a43fd-4186-4f29-be3c-9ef35b723422" containerName="revoke-keys" Jul 17 06:22:45.793134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.793115 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" containerName="revoke-keys" Jul 17 06:22:45.793134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.793121 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" containerName="revoke-keys" Jul 17 06:22:45.793413 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.793176 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3a43fd-4186-4f29-be3c-9ef35b723422" containerName="revoke-keys" Jul 17 06:22:45.793413 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.793184 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" containerName="revoke-keys" Jul 17 06:22:45.793413 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.793191 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" containerName="revoke-keys" Jul 17 06:22:45.793413 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.793196 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" containerName="revoke-keys" Jul 17 06:22:45.797306 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.797290 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" Jul 17 06:22:45.800949 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.800925 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:22:45.804922 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.804898 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68"] Jul 17 06:22:45.917184 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:45.917136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8eef6ab-4f03-4b25-8936-b0562a7fa424-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-chz68\" (UID: \"d8eef6ab-4f03-4b25-8936-b0562a7fa424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" Jul 17 06:22:46.017986 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:46.017950 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8eef6ab-4f03-4b25-8936-b0562a7fa424-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-chz68\" (UID: \"d8eef6ab-4f03-4b25-8936-b0562a7fa424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" Jul 17 06:22:46.018617 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:46.018595 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8eef6ab-4f03-4b25-8936-b0562a7fa424-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-chz68\" (UID: \"d8eef6ab-4f03-4b25-8936-b0562a7fa424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" Jul 17 06:22:46.107970 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:46.107939 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" Jul 17 06:22:46.240175 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:46.240136 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68"] Jul 17 06:22:46.243215 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:22:46.243184 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8eef6ab_4f03_4b25_8936_b0562a7fa424.slice/crio-6eefe8b4bf4203b64a4688eb3f077605bb601bff9b60a761ba96fdfdb23a2848 WatchSource:0}: Error finding container 6eefe8b4bf4203b64a4688eb3f077605bb601bff9b60a761ba96fdfdb23a2848: Status 404 returned error can't find the container with id 6eefe8b4bf4203b64a4688eb3f077605bb601bff9b60a761ba96fdfdb23a2848 Jul 17 06:22:46.399164 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:46.399078 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" event={"ID":"d8eef6ab-4f03-4b25-8936-b0562a7fa424","Type":"ContainerStarted","Data":"6eefe8b4bf4203b64a4688eb3f077605bb601bff9b60a761ba96fdfdb23a2848"} Jul 17 06:22:46.888837 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:46.888805 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd" path="/var/lib/kubelet/pods/673b9685-5c3a-4c7f-a6dd-cb8b2a6021bd/volumes" Jul 17 06:22:47.404961 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:47.404923 2580 generic.go:356] "Generic (PLEG): container finished" podID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" containerID="e5d0dfa4a205164b20300865f6c283e932556317aadcacf969b4efa78e076d22" exitCode=6 Jul 17 06:22:47.405358 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:47.405002 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" event={"ID":"d8eef6ab-4f03-4b25-8936-b0562a7fa424","Type":"ContainerDied","Data":"e5d0dfa4a205164b20300865f6c283e932556317aadcacf969b4efa78e076d22"} Jul 17 06:22:47.405358 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:47.405227 2580 scope.go:117] "RemoveContainer" containerID="e5d0dfa4a205164b20300865f6c283e932556317aadcacf969b4efa78e076d22" Jul 17 06:22:48.411474 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:48.411353 2580 generic.go:356] "Generic (PLEG): container finished" podID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" containerID="1b9301b3c693720705a38a512b5ca5f1e0e0170135adc6dde6837597916d8dfc" exitCode=6 Jul 17 06:22:48.411474 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:48.411450 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" event={"ID":"d8eef6ab-4f03-4b25-8936-b0562a7fa424","Type":"ContainerDied","Data":"1b9301b3c693720705a38a512b5ca5f1e0e0170135adc6dde6837597916d8dfc"} Jul 17 06:22:48.411978 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:48.411511 2580 scope.go:117] "RemoveContainer" containerID="e5d0dfa4a205164b20300865f6c283e932556317aadcacf969b4efa78e076d22" Jul 17 06:22:48.411978 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:48.411745 2580 scope.go:117] "RemoveContainer" containerID="1b9301b3c693720705a38a512b5ca5f1e0e0170135adc6dde6837597916d8dfc" Jul 17 06:22:48.411978 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:22:48.411952 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-chz68_odh-ai-gateway-infra(d8eef6ab-4f03-4b25-8936-b0562a7fa424)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" podUID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" Jul 17 06:22:49.417320 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:49.417294 2580 scope.go:117] "RemoveContainer" containerID="1b9301b3c693720705a38a512b5ca5f1e0e0170135adc6dde6837597916d8dfc" Jul 17 06:22:49.417740 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:22:49.417526 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-chz68_odh-ai-gateway-infra(d8eef6ab-4f03-4b25-8936-b0562a7fa424)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" podUID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" Jul 17 06:22:50.884895 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:50.884864 2580 scope.go:117] "RemoveContainer" containerID="a20c99b95b89af91445845ba1f2ac1b6efef7960b55cf918d551701d507906ca" Jul 17 06:22:50.885276 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:22:50.885043 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm_odh-ai-gateway-infra(a65b4941-22fa-4d95-8c9d-43fc877c071e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" Jul 17 06:22:52.085637 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:52.085603 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:22:52.155801 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:52.155753 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:22:55.020448 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:55.020412 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:22:57.765257 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:57.765217 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm"] Jul 17 06:22:57.797893 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:57.797856 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:22:57.903589 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:57.903561 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:22:58.024145 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.024040 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nldwp\" (UniqueName: \"kubernetes.io/projected/a65b4941-22fa-4d95-8c9d-43fc877c071e-kube-api-access-nldwp\") pod \"a65b4941-22fa-4d95-8c9d-43fc877c071e\" (UID: \"a65b4941-22fa-4d95-8c9d-43fc877c071e\") " Jul 17 06:22:58.024145 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.024083 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65b4941-22fa-4d95-8c9d-43fc877c071e-openshift-service-ca\") pod \"a65b4941-22fa-4d95-8c9d-43fc877c071e\" (UID: \"a65b4941-22fa-4d95-8c9d-43fc877c071e\") " Jul 17 06:22:58.024427 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.024179 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65b4941-22fa-4d95-8c9d-43fc877c071e-maas-api-tls\") pod \"a65b4941-22fa-4d95-8c9d-43fc877c071e\" (UID: \"a65b4941-22fa-4d95-8c9d-43fc877c071e\") " Jul 17 06:22:58.024601 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.024574 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65b4941-22fa-4d95-8c9d-43fc877c071e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a65b4941-22fa-4d95-8c9d-43fc877c071e" (UID: "a65b4941-22fa-4d95-8c9d-43fc877c071e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:22:58.026224 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.026191 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65b4941-22fa-4d95-8c9d-43fc877c071e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a65b4941-22fa-4d95-8c9d-43fc877c071e" (UID: "a65b4941-22fa-4d95-8c9d-43fc877c071e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:22:58.026324 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.026252 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65b4941-22fa-4d95-8c9d-43fc877c071e-kube-api-access-nldwp" (OuterVolumeSpecName: "kube-api-access-nldwp") pod "a65b4941-22fa-4d95-8c9d-43fc877c071e" (UID: "a65b4941-22fa-4d95-8c9d-43fc877c071e"). InnerVolumeSpecName "kube-api-access-nldwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:22:58.124876 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.124838 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nldwp\" (UniqueName: \"kubernetes.io/projected/a65b4941-22fa-4d95-8c9d-43fc877c071e-kube-api-access-nldwp\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:22:58.124876 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.124871 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65b4941-22fa-4d95-8c9d-43fc877c071e-openshift-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:22:58.124876 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.124881 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65b4941-22fa-4d95-8c9d-43fc877c071e-maas-api-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:22:58.136401 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.136353 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:22:58.457702 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.457667 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" Jul 17 06:22:58.457969 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.457670 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm" event={"ID":"a65b4941-22fa-4d95-8c9d-43fc877c071e","Type":"ContainerDied","Data":"c68c5f474215838367c7c46bd0144f325bc744444faf2650e73cb16ee3b6b0b4"} Jul 17 06:22:58.457969 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.457806 2580 scope.go:117] "RemoveContainer" containerID="a20c99b95b89af91445845ba1f2ac1b6efef7960b55cf918d551701d507906ca" Jul 17 06:22:58.480361 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.480332 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm"] Jul 17 06:22:58.481708 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.481687 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-572ab047-577d4bb8b5-hxlfm"] Jul 17 06:22:58.889558 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:22:58.889525 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" path="/var/lib/kubelet/pods/a65b4941-22fa-4d95-8c9d-43fc877c071e/volumes" Jul 17 06:23:03.833677 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:03.833637 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:23:03.883913 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:03.883881 2580 scope.go:117] "RemoveContainer" containerID="1b9301b3c693720705a38a512b5ca5f1e0e0170135adc6dde6837597916d8dfc" Jul 17 06:23:03.932136 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:03.932106 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:23:04.150425 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:04.150350 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:23:04.485713 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:04.485681 2580 generic.go:356] "Generic (PLEG): container finished" podID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" containerID="47d39ecda21269f42c3cb35c1d33e17e9bfdbba805ae634f1f88b3f7fe2db23a" exitCode=6 Jul 17 06:23:04.485904 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:04.485742 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" event={"ID":"d8eef6ab-4f03-4b25-8936-b0562a7fa424","Type":"ContainerDied","Data":"47d39ecda21269f42c3cb35c1d33e17e9bfdbba805ae634f1f88b3f7fe2db23a"} Jul 17 06:23:04.485904 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:04.485788 2580 scope.go:117] "RemoveContainer" containerID="1b9301b3c693720705a38a512b5ca5f1e0e0170135adc6dde6837597916d8dfc" Jul 17 06:23:04.486146 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:04.486119 2580 scope.go:117] "RemoveContainer" containerID="47d39ecda21269f42c3cb35c1d33e17e9bfdbba805ae634f1f88b3f7fe2db23a" Jul 17 06:23:04.486374 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:23:04.486336 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-chz68_odh-ai-gateway-infra(d8eef6ab-4f03-4b25-8936-b0562a7fa424)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" podUID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" Jul 17 06:23:05.350944 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.350896 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:23:05.433984 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.433942 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:23:05.506272 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506238 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j"] Jul 17 06:23:05.506618 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506604 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506668 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506619 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506668 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506628 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506668 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506633 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506668 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506657 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506668 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506662 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506668 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506668 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506851 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506673 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506851 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506678 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506851 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506684 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506851 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506736 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506851 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506744 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506851 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506752 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506851 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506758 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506851 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506763 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.506851 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.506768 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:05.511179 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.511160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:05.513794 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.513769 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-51e238b9-serving-cert\"" Jul 17 06:23:05.513897 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.513790 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 06:23:05.513897 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.513820 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpsw\"" Jul 17 06:23:05.520142 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.520119 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j"] Jul 17 06:23:05.523862 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.523841 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68"] Jul 17 06:23:05.595516 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.595195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckst5\" (UniqueName: \"kubernetes.io/projected/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-kube-api-access-ckst5\") pod \"maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j\" (UID: \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:05.595516 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.595257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-openshift-service-ca\") pod \"maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j\" (UID: \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:05.595516 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.595375 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-maas-api-tls\") pod \"maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j\" (UID: \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:05.612150 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.612116 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:23:05.664603 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.664579 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" Jul 17 06:23:05.696714 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.696680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-maas-api-tls\") pod \"maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j\" (UID: \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:05.696913 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.696755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckst5\" (UniqueName: \"kubernetes.io/projected/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-kube-api-access-ckst5\") pod \"maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j\" (UID: \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:05.696913 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.696791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-openshift-service-ca\") pod \"maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j\" (UID: \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:05.697595 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.697570 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-openshift-service-ca\") pod \"maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j\" (UID: \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:05.699653 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.699628 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-maas-api-tls\") pod \"maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j\" (UID: \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:05.708142 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.708114 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckst5\" (UniqueName: \"kubernetes.io/projected/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-kube-api-access-ckst5\") pod \"maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j\" (UID: \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:05.711626 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.711597 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:23:05.797558 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.797526 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8eef6ab-4f03-4b25-8936-b0562a7fa424-maas-api-service-ca\") pod \"d8eef6ab-4f03-4b25-8936-b0562a7fa424\" (UID: \"d8eef6ab-4f03-4b25-8936-b0562a7fa424\") " Jul 17 06:23:05.797914 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.797889 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8eef6ab-4f03-4b25-8936-b0562a7fa424-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8eef6ab-4f03-4b25-8936-b0562a7fa424" (UID: "d8eef6ab-4f03-4b25-8936-b0562a7fa424"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:23:05.822673 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.822644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:05.898264 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.898234 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8eef6ab-4f03-4b25-8936-b0562a7fa424-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:23:05.958090 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:05.958062 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j"] Jul 17 06:23:05.960256 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:23:05.960224 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb301f8c_1e8f_444d_89c5_3161a7e48ffa.slice/crio-74221cf0dc103ccd752bff32d6fef4d49328f19b02499a6238ee5d32b7de9893 WatchSource:0}: Error finding container 74221cf0dc103ccd752bff32d6fef4d49328f19b02499a6238ee5d32b7de9893: Status 404 returned error can't find the container with id 74221cf0dc103ccd752bff32d6fef4d49328f19b02499a6238ee5d32b7de9893 Jul 17 06:23:06.115902 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:06.115859 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:23:06.215235 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:06.215146 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:23:06.500288 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:06.500159 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" Jul 17 06:23:06.500288 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:06.500204 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68" event={"ID":"d8eef6ab-4f03-4b25-8936-b0562a7fa424","Type":"ContainerDied","Data":"6eefe8b4bf4203b64a4688eb3f077605bb601bff9b60a761ba96fdfdb23a2848"} Jul 17 06:23:06.500288 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:06.500285 2580 scope.go:117] "RemoveContainer" containerID="47d39ecda21269f42c3cb35c1d33e17e9bfdbba805ae634f1f88b3f7fe2db23a" Jul 17 06:23:06.502237 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:06.502201 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" event={"ID":"bb301f8c-1e8f-444d-89c5-3161a7e48ffa","Type":"ContainerStarted","Data":"091e73616f021d3a5218c0f01e9a95a7e831cefbbc57a813623d9d7c7ad5731b"} Jul 17 06:23:06.502426 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:06.502261 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" event={"ID":"bb301f8c-1e8f-444d-89c5-3161a7e48ffa","Type":"ContainerStarted","Data":"74221cf0dc103ccd752bff32d6fef4d49328f19b02499a6238ee5d32b7de9893"} Jul 17 06:23:06.502426 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:06.502329 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:06.559898 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:06.559845 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podStartSLOduration=1.208113655 podStartE2EDuration="1.559827995s" totalImagesPullingTime="351.71434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:23:06.559460194 +0000 UTC m=+2511.304139546" watchObservedRunningTime="2026-07-17 06:23:06.559827995 +0000 UTC m=+2511.304507323" Jul 17 06:23:06.580589 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:06.580553 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68"] Jul 17 06:23:06.588705 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:06.588674 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-chz68"] Jul 17 06:23:06.889858 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:06.889815 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" path="/var/lib/kubelet/pods/d8eef6ab-4f03-4b25-8936-b0562a7fa424/volumes" Jul 17 06:23:07.508224 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:07.508188 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerID="091e73616f021d3a5218c0f01e9a95a7e831cefbbc57a813623d9d7c7ad5731b" exitCode=1 Jul 17 06:23:07.508703 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:07.508270 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" event={"ID":"bb301f8c-1e8f-444d-89c5-3161a7e48ffa","Type":"ContainerDied","Data":"091e73616f021d3a5218c0f01e9a95a7e831cefbbc57a813623d9d7c7ad5731b"} Jul 17 06:23:07.508703 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:07.508607 2580 scope.go:117] "RemoveContainer" containerID="091e73616f021d3a5218c0f01e9a95a7e831cefbbc57a813623d9d7c7ad5731b" Jul 17 06:23:07.986842 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:07.986807 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:23:08.515551 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:08.515460 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerID="7410eee97051fcad1d0e063d3922846fec020e6a0b23774de5ee2d3e1911e943" exitCode=1 Jul 17 06:23:08.515551 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:08.515506 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" event={"ID":"bb301f8c-1e8f-444d-89c5-3161a7e48ffa","Type":"ContainerDied","Data":"7410eee97051fcad1d0e063d3922846fec020e6a0b23774de5ee2d3e1911e943"} Jul 17 06:23:08.515999 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:08.515571 2580 scope.go:117] "RemoveContainer" containerID="091e73616f021d3a5218c0f01e9a95a7e831cefbbc57a813623d9d7c7ad5731b" Jul 17 06:23:08.515999 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:08.515811 2580 scope.go:117] "RemoveContainer" containerID="7410eee97051fcad1d0e063d3922846fec020e6a0b23774de5ee2d3e1911e943" Jul 17 06:23:08.516072 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:23:08.516030 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:23:09.521928 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.521897 2580 scope.go:117] "RemoveContainer" containerID="7410eee97051fcad1d0e063d3922846fec020e6a0b23774de5ee2d3e1911e943" Jul 17 06:23:09.522320 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:23:09.522084 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:23:09.682843 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.682804 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb"] Jul 17 06:23:09.683178 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.683165 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:09.683232 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.683180 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:09.683232 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.683188 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" containerName="revoke-keys" Jul 17 06:23:09.683232 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.683193 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" containerName="revoke-keys" Jul 17 06:23:09.683232 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.683205 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:09.683232 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.683211 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:09.683232 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.683229 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" containerName="revoke-keys" Jul 17 06:23:09.683446 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.683238 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" containerName="revoke-keys" Jul 17 06:23:09.683446 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.683291 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65b4941-22fa-4d95-8c9d-43fc877c071e" containerName="maas-api" Jul 17 06:23:09.683446 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.683299 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" containerName="revoke-keys" Jul 17 06:23:09.683446 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.683309 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" containerName="revoke-keys" Jul 17 06:23:09.687684 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.687652 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" Jul 17 06:23:09.690065 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.690038 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:23:09.695281 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.695245 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb"] Jul 17 06:23:09.832675 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.832573 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee34a62b-901e-41cb-b98b-99a352e9c4d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-72mfb\" (UID: \"ee34a62b-901e-41cb-b98b-99a352e9c4d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" Jul 17 06:23:09.933505 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.933466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee34a62b-901e-41cb-b98b-99a352e9c4d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-72mfb\" (UID: \"ee34a62b-901e-41cb-b98b-99a352e9c4d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" Jul 17 06:23:09.934102 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.934082 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee34a62b-901e-41cb-b98b-99a352e9c4d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-72mfb\" (UID: \"ee34a62b-901e-41cb-b98b-99a352e9c4d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" Jul 17 06:23:09.999150 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:09.999114 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" Jul 17 06:23:10.339031 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:10.339002 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb"] Jul 17 06:23:10.340688 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:23:10.340655 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee34a62b_901e_41cb_b98b_99a352e9c4d8.slice/crio-f3da01115c50d94a1bad26026a283893c541931a41164dd410182332bc620e86 WatchSource:0}: Error finding container f3da01115c50d94a1bad26026a283893c541931a41164dd410182332bc620e86: Status 404 returned error can't find the container with id f3da01115c50d94a1bad26026a283893c541931a41164dd410182332bc620e86 Jul 17 06:23:10.527255 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:10.527216 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" event={"ID":"ee34a62b-901e-41cb-b98b-99a352e9c4d8","Type":"ContainerStarted","Data":"f3da01115c50d94a1bad26026a283893c541931a41164dd410182332bc620e86"} Jul 17 06:23:11.502967 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:11.502926 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:11.503321 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:11.503307 2580 scope.go:117] "RemoveContainer" containerID="7410eee97051fcad1d0e063d3922846fec020e6a0b23774de5ee2d3e1911e943" Jul 17 06:23:11.503518 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:23:11.503500 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:23:11.533508 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:11.533473 2580 generic.go:356] "Generic (PLEG): container finished" podID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" containerID="c0b18c83e4eb9a42dac59f741d7a06fa78a8d080c1e3ac4ca73f8e3776a6a3c8" exitCode=7 Jul 17 06:23:11.533933 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:11.533512 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" event={"ID":"ee34a62b-901e-41cb-b98b-99a352e9c4d8","Type":"ContainerDied","Data":"c0b18c83e4eb9a42dac59f741d7a06fa78a8d080c1e3ac4ca73f8e3776a6a3c8"} Jul 17 06:23:11.533933 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:11.533693 2580 scope.go:117] "RemoveContainer" containerID="c0b18c83e4eb9a42dac59f741d7a06fa78a8d080c1e3ac4ca73f8e3776a6a3c8" Jul 17 06:23:12.540949 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:12.540824 2580 generic.go:356] "Generic (PLEG): container finished" podID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" containerID="a4165e76a1bc463d6d526bbdd10b096ad14b26ca6c1e5557c409f7d99f35c55f" exitCode=7 Jul 17 06:23:12.540949 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:12.540900 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" event={"ID":"ee34a62b-901e-41cb-b98b-99a352e9c4d8","Type":"ContainerDied","Data":"a4165e76a1bc463d6d526bbdd10b096ad14b26ca6c1e5557c409f7d99f35c55f"} Jul 17 06:23:12.541509 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:12.541015 2580 scope.go:117] "RemoveContainer" containerID="c0b18c83e4eb9a42dac59f741d7a06fa78a8d080c1e3ac4ca73f8e3776a6a3c8" Jul 17 06:23:12.541509 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:12.541271 2580 scope.go:117] "RemoveContainer" containerID="a4165e76a1bc463d6d526bbdd10b096ad14b26ca6c1e5557c409f7d99f35c55f" Jul 17 06:23:12.541644 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:23:12.541566 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-72mfb_odh-ai-gateway-infra(ee34a62b-901e-41cb-b98b-99a352e9c4d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" podUID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" Jul 17 06:23:13.547098 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:13.547067 2580 scope.go:117] "RemoveContainer" containerID="a4165e76a1bc463d6d526bbdd10b096ad14b26ca6c1e5557c409f7d99f35c55f" Jul 17 06:23:13.547521 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:23:13.547275 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-72mfb_odh-ai-gateway-infra(ee34a62b-901e-41cb-b98b-99a352e9c4d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" podUID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" Jul 17 06:23:15.823153 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:15.823120 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:15.823577 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:15.823507 2580 scope.go:117] "RemoveContainer" containerID="7410eee97051fcad1d0e063d3922846fec020e6a0b23774de5ee2d3e1911e943" Jul 17 06:23:15.823722 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:23:15.823702 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:23:25.884726 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:25.884686 2580 scope.go:117] "RemoveContainer" containerID="7410eee97051fcad1d0e063d3922846fec020e6a0b23774de5ee2d3e1911e943" Jul 17 06:23:26.610782 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:26.610748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" event={"ID":"bb301f8c-1e8f-444d-89c5-3161a7e48ffa","Type":"ContainerStarted","Data":"bb97167f8041ea8e7f6d7de0608f250070f8747af74aa13c0aeb2ef2a675b803"} Jul 17 06:23:26.610977 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:26.610956 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:27.617188 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:27.617148 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerID="bb97167f8041ea8e7f6d7de0608f250070f8747af74aa13c0aeb2ef2a675b803" exitCode=1 Jul 17 06:23:27.617719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:27.617235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" event={"ID":"bb301f8c-1e8f-444d-89c5-3161a7e48ffa","Type":"ContainerDied","Data":"bb97167f8041ea8e7f6d7de0608f250070f8747af74aa13c0aeb2ef2a675b803"} Jul 17 06:23:27.617719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:27.617298 2580 scope.go:117] "RemoveContainer" containerID="7410eee97051fcad1d0e063d3922846fec020e6a0b23774de5ee2d3e1911e943" Jul 17 06:23:27.617719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:27.617570 2580 scope.go:117] "RemoveContainer" containerID="bb97167f8041ea8e7f6d7de0608f250070f8747af74aa13c0aeb2ef2a675b803" Jul 17 06:23:27.617872 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:23:27.617805 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:23:28.623617 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:28.623592 2580 scope.go:117] "RemoveContainer" containerID="bb97167f8041ea8e7f6d7de0608f250070f8747af74aa13c0aeb2ef2a675b803" Jul 17 06:23:28.624009 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:23:28.623798 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:23:28.883919 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:28.883838 2580 scope.go:117] "RemoveContainer" containerID="a4165e76a1bc463d6d526bbdd10b096ad14b26ca6c1e5557c409f7d99f35c55f" Jul 17 06:23:29.629210 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:29.629177 2580 generic.go:356] "Generic (PLEG): container finished" podID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" containerID="0d083ab5450a97cbfe28da1377a0aa6ccb7f6ebcffd0fd3215310f7dbfe0c168" exitCode=7 Jul 17 06:23:29.629661 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:29.629252 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" event={"ID":"ee34a62b-901e-41cb-b98b-99a352e9c4d8","Type":"ContainerDied","Data":"0d083ab5450a97cbfe28da1377a0aa6ccb7f6ebcffd0fd3215310f7dbfe0c168"} Jul 17 06:23:29.629661 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:29.629311 2580 scope.go:117] "RemoveContainer" containerID="a4165e76a1bc463d6d526bbdd10b096ad14b26ca6c1e5557c409f7d99f35c55f" Jul 17 06:23:29.629661 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:29.629585 2580 scope.go:117] "RemoveContainer" containerID="0d083ab5450a97cbfe28da1377a0aa6ccb7f6ebcffd0fd3215310f7dbfe0c168" Jul 17 06:23:29.629861 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:23:29.629835 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-72mfb_odh-ai-gateway-infra(ee34a62b-901e-41cb-b98b-99a352e9c4d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" podUID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" Jul 17 06:23:30.657674 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:30.657632 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb"] Jul 17 06:23:30.793161 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:30.793136 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" Jul 17 06:23:30.908466 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:30.908356 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee34a62b-901e-41cb-b98b-99a352e9c4d8-maas-api-service-ca\") pod \"ee34a62b-901e-41cb-b98b-99a352e9c4d8\" (UID: \"ee34a62b-901e-41cb-b98b-99a352e9c4d8\") " Jul 17 06:23:30.908724 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:30.908699 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee34a62b-901e-41cb-b98b-99a352e9c4d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ee34a62b-901e-41cb-b98b-99a352e9c4d8" (UID: "ee34a62b-901e-41cb-b98b-99a352e9c4d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:23:31.009413 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:31.009341 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee34a62b-901e-41cb-b98b-99a352e9c4d8-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:23:31.640658 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:31.640614 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" event={"ID":"ee34a62b-901e-41cb-b98b-99a352e9c4d8","Type":"ContainerDied","Data":"f3da01115c50d94a1bad26026a283893c541931a41164dd410182332bc620e86"} Jul 17 06:23:31.640658 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:31.640651 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb" Jul 17 06:23:31.640886 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:31.640696 2580 scope.go:117] "RemoveContainer" containerID="0d083ab5450a97cbfe28da1377a0aa6ccb7f6ebcffd0fd3215310f7dbfe0c168" Jul 17 06:23:31.662875 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:31.662839 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb"] Jul 17 06:23:31.668546 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:31.668517 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-72mfb"] Jul 17 06:23:32.889591 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:32.889544 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" path="/var/lib/kubelet/pods/ee34a62b-901e-41cb-b98b-99a352e9c4d8/volumes" Jul 17 06:23:35.823642 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:35.823600 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:35.824084 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:35.824064 2580 scope.go:117] "RemoveContainer" containerID="bb97167f8041ea8e7f6d7de0608f250070f8747af74aa13c0aeb2ef2a675b803" Jul 17 06:23:35.824296 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:23:35.824275 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:23:36.861208 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:36.861169 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:23:49.884165 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:49.884135 2580 scope.go:117] "RemoveContainer" containerID="bb97167f8041ea8e7f6d7de0608f250070f8747af74aa13c0aeb2ef2a675b803" Jul 17 06:23:50.726218 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:50.726184 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerID="4a17ac32abfd1d654aa6a00356971a80ce9f4a46024a0542e56ba25da0ec00b6" exitCode=1 Jul 17 06:23:50.726438 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:50.726258 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" event={"ID":"bb301f8c-1e8f-444d-89c5-3161a7e48ffa","Type":"ContainerDied","Data":"4a17ac32abfd1d654aa6a00356971a80ce9f4a46024a0542e56ba25da0ec00b6"} Jul 17 06:23:50.726438 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:50.726324 2580 scope.go:117] "RemoveContainer" containerID="bb97167f8041ea8e7f6d7de0608f250070f8747af74aa13c0aeb2ef2a675b803" Jul 17 06:23:50.726704 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:50.726687 2580 scope.go:117] "RemoveContainer" containerID="4a17ac32abfd1d654aa6a00356971a80ce9f4a46024a0542e56ba25da0ec00b6" Jul 17 06:23:50.726942 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:23:50.726922 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:23:51.612066 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:51.612029 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:51.732129 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:51.732092 2580 scope.go:117] "RemoveContainer" containerID="4a17ac32abfd1d654aa6a00356971a80ce9f4a46024a0542e56ba25da0ec00b6" Jul 17 06:23:51.732315 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:23:51.732295 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:23:55.823572 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:55.823533 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:23:55.824007 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:23:55.823969 2580 scope.go:117] "RemoveContainer" containerID="4a17ac32abfd1d654aa6a00356971a80ce9f4a46024a0542e56ba25da0ec00b6" Jul 17 06:23:55.824159 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:23:55.824136 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:24:07.883944 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:07.883908 2580 scope.go:117] "RemoveContainer" containerID="4a17ac32abfd1d654aa6a00356971a80ce9f4a46024a0542e56ba25da0ec00b6" Jul 17 06:24:07.884497 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:24:07.884096 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:24:18.622764 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:18.622719 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:24:18.743308 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:18.743269 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:24:18.884808 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:18.884780 2580 scope.go:117] "RemoveContainer" containerID="4a17ac32abfd1d654aa6a00356971a80ce9f4a46024a0542e56ba25da0ec00b6" Jul 17 06:24:18.884999 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:24:18.884978 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:24:22.100974 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:22.100942 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:24:30.889915 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:30.889885 2580 scope.go:117] "RemoveContainer" containerID="4a17ac32abfd1d654aa6a00356971a80ce9f4a46024a0542e56ba25da0ec00b6" Jul 17 06:24:31.914255 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:31.914212 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerID="36bcd38c7c7b870cad2099f5d3004451ee04fc7b2c85e0030a54d1a1a8bcad5a" exitCode=1 Jul 17 06:24:31.914873 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:31.914284 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" event={"ID":"bb301f8c-1e8f-444d-89c5-3161a7e48ffa","Type":"ContainerDied","Data":"36bcd38c7c7b870cad2099f5d3004451ee04fc7b2c85e0030a54d1a1a8bcad5a"} Jul 17 06:24:31.914873 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:31.914366 2580 scope.go:117] "RemoveContainer" containerID="4a17ac32abfd1d654aa6a00356971a80ce9f4a46024a0542e56ba25da0ec00b6" Jul 17 06:24:31.914873 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:31.914830 2580 scope.go:117] "RemoveContainer" containerID="36bcd38c7c7b870cad2099f5d3004451ee04fc7b2c85e0030a54d1a1a8bcad5a" Jul 17 06:24:31.915114 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:24:31.915079 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:24:35.823163 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.823105 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:24:35.823787 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.823678 2580 scope.go:117] "RemoveContainer" containerID="36bcd38c7c7b870cad2099f5d3004451ee04fc7b2c85e0030a54d1a1a8bcad5a" Jul 17 06:24:35.823952 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:24:35.823927 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:24:35.898362 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.898325 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b"] Jul 17 06:24:35.898676 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.898663 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" containerName="revoke-keys" Jul 17 06:24:35.898722 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.898677 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" containerName="revoke-keys" Jul 17 06:24:35.898722 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.898687 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" containerName="revoke-keys" Jul 17 06:24:35.898722 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.898694 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" containerName="revoke-keys" Jul 17 06:24:35.898722 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.898701 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" containerName="revoke-keys" Jul 17 06:24:35.898722 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.898706 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" containerName="revoke-keys" Jul 17 06:24:35.898875 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.898724 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" containerName="revoke-keys" Jul 17 06:24:35.898875 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.898730 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" containerName="revoke-keys" Jul 17 06:24:35.898875 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.898777 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" containerName="revoke-keys" Jul 17 06:24:35.898875 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.898785 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8eef6ab-4f03-4b25-8936-b0562a7fa424" containerName="revoke-keys" Jul 17 06:24:35.898875 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.898795 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" containerName="revoke-keys" Jul 17 06:24:35.902813 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.902793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" Jul 17 06:24:35.905303 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.905280 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:24:35.909613 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.909364 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b"] Jul 17 06:24:35.995885 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:35.995846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b\" (UID: \"1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" Jul 17 06:24:36.097238 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:36.097136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b\" (UID: \"1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" Jul 17 06:24:36.097813 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:36.097791 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b\" (UID: \"1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" Jul 17 06:24:36.214888 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:36.214849 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" Jul 17 06:24:36.346244 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:36.346219 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b"] Jul 17 06:24:36.348540 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:24:36.348480 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8f5d43_ac48_4353_beb5_bb8ebaea7f3f.slice/crio-6e5d799f7110807032cc21e14569463212e18f574fea5bfbafcab87d23d5485e WatchSource:0}: Error finding container 6e5d799f7110807032cc21e14569463212e18f574fea5bfbafcab87d23d5485e: Status 404 returned error can't find the container with id 6e5d799f7110807032cc21e14569463212e18f574fea5bfbafcab87d23d5485e Jul 17 06:24:36.611411 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:36.611363 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:24:36.611798 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:36.611784 2580 scope.go:117] "RemoveContainer" containerID="36bcd38c7c7b870cad2099f5d3004451ee04fc7b2c85e0030a54d1a1a8bcad5a" Jul 17 06:24:36.611988 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:24:36.611967 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:24:36.940246 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:36.940159 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" event={"ID":"1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f","Type":"ContainerStarted","Data":"315b0481899da2c884d646605e499934876baefc323c4fa4a7765fd3413b3d2f"} Jul 17 06:24:36.940246 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:36.940215 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" event={"ID":"1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f","Type":"ContainerStarted","Data":"6e5d799f7110807032cc21e14569463212e18f574fea5bfbafcab87d23d5485e"} Jul 17 06:24:36.957570 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:36.957511 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" podStartSLOduration=1.957493605 podStartE2EDuration="1.957493605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:24:36.956108038 +0000 UTC m=+2601.700787366" watchObservedRunningTime="2026-07-17 06:24:36.957493605 +0000 UTC m=+2601.702172933" Jul 17 06:24:37.946854 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:37.946821 2580 generic.go:356] "Generic (PLEG): container finished" podID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" containerID="315b0481899da2c884d646605e499934876baefc323c4fa4a7765fd3413b3d2f" exitCode=6 Jul 17 06:24:37.947257 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:37.946907 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" event={"ID":"1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f","Type":"ContainerDied","Data":"315b0481899da2c884d646605e499934876baefc323c4fa4a7765fd3413b3d2f"} Jul 17 06:24:37.947257 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:37.947061 2580 scope.go:117] "RemoveContainer" containerID="315b0481899da2c884d646605e499934876baefc323c4fa4a7765fd3413b3d2f" Jul 17 06:24:38.953144 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:38.953047 2580 generic.go:356] "Generic (PLEG): container finished" podID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" containerID="623a4f23785efd25b2052037fc94cc1952cd6993e1584eee96cdc2c792bd6218" exitCode=6 Jul 17 06:24:38.953144 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:38.953132 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" event={"ID":"1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f","Type":"ContainerDied","Data":"623a4f23785efd25b2052037fc94cc1952cd6993e1584eee96cdc2c792bd6218"} Jul 17 06:24:38.953783 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:38.953195 2580 scope.go:117] "RemoveContainer" containerID="315b0481899da2c884d646605e499934876baefc323c4fa4a7765fd3413b3d2f" Jul 17 06:24:38.953783 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:38.953416 2580 scope.go:117] "RemoveContainer" containerID="623a4f23785efd25b2052037fc94cc1952cd6993e1584eee96cdc2c792bd6218" Jul 17 06:24:38.953783 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:24:38.953632 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b_odh-ai-gateway-infra(1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" podUID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" Jul 17 06:24:39.959237 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:39.959206 2580 scope.go:117] "RemoveContainer" containerID="623a4f23785efd25b2052037fc94cc1952cd6993e1584eee96cdc2c792bd6218" Jul 17 06:24:39.959659 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:24:39.959409 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b_odh-ai-gateway-infra(1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" podUID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" Jul 17 06:24:47.347779 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:47.347741 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:24:48.884365 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:48.884330 2580 scope.go:117] "RemoveContainer" containerID="36bcd38c7c7b870cad2099f5d3004451ee04fc7b2c85e0030a54d1a1a8bcad5a" Jul 17 06:24:48.884785 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:24:48.884527 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:24:50.884204 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:50.884172 2580 scope.go:117] "RemoveContainer" containerID="623a4f23785efd25b2052037fc94cc1952cd6993e1584eee96cdc2c792bd6218" Jul 17 06:24:52.013032 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:52.012999 2580 generic.go:356] "Generic (PLEG): container finished" podID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" containerID="15900f354eb18be91e447f72a3b9bffe6245b7950bb59d047ec892ade3755c6d" exitCode=6 Jul 17 06:24:52.013516 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:52.013058 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" event={"ID":"1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f","Type":"ContainerDied","Data":"15900f354eb18be91e447f72a3b9bffe6245b7950bb59d047ec892ade3755c6d"} Jul 17 06:24:52.013516 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:52.013107 2580 scope.go:117] "RemoveContainer" containerID="623a4f23785efd25b2052037fc94cc1952cd6993e1584eee96cdc2c792bd6218" Jul 17 06:24:52.013516 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:52.013409 2580 scope.go:117] "RemoveContainer" containerID="15900f354eb18be91e447f72a3b9bffe6245b7950bb59d047ec892ade3755c6d" Jul 17 06:24:52.013630 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:24:52.013612 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b_odh-ai-gateway-infra(1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" podUID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" Jul 17 06:24:53.040641 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:53.040607 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b"] Jul 17 06:24:53.174118 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:53.174090 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" Jul 17 06:24:53.252648 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:53.252614 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f-maas-api-service-ca\") pod \"1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f\" (UID: \"1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f\") " Jul 17 06:24:53.253016 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:53.252990 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" (UID: "1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:24:53.353760 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:53.353726 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:24:54.030867 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:54.030842 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" Jul 17 06:24:54.030867 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:54.030857 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b" event={"ID":"1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f","Type":"ContainerDied","Data":"6e5d799f7110807032cc21e14569463212e18f574fea5bfbafcab87d23d5485e"} Jul 17 06:24:54.031154 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:54.030922 2580 scope.go:117] "RemoveContainer" containerID="15900f354eb18be91e447f72a3b9bffe6245b7950bb59d047ec892ade3755c6d" Jul 17 06:24:54.052627 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:54.052598 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b"] Jul 17 06:24:54.058308 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:54.058279 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rrv2b"] Jul 17 06:24:54.889248 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:54.889206 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" path="/var/lib/kubelet/pods/1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f/volumes" Jul 17 06:24:59.784267 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.784221 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f"] Jul 17 06:24:59.784724 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.784573 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" containerName="revoke-keys" Jul 17 06:24:59.784724 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.784585 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" containerName="revoke-keys" Jul 17 06:24:59.784724 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.784597 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" containerName="revoke-keys" Jul 17 06:24:59.784724 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.784602 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" containerName="revoke-keys" Jul 17 06:24:59.784724 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.784609 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" containerName="revoke-keys" Jul 17 06:24:59.784724 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.784615 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" containerName="revoke-keys" Jul 17 06:24:59.784724 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.784682 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" containerName="revoke-keys" Jul 17 06:24:59.784724 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.784690 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" containerName="revoke-keys" Jul 17 06:24:59.784724 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.784699 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8f5d43-ac48-4353-beb5-bb8ebaea7f3f" containerName="revoke-keys" Jul 17 06:24:59.784724 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.784707 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee34a62b-901e-41cb-b98b-99a352e9c4d8" containerName="revoke-keys" Jul 17 06:24:59.788834 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.788811 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" Jul 17 06:24:59.791365 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.791335 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:24:59.797267 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.797244 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f"] Jul 17 06:24:59.809569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.809537 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2270dbb6-3038-487a-bd35-b801a741a695-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-qf64f\" (UID: \"2270dbb6-3038-487a-bd35-b801a741a695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" Jul 17 06:24:59.911024 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.910976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2270dbb6-3038-487a-bd35-b801a741a695-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-qf64f\" (UID: \"2270dbb6-3038-487a-bd35-b801a741a695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" Jul 17 06:24:59.911629 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:24:59.911609 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2270dbb6-3038-487a-bd35-b801a741a695-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-qf64f\" (UID: \"2270dbb6-3038-487a-bd35-b801a741a695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" Jul 17 06:25:00.100361 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:00.100199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" Jul 17 06:25:00.231435 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:00.231407 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f"] Jul 17 06:25:00.234059 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:25:00.234031 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2270dbb6_3038_487a_bd35_b801a741a695.slice/crio-d49dddee870559e45b1cd8a06067aa6bb5dcfe22b70ce6862cfa40a9bf242e56 WatchSource:0}: Error finding container d49dddee870559e45b1cd8a06067aa6bb5dcfe22b70ce6862cfa40a9bf242e56: Status 404 returned error can't find the container with id d49dddee870559e45b1cd8a06067aa6bb5dcfe22b70ce6862cfa40a9bf242e56 Jul 17 06:25:00.883888 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:00.883860 2580 scope.go:117] "RemoveContainer" containerID="36bcd38c7c7b870cad2099f5d3004451ee04fc7b2c85e0030a54d1a1a8bcad5a" Jul 17 06:25:00.884264 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:25:00.884046 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:25:01.063403 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:01.063357 2580 generic.go:356] "Generic (PLEG): container finished" podID="2270dbb6-3038-487a-bd35-b801a741a695" containerID="86c5c82573de31757c39ee5f5391328d80f409af4d2cc88590a8bd1b7d0fd755" exitCode=7 Jul 17 06:25:01.063604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:01.063440 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" event={"ID":"2270dbb6-3038-487a-bd35-b801a741a695","Type":"ContainerDied","Data":"86c5c82573de31757c39ee5f5391328d80f409af4d2cc88590a8bd1b7d0fd755"} Jul 17 06:25:01.063604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:01.063491 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" event={"ID":"2270dbb6-3038-487a-bd35-b801a741a695","Type":"ContainerStarted","Data":"d49dddee870559e45b1cd8a06067aa6bb5dcfe22b70ce6862cfa40a9bf242e56"} Jul 17 06:25:01.063730 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:01.063655 2580 scope.go:117] "RemoveContainer" containerID="86c5c82573de31757c39ee5f5391328d80f409af4d2cc88590a8bd1b7d0fd755" Jul 17 06:25:02.069356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:02.069262 2580 generic.go:356] "Generic (PLEG): container finished" podID="2270dbb6-3038-487a-bd35-b801a741a695" containerID="dd43de235901d8bd19cf366586dac82f7f2be3a713f842897c8c7a063e989bf1" exitCode=7 Jul 17 06:25:02.069356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:02.069324 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" event={"ID":"2270dbb6-3038-487a-bd35-b801a741a695","Type":"ContainerDied","Data":"dd43de235901d8bd19cf366586dac82f7f2be3a713f842897c8c7a063e989bf1"} Jul 17 06:25:02.069804 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:02.069415 2580 scope.go:117] "RemoveContainer" containerID="86c5c82573de31757c39ee5f5391328d80f409af4d2cc88590a8bd1b7d0fd755" Jul 17 06:25:02.069804 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:02.069642 2580 scope.go:117] "RemoveContainer" containerID="dd43de235901d8bd19cf366586dac82f7f2be3a713f842897c8c7a063e989bf1" Jul 17 06:25:02.069906 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:25:02.069884 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-qf64f_odh-ai-gateway-infra(2270dbb6-3038-487a-bd35-b801a741a695)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" podUID="2270dbb6-3038-487a-bd35-b801a741a695" Jul 17 06:25:03.075496 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:03.075468 2580 scope.go:117] "RemoveContainer" containerID="dd43de235901d8bd19cf366586dac82f7f2be3a713f842897c8c7a063e989bf1" Jul 17 06:25:03.075889 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:25:03.075648 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-qf64f_odh-ai-gateway-infra(2270dbb6-3038-487a-bd35-b801a741a695)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" podUID="2270dbb6-3038-487a-bd35-b801a741a695" Jul 17 06:25:15.884470 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:15.884426 2580 scope.go:117] "RemoveContainer" containerID="36bcd38c7c7b870cad2099f5d3004451ee04fc7b2c85e0030a54d1a1a8bcad5a" Jul 17 06:25:15.884948 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:25:15.884629 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:25:16.885832 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:16.885796 2580 scope.go:117] "RemoveContainer" containerID="dd43de235901d8bd19cf366586dac82f7f2be3a713f842897c8c7a063e989bf1" Jul 17 06:25:17.138633 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:17.138538 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" event={"ID":"2270dbb6-3038-487a-bd35-b801a741a695","Type":"ContainerStarted","Data":"2408f4429821fbe00764ba35833f146dea9f94c097f5e6779cbad120d57879c6"} Jul 17 06:25:17.155115 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:17.155057 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" podStartSLOduration=18.155040118 podStartE2EDuration="18.155040118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:25:17.154650406 +0000 UTC m=+2641.899329736" watchObservedRunningTime="2026-07-17 06:25:17.155040118 +0000 UTC m=+2641.899719446" Jul 17 06:25:17.209754 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:17.209715 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:25:17.327464 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:17.327428 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:25:18.148804 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:18.148768 2580 generic.go:356] "Generic (PLEG): container finished" podID="2270dbb6-3038-487a-bd35-b801a741a695" containerID="2408f4429821fbe00764ba35833f146dea9f94c097f5e6779cbad120d57879c6" exitCode=7 Jul 17 06:25:18.149219 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:18.148813 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" event={"ID":"2270dbb6-3038-487a-bd35-b801a741a695","Type":"ContainerDied","Data":"2408f4429821fbe00764ba35833f146dea9f94c097f5e6779cbad120d57879c6"} Jul 17 06:25:18.149219 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:18.148869 2580 scope.go:117] "RemoveContainer" containerID="dd43de235901d8bd19cf366586dac82f7f2be3a713f842897c8c7a063e989bf1" Jul 17 06:25:18.149219 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:18.149175 2580 scope.go:117] "RemoveContainer" containerID="2408f4429821fbe00764ba35833f146dea9f94c097f5e6779cbad120d57879c6" Jul 17 06:25:18.149451 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:25:18.149427 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-qf64f_odh-ai-gateway-infra(2270dbb6-3038-487a-bd35-b801a741a695)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" podUID="2270dbb6-3038-487a-bd35-b801a741a695" Jul 17 06:25:18.164747 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:18.164722 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f"] Jul 17 06:25:19.273676 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:19.273637 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" Jul 17 06:25:19.390719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:19.390683 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2270dbb6-3038-487a-bd35-b801a741a695-maas-api-service-ca\") pod \"2270dbb6-3038-487a-bd35-b801a741a695\" (UID: \"2270dbb6-3038-487a-bd35-b801a741a695\") " Jul 17 06:25:19.391079 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:19.391054 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2270dbb6-3038-487a-bd35-b801a741a695-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2270dbb6-3038-487a-bd35-b801a741a695" (UID: "2270dbb6-3038-487a-bd35-b801a741a695"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:25:19.492157 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:19.492066 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2270dbb6-3038-487a-bd35-b801a741a695-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:25:20.159293 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:20.159265 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" Jul 17 06:25:20.159487 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:20.159298 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f" event={"ID":"2270dbb6-3038-487a-bd35-b801a741a695","Type":"ContainerDied","Data":"d49dddee870559e45b1cd8a06067aa6bb5dcfe22b70ce6862cfa40a9bf242e56"} Jul 17 06:25:20.159487 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:20.159363 2580 scope.go:117] "RemoveContainer" containerID="2408f4429821fbe00764ba35833f146dea9f94c097f5e6779cbad120d57879c6" Jul 17 06:25:20.183686 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:20.183651 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f"] Jul 17 06:25:20.189682 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:20.189652 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-qf64f"] Jul 17 06:25:20.889646 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:20.889607 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2270dbb6-3038-487a-bd35-b801a741a695" path="/var/lib/kubelet/pods/2270dbb6-3038-487a-bd35-b801a741a695/volumes" Jul 17 06:25:27.884118 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:27.884086 2580 scope.go:117] "RemoveContainer" containerID="36bcd38c7c7b870cad2099f5d3004451ee04fc7b2c85e0030a54d1a1a8bcad5a" Jul 17 06:25:27.884651 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:25:27.884310 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:25:38.218431 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:38.218366 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:25:38.322046 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:38.322008 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:25:42.884476 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:42.884439 2580 scope.go:117] "RemoveContainer" containerID="36bcd38c7c7b870cad2099f5d3004451ee04fc7b2c85e0030a54d1a1a8bcad5a" Jul 17 06:25:42.884995 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:25:42.884634 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:25:50.906926 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:50.906884 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:25:51.002141 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:51.002101 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:25:57.884156 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:57.884104 2580 scope.go:117] "RemoveContainer" containerID="36bcd38c7c7b870cad2099f5d3004451ee04fc7b2c85e0030a54d1a1a8bcad5a" Jul 17 06:25:59.328145 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:59.328109 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerID="d88182f1d0bc330ea8c37fbd73a810d16f580aec33372e0fb94be0810193466f" exitCode=1 Jul 17 06:25:59.328600 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:59.328158 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" event={"ID":"bb301f8c-1e8f-444d-89c5-3161a7e48ffa","Type":"ContainerDied","Data":"d88182f1d0bc330ea8c37fbd73a810d16f580aec33372e0fb94be0810193466f"} Jul 17 06:25:59.328600 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:59.328222 2580 scope.go:117] "RemoveContainer" containerID="36bcd38c7c7b870cad2099f5d3004451ee04fc7b2c85e0030a54d1a1a8bcad5a" Jul 17 06:25:59.328739 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:25:59.328654 2580 scope.go:117] "RemoveContainer" containerID="d88182f1d0bc330ea8c37fbd73a810d16f580aec33372e0fb94be0810193466f" Jul 17 06:25:59.328872 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:25:59.328851 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:26:01.611102 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:01.611045 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:26:01.611712 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:01.611690 2580 scope.go:117] "RemoveContainer" containerID="d88182f1d0bc330ea8c37fbd73a810d16f580aec33372e0fb94be0810193466f" Jul 17 06:26:01.612006 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:26:01.611976 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:26:03.841141 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:03.841089 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc"] Jul 17 06:26:03.841675 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:03.841548 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2270dbb6-3038-487a-bd35-b801a741a695" containerName="revoke-keys" Jul 17 06:26:03.841675 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:03.841562 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2270dbb6-3038-487a-bd35-b801a741a695" containerName="revoke-keys" Jul 17 06:26:03.841675 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:03.841589 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2270dbb6-3038-487a-bd35-b801a741a695" containerName="revoke-keys" Jul 17 06:26:03.841675 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:03.841595 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2270dbb6-3038-487a-bd35-b801a741a695" containerName="revoke-keys" Jul 17 06:26:03.841675 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:03.841649 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2270dbb6-3038-487a-bd35-b801a741a695" containerName="revoke-keys" Jul 17 06:26:03.841675 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:03.841657 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2270dbb6-3038-487a-bd35-b801a741a695" containerName="revoke-keys" Jul 17 06:26:03.841675 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:03.841664 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2270dbb6-3038-487a-bd35-b801a741a695" containerName="revoke-keys" Jul 17 06:26:03.844571 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:03.844554 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" Jul 17 06:26:03.846961 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:03.846935 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:26:03.851344 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:03.851285 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc"] Jul 17 06:26:04.001626 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:04.001588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c8332b-19e8-473a-85be-dc5d38162767-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc\" (UID: \"59c8332b-19e8-473a-85be-dc5d38162767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" Jul 17 06:26:04.102447 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:04.102320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c8332b-19e8-473a-85be-dc5d38162767-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc\" (UID: \"59c8332b-19e8-473a-85be-dc5d38162767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" Jul 17 06:26:04.103014 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:04.102992 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c8332b-19e8-473a-85be-dc5d38162767-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc\" (UID: \"59c8332b-19e8-473a-85be-dc5d38162767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" Jul 17 06:26:04.156192 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:04.156144 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" Jul 17 06:26:04.284752 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:04.284718 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc"] Jul 17 06:26:04.288339 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:26:04.288314 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c8332b_19e8_473a_85be_dc5d38162767.slice/crio-04df046f7e6bf599d277ceee076f3d7424818f45e5062c45b662169b48c5da01 WatchSource:0}: Error finding container 04df046f7e6bf599d277ceee076f3d7424818f45e5062c45b662169b48c5da01: Status 404 returned error can't find the container with id 04df046f7e6bf599d277ceee076f3d7424818f45e5062c45b662169b48c5da01 Jul 17 06:26:04.352430 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:04.352371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" event={"ID":"59c8332b-19e8-473a-85be-dc5d38162767","Type":"ContainerStarted","Data":"04df046f7e6bf599d277ceee076f3d7424818f45e5062c45b662169b48c5da01"} Jul 17 06:26:05.357990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:05.357953 2580 generic.go:356] "Generic (PLEG): container finished" podID="59c8332b-19e8-473a-85be-dc5d38162767" containerID="c646d8396f0d51ce1b9549e9a0dcb790ddae34146aac19404bd068b467d64ccf" exitCode=6 Jul 17 06:26:05.358413 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:05.358037 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" event={"ID":"59c8332b-19e8-473a-85be-dc5d38162767","Type":"ContainerDied","Data":"c646d8396f0d51ce1b9549e9a0dcb790ddae34146aac19404bd068b467d64ccf"} Jul 17 06:26:05.358413 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:05.358226 2580 scope.go:117] "RemoveContainer" containerID="c646d8396f0d51ce1b9549e9a0dcb790ddae34146aac19404bd068b467d64ccf" Jul 17 06:26:05.822807 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:05.822768 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:26:05.823199 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:05.823181 2580 scope.go:117] "RemoveContainer" containerID="d88182f1d0bc330ea8c37fbd73a810d16f580aec33372e0fb94be0810193466f" Jul 17 06:26:05.823452 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:26:05.823428 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j_odh-ai-gateway-infra(bb301f8c-1e8f-444d-89c5-3161a7e48ffa)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" Jul 17 06:26:06.366731 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:06.366692 2580 generic.go:356] "Generic (PLEG): container finished" podID="59c8332b-19e8-473a-85be-dc5d38162767" containerID="9bf494e7be173ccddf612e79240a31da5c5ee5694030ddb7d704159adb6682dd" exitCode=6 Jul 17 06:26:06.367175 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:06.366751 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" event={"ID":"59c8332b-19e8-473a-85be-dc5d38162767","Type":"ContainerDied","Data":"9bf494e7be173ccddf612e79240a31da5c5ee5694030ddb7d704159adb6682dd"} Jul 17 06:26:06.367175 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:06.366812 2580 scope.go:117] "RemoveContainer" containerID="c646d8396f0d51ce1b9549e9a0dcb790ddae34146aac19404bd068b467d64ccf" Jul 17 06:26:06.367175 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:06.366991 2580 scope.go:117] "RemoveContainer" containerID="9bf494e7be173ccddf612e79240a31da5c5ee5694030ddb7d704159adb6682dd" Jul 17 06:26:06.367342 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:26:06.367218 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc_odh-ai-gateway-infra(59c8332b-19e8-473a-85be-dc5d38162767)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" podUID="59c8332b-19e8-473a-85be-dc5d38162767" Jul 17 06:26:07.373366 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:07.373291 2580 scope.go:117] "RemoveContainer" containerID="9bf494e7be173ccddf612e79240a31da5c5ee5694030ddb7d704159adb6682dd" Jul 17 06:26:07.373738 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:26:07.373510 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc_odh-ai-gateway-infra(59c8332b-19e8-473a-85be-dc5d38162767)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" podUID="59c8332b-19e8-473a-85be-dc5d38162767" Jul 17 06:26:09.845009 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:09.844957 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:09.940239 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:09.940201 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:12.509684 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:12.509647 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:26:15.157273 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.157238 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j"] Jul 17 06:26:15.188170 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.188128 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:15.293702 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.293671 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:26:15.405830 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.405746 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckst5\" (UniqueName: \"kubernetes.io/projected/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-kube-api-access-ckst5\") pod \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\" (UID: \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\") " Jul 17 06:26:15.405830 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.405792 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-maas-api-tls\") pod \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\" (UID: \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\") " Jul 17 06:26:15.405830 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.405816 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-openshift-service-ca\") pod \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\" (UID: \"bb301f8c-1e8f-444d-89c5-3161a7e48ffa\") " Jul 17 06:26:15.406181 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.406153 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb301f8c-1e8f-444d-89c5-3161a7e48ffa" (UID: "bb301f8c-1e8f-444d-89c5-3161a7e48ffa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:26:15.408009 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.407977 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb301f8c-1e8f-444d-89c5-3161a7e48ffa" (UID: "bb301f8c-1e8f-444d-89c5-3161a7e48ffa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:26:15.408130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.408113 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-kube-api-access-ckst5" (OuterVolumeSpecName: "kube-api-access-ckst5") pod "bb301f8c-1e8f-444d-89c5-3161a7e48ffa" (UID: "bb301f8c-1e8f-444d-89c5-3161a7e48ffa"). InnerVolumeSpecName "kube-api-access-ckst5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:26:15.410449 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.410423 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" event={"ID":"bb301f8c-1e8f-444d-89c5-3161a7e48ffa","Type":"ContainerDied","Data":"74221cf0dc103ccd752bff32d6fef4d49328f19b02499a6238ee5d32b7de9893"} Jul 17 06:26:15.410547 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.410485 2580 scope.go:117] "RemoveContainer" containerID="d88182f1d0bc330ea8c37fbd73a810d16f580aec33372e0fb94be0810193466f" Jul 17 06:26:15.410547 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.410431 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j" Jul 17 06:26:15.436098 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.436073 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j"] Jul 17 06:26:15.440369 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.440347 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-51e238b9-67bd7d946c-vth6j"] Jul 17 06:26:15.506644 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.506605 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckst5\" (UniqueName: \"kubernetes.io/projected/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-kube-api-access-ckst5\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:26:15.506644 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.506641 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-maas-api-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:26:15.506644 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:15.506650 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb301f8c-1e8f-444d-89c5-3161a7e48ffa-openshift-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:26:16.258092 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:16.258055 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:16.888937 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:16.888899 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" path="/var/lib/kubelet/pods/bb301f8c-1e8f-444d-89c5-3161a7e48ffa/volumes" Jul 17 06:26:17.883861 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:17.883827 2580 scope.go:117] "RemoveContainer" containerID="9bf494e7be173ccddf612e79240a31da5c5ee5694030ddb7d704159adb6682dd" Jul 17 06:26:18.425785 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:18.425741 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" event={"ID":"59c8332b-19e8-473a-85be-dc5d38162767","Type":"ContainerStarted","Data":"318d14a9b00a1853973b1ca48e044b4d9b1e6073c49c4815154d20249bc74eb2"} Jul 17 06:26:18.442600 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:18.442548 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" podStartSLOduration=15.442531833 podStartE2EDuration="15.442531833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:26:18.441915751 +0000 UTC m=+2703.186595080" watchObservedRunningTime="2026-07-17 06:26:18.442531833 +0000 UTC m=+2703.187211162" Jul 17 06:26:19.432193 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:19.432157 2580 generic.go:356] "Generic (PLEG): container finished" podID="59c8332b-19e8-473a-85be-dc5d38162767" containerID="318d14a9b00a1853973b1ca48e044b4d9b1e6073c49c4815154d20249bc74eb2" exitCode=6 Jul 17 06:26:19.432635 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:19.432228 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" event={"ID":"59c8332b-19e8-473a-85be-dc5d38162767","Type":"ContainerDied","Data":"318d14a9b00a1853973b1ca48e044b4d9b1e6073c49c4815154d20249bc74eb2"} Jul 17 06:26:19.432635 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:19.432286 2580 scope.go:117] "RemoveContainer" containerID="9bf494e7be173ccddf612e79240a31da5c5ee5694030ddb7d704159adb6682dd" Jul 17 06:26:19.432635 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:19.432620 2580 scope.go:117] "RemoveContainer" containerID="318d14a9b00a1853973b1ca48e044b4d9b1e6073c49c4815154d20249bc74eb2" Jul 17 06:26:19.432858 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:26:19.432837 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc_odh-ai-gateway-infra(59c8332b-19e8-473a-85be-dc5d38162767)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" podUID="59c8332b-19e8-473a-85be-dc5d38162767" Jul 17 06:26:19.452790 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:19.452764 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc"] Jul 17 06:26:20.565509 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:20.565485 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" Jul 17 06:26:20.652119 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:20.652080 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c8332b-19e8-473a-85be-dc5d38162767-maas-api-service-ca\") pod \"59c8332b-19e8-473a-85be-dc5d38162767\" (UID: \"59c8332b-19e8-473a-85be-dc5d38162767\") " Jul 17 06:26:20.652517 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:20.652489 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c8332b-19e8-473a-85be-dc5d38162767-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59c8332b-19e8-473a-85be-dc5d38162767" (UID: "59c8332b-19e8-473a-85be-dc5d38162767"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:26:20.752995 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:20.752914 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c8332b-19e8-473a-85be-dc5d38162767-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:26:21.443294 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:21.443264 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" Jul 17 06:26:21.443494 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:21.443272 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc" event={"ID":"59c8332b-19e8-473a-85be-dc5d38162767","Type":"ContainerDied","Data":"04df046f7e6bf599d277ceee076f3d7424818f45e5062c45b662169b48c5da01"} Jul 17 06:26:21.443494 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:21.443410 2580 scope.go:117] "RemoveContainer" containerID="318d14a9b00a1853973b1ca48e044b4d9b1e6073c49c4815154d20249bc74eb2" Jul 17 06:26:21.469240 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:21.469207 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc"] Jul 17 06:26:21.473088 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:21.473064 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-vhsnc"] Jul 17 06:26:22.459168 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:22.459131 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:22.565703 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:22.565668 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:22.893856 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:22.893823 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59c8332b-19e8-473a-85be-dc5d38162767" path="/var/lib/kubelet/pods/59c8332b-19e8-473a-85be-dc5d38162767/volumes" Jul 17 06:26:23.016834 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:23.016791 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:23.113598 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:23.113565 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:24.802149 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:24.802107 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:26.560898 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:26.560869 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:32.072194 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:32.072146 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:32.192825 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:32.192790 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:32.310821 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:32.310784 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:32.422117 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:32.422074 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:32.962164 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:32.962120 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:26:58.850651 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:26:58.850610 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:05.530472 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:05.530428 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:05.648325 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:05.648292 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:05.751530 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:05.751482 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:21.654315 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:21.654269 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:27.542767 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:27.542726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:27.661574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:27.661534 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:27.787114 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:27.787072 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:38.314438 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:38.314322 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:39.098872 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:39.098826 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:41.664573 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:41.664532 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:41.764573 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:41.764490 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:46.889129 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:46.889085 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:46.998709 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:46.998675 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:47.118947 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:47.118894 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:47.229306 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:47.229216 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:47.967880 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:47.967838 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:48.044450 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:48.044417 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:48.197241 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:48.197192 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:49.828935 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:49.828897 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:51.085514 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:51.085475 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:27:55.656168 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:55.656128 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:55.737307 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:55.737271 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:27:56.346442 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:27:56.346379 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:02.365694 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:02.365657 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:02.436968 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:02.436931 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:02.598788 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:02.598746 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:02.692504 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:02.692425 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:28:03.002376 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:03.002282 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:03.445469 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:03.445431 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:06.272861 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.272832 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:06.363513 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.363469 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:06.364186 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364160 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv"] Jul 17 06:28:06.364499 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364484 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:06.364499 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364499 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:06.364650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364507 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2270dbb6-3038-487a-bd35-b801a741a695" containerName="revoke-keys" Jul 17 06:28:06.364650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364513 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2270dbb6-3038-487a-bd35-b801a741a695" containerName="revoke-keys" Jul 17 06:28:06.364650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364523 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:06.364650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364529 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:06.364650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364535 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c8332b-19e8-473a-85be-dc5d38162767" containerName="revoke-keys" Jul 17 06:28:06.364650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364540 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c8332b-19e8-473a-85be-dc5d38162767" containerName="revoke-keys" Jul 17 06:28:06.364650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364559 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:06.364650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364566 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:06.364650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364580 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:06.364650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364589 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:06.364650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364602 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c8332b-19e8-473a-85be-dc5d38162767" containerName="revoke-keys" Jul 17 06:28:06.364650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364610 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c8332b-19e8-473a-85be-dc5d38162767" containerName="revoke-keys" Jul 17 06:28:06.365158 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364679 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:06.365158 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364687 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:06.365158 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364698 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c8332b-19e8-473a-85be-dc5d38162767" containerName="revoke-keys" Jul 17 06:28:06.365158 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364708 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:06.365158 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364719 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c8332b-19e8-473a-85be-dc5d38162767" containerName="revoke-keys" Jul 17 06:28:06.365158 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.364728 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c8332b-19e8-473a-85be-dc5d38162767" containerName="revoke-keys" Jul 17 06:28:06.367858 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.367841 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:06.370472 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.370453 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 06:28:06.370577 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.370530 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpsw\"" Jul 17 06:28:06.371104 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.371080 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bd7a398a-serving-cert\"" Jul 17 06:28:06.371211 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.371090 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 06:28:06.380942 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.380918 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv"] Jul 17 06:28:06.386949 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.386926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-986dl\" (UniqueName: \"kubernetes.io/projected/02398a35-77eb-4986-8220-4e0af150ad7b-kube-api-access-986dl\") pod \"maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv\" (UID: \"02398a35-77eb-4986-8220-4e0af150ad7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:06.387045 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.386964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02398a35-77eb-4986-8220-4e0af150ad7b-openshift-service-ca\") pod \"maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv\" (UID: \"02398a35-77eb-4986-8220-4e0af150ad7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:06.387045 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.386992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02398a35-77eb-4986-8220-4e0af150ad7b-maas-api-tls\") pod \"maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv\" (UID: \"02398a35-77eb-4986-8220-4e0af150ad7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:06.457322 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.457284 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:06.488350 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.488319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02398a35-77eb-4986-8220-4e0af150ad7b-openshift-service-ca\") pod \"maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv\" (UID: \"02398a35-77eb-4986-8220-4e0af150ad7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:06.488531 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.488361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02398a35-77eb-4986-8220-4e0af150ad7b-maas-api-tls\") pod \"maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv\" (UID: \"02398a35-77eb-4986-8220-4e0af150ad7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:06.488531 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.488436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-986dl\" (UniqueName: \"kubernetes.io/projected/02398a35-77eb-4986-8220-4e0af150ad7b-kube-api-access-986dl\") pod \"maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv\" (UID: \"02398a35-77eb-4986-8220-4e0af150ad7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:06.488668 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:28:06.488577 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-serving-cert: secret "maas-api-e2e-mt-bd7a398a-serving-cert" not found Jul 17 06:28:06.488668 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:28:06.488645 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02398a35-77eb-4986-8220-4e0af150ad7b-maas-api-tls podName:02398a35-77eb-4986-8220-4e0af150ad7b nodeName:}" failed. No retries permitted until 2026-07-17 06:28:06.988624066 +0000 UTC m=+2811.733303391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02398a35-77eb-4986-8220-4e0af150ad7b-maas-api-tls") pod "maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" (UID: "02398a35-77eb-4986-8220-4e0af150ad7b") : secret "maas-api-e2e-mt-bd7a398a-serving-cert" not found Jul 17 06:28:06.488973 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.488954 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02398a35-77eb-4986-8220-4e0af150ad7b-openshift-service-ca\") pod \"maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv\" (UID: \"02398a35-77eb-4986-8220-4e0af150ad7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:06.499213 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.499183 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-986dl\" (UniqueName: \"kubernetes.io/projected/02398a35-77eb-4986-8220-4e0af150ad7b-kube-api-access-986dl\") pod \"maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv\" (UID: \"02398a35-77eb-4986-8220-4e0af150ad7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:06.774132 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.774097 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:06.976702 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.976671 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:06.994193 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.994038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02398a35-77eb-4986-8220-4e0af150ad7b-maas-api-tls\") pod \"maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv\" (UID: \"02398a35-77eb-4986-8220-4e0af150ad7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:06.997116 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:06.997080 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02398a35-77eb-4986-8220-4e0af150ad7b-maas-api-tls\") pod \"maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv\" (UID: \"02398a35-77eb-4986-8220-4e0af150ad7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:07.077671 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:07.077573 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:07.280615 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:07.280562 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:07.412464 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:07.412437 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv"] Jul 17 06:28:07.415005 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:28:07.414979 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02398a35_77eb_4986_8220_4e0af150ad7b.slice/crio-0775f64790935af743b2273caafa2cf9fc872f7852f6608ece8ced4e9c41ce65 WatchSource:0}: Error finding container 0775f64790935af743b2273caafa2cf9fc872f7852f6608ece8ced4e9c41ce65: Status 404 returned error can't find the container with id 0775f64790935af743b2273caafa2cf9fc872f7852f6608ece8ced4e9c41ce65 Jul 17 06:28:07.416270 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:07.416250 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 06:28:07.915417 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:07.915365 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" event={"ID":"02398a35-77eb-4986-8220-4e0af150ad7b","Type":"ContainerStarted","Data":"0775f64790935af743b2273caafa2cf9fc872f7852f6608ece8ced4e9c41ce65"} Jul 17 06:28:08.921454 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:08.921415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" event={"ID":"02398a35-77eb-4986-8220-4e0af150ad7b","Type":"ContainerStarted","Data":"fb4d880dc3b2754a60c050af6197c2b58d69853658523baa20ec65de751e23c7"} Jul 17 06:28:08.921949 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:08.921502 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:08.953155 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:08.953098 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" podStartSLOduration=2.485137672 podStartE2EDuration="2.953079442s" totalImagesPullingTime="467.94177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:28:08.950799228 +0000 UTC m=+2813.695478558" watchObservedRunningTime="2026-07-17 06:28:08.953079442 +0000 UTC m=+2813.697758770" Jul 17 06:28:10.690283 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:10.690239 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:11.416430 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:11.416399 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:11.539465 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:11.539419 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:11.899317 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:11.899284 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:14.931114 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:14.931082 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:17.386083 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:17.386043 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:28:17.790594 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:17.790503 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:17.871571 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:17.871534 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:18.144994 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.144955 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq"] Jul 17 06:28:18.145317 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.145302 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:18.145363 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.145319 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:18.145363 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.145328 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c8332b-19e8-473a-85be-dc5d38162767" containerName="revoke-keys" Jul 17 06:28:18.145363 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.145334 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c8332b-19e8-473a-85be-dc5d38162767" containerName="revoke-keys" Jul 17 06:28:18.145492 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.145413 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:18.145492 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.145421 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:18.145492 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.145428 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:18.148584 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.148564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq" Jul 17 06:28:18.151020 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.150999 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:28:18.156034 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.156006 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq"] Jul 17 06:28:18.290651 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.290611 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2282024-e221-4907-8e2d-2a9775f525d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq\" (UID: \"b2282024-e221-4907-8e2d-2a9775f525d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq" Jul 17 06:28:18.392107 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.392064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2282024-e221-4907-8e2d-2a9775f525d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq\" (UID: \"b2282024-e221-4907-8e2d-2a9775f525d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq" Jul 17 06:28:18.392736 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.392714 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2282024-e221-4907-8e2d-2a9775f525d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq\" (UID: \"b2282024-e221-4907-8e2d-2a9775f525d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq" Jul 17 06:28:18.459164 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.459077 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq" Jul 17 06:28:18.584094 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.583960 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq"] Jul 17 06:28:18.586778 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:28:18.586749 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2282024_e221_4907_8e2d_2a9775f525d7.slice/crio-a8dee04f3216cd1feb157317563b1035fbad205c6795495c8214a5a069bc2158 WatchSource:0}: Error finding container a8dee04f3216cd1feb157317563b1035fbad205c6795495c8214a5a069bc2158: Status 404 returned error can't find the container with id a8dee04f3216cd1feb157317563b1035fbad205c6795495c8214a5a069bc2158 Jul 17 06:28:18.966520 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.966484 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq" event={"ID":"b2282024-e221-4907-8e2d-2a9775f525d7","Type":"ContainerStarted","Data":"939436c2450b32ec8dc6796f9f6b264dac15299dca3d410b3e38ecf279e862fa"} Jul 17 06:28:18.966706 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.966545 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq" event={"ID":"b2282024-e221-4907-8e2d-2a9775f525d7","Type":"ContainerStarted","Data":"a8dee04f3216cd1feb157317563b1035fbad205c6795495c8214a5a069bc2158"} Jul 17 06:28:18.981583 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:18.981536 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq" podStartSLOduration=0.981522499 podStartE2EDuration="981.522499ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:28:18.979870725 +0000 UTC m=+2823.724550073" watchObservedRunningTime="2026-07-17 06:28:18.981522499 +0000 UTC m=+2823.726201827" Jul 17 06:28:19.972110 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:19.972076 2580 generic.go:356] "Generic (PLEG): container finished" podID="b2282024-e221-4907-8e2d-2a9775f525d7" containerID="939436c2450b32ec8dc6796f9f6b264dac15299dca3d410b3e38ecf279e862fa" exitCode=0 Jul 17 06:28:19.972532 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:19.972158 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq" event={"ID":"b2282024-e221-4907-8e2d-2a9775f525d7","Type":"ContainerDied","Data":"939436c2450b32ec8dc6796f9f6b264dac15299dca3d410b3e38ecf279e862fa"} Jul 17 06:28:21.106216 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:21.106192 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq" Jul 17 06:28:21.218413 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:21.218366 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2282024-e221-4907-8e2d-2a9775f525d7-maas-api-service-ca\") pod \"b2282024-e221-4907-8e2d-2a9775f525d7\" (UID: \"b2282024-e221-4907-8e2d-2a9775f525d7\") " Jul 17 06:28:21.218731 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:21.218707 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2282024-e221-4907-8e2d-2a9775f525d7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2282024-e221-4907-8e2d-2a9775f525d7" (UID: "b2282024-e221-4907-8e2d-2a9775f525d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:28:21.319423 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:21.319312 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2282024-e221-4907-8e2d-2a9775f525d7-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:28:21.982147 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:21.982117 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq" Jul 17 06:28:21.982147 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:21.982141 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq" event={"ID":"b2282024-e221-4907-8e2d-2a9775f525d7","Type":"ContainerDied","Data":"a8dee04f3216cd1feb157317563b1035fbad205c6795495c8214a5a069bc2158"} Jul 17 06:28:21.982363 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:21.982186 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8dee04f3216cd1feb157317563b1035fbad205c6795495c8214a5a069bc2158" Jul 17 06:28:28.621672 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:28.621641 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv"] Jul 17 06:28:28.622147 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:28.621852 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" podUID="02398a35-77eb-4986-8220-4e0af150ad7b" containerName="maas-api" containerID="cri-o://fb4d880dc3b2754a60c050af6197c2b58d69853658523baa20ec65de751e23c7" gracePeriod=30 Jul 17 06:28:28.652659 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:28.652618 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:28.863455 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:28.863432 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:28.991261 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:28.991163 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02398a35-77eb-4986-8220-4e0af150ad7b-maas-api-tls\") pod \"02398a35-77eb-4986-8220-4e0af150ad7b\" (UID: \"02398a35-77eb-4986-8220-4e0af150ad7b\") " Jul 17 06:28:28.991261 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:28.991233 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02398a35-77eb-4986-8220-4e0af150ad7b-openshift-service-ca\") pod \"02398a35-77eb-4986-8220-4e0af150ad7b\" (UID: \"02398a35-77eb-4986-8220-4e0af150ad7b\") " Jul 17 06:28:28.991501 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:28.991336 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-986dl\" (UniqueName: \"kubernetes.io/projected/02398a35-77eb-4986-8220-4e0af150ad7b-kube-api-access-986dl\") pod \"02398a35-77eb-4986-8220-4e0af150ad7b\" (UID: \"02398a35-77eb-4986-8220-4e0af150ad7b\") " Jul 17 06:28:28.991588 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:28.991570 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02398a35-77eb-4986-8220-4e0af150ad7b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "02398a35-77eb-4986-8220-4e0af150ad7b" (UID: "02398a35-77eb-4986-8220-4e0af150ad7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:28:28.991723 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:28.991705 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02398a35-77eb-4986-8220-4e0af150ad7b-openshift-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:28:28.993463 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:28.993440 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02398a35-77eb-4986-8220-4e0af150ad7b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "02398a35-77eb-4986-8220-4e0af150ad7b" (UID: "02398a35-77eb-4986-8220-4e0af150ad7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:28:28.993572 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:28.993471 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02398a35-77eb-4986-8220-4e0af150ad7b-kube-api-access-986dl" (OuterVolumeSpecName: "kube-api-access-986dl") pod "02398a35-77eb-4986-8220-4e0af150ad7b" (UID: "02398a35-77eb-4986-8220-4e0af150ad7b"). InnerVolumeSpecName "kube-api-access-986dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:28:29.011374 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:29.011348 2580 generic.go:356] "Generic (PLEG): container finished" podID="02398a35-77eb-4986-8220-4e0af150ad7b" containerID="fb4d880dc3b2754a60c050af6197c2b58d69853658523baa20ec65de751e23c7" exitCode=0 Jul 17 06:28:29.011502 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:29.011412 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" Jul 17 06:28:29.011502 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:29.011415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" event={"ID":"02398a35-77eb-4986-8220-4e0af150ad7b","Type":"ContainerDied","Data":"fb4d880dc3b2754a60c050af6197c2b58d69853658523baa20ec65de751e23c7"} Jul 17 06:28:29.011585 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:29.011525 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv" event={"ID":"02398a35-77eb-4986-8220-4e0af150ad7b","Type":"ContainerDied","Data":"0775f64790935af743b2273caafa2cf9fc872f7852f6608ece8ced4e9c41ce65"} Jul 17 06:28:29.011585 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:29.011548 2580 scope.go:117] "RemoveContainer" containerID="fb4d880dc3b2754a60c050af6197c2b58d69853658523baa20ec65de751e23c7" Jul 17 06:28:29.022429 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:29.022406 2580 scope.go:117] "RemoveContainer" containerID="fb4d880dc3b2754a60c050af6197c2b58d69853658523baa20ec65de751e23c7" Jul 17 06:28:29.022733 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:28:29.022712 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4d880dc3b2754a60c050af6197c2b58d69853658523baa20ec65de751e23c7\": container with ID starting with fb4d880dc3b2754a60c050af6197c2b58d69853658523baa20ec65de751e23c7 not found: ID does not exist" containerID="fb4d880dc3b2754a60c050af6197c2b58d69853658523baa20ec65de751e23c7" Jul 17 06:28:29.022822 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:29.022744 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4d880dc3b2754a60c050af6197c2b58d69853658523baa20ec65de751e23c7"} err="failed to get container status \"fb4d880dc3b2754a60c050af6197c2b58d69853658523baa20ec65de751e23c7\": rpc error: code = NotFound desc = could not find container \"fb4d880dc3b2754a60c050af6197c2b58d69853658523baa20ec65de751e23c7\": container with ID starting with fb4d880dc3b2754a60c050af6197c2b58d69853658523baa20ec65de751e23c7 not found: ID does not exist" Jul 17 06:28:29.036084 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:29.036057 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv"] Jul 17 06:28:29.042298 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:29.042274 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bd7a398a-5d5fbddd79-rjvqv"] Jul 17 06:28:29.092789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:29.092755 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-986dl\" (UniqueName: \"kubernetes.io/projected/02398a35-77eb-4986-8220-4e0af150ad7b-kube-api-access-986dl\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:28:29.092789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:29.092782 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02398a35-77eb-4986-8220-4e0af150ad7b-maas-api-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:28:30.889781 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:30.889744 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02398a35-77eb-4986-8220-4e0af150ad7b" path="/var/lib/kubelet/pods/02398a35-77eb-4986-8220-4e0af150ad7b/volumes" Jul 17 06:28:31.680427 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:31.680375 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:28:33.268455 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:33.268419 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:33.628360 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:33.628329 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:39.371370 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:39.371337 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:39.444182 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:39.444143 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:39.638845 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:39.638711 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:39.843222 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:39.843189 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:39.985608 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:39.985520 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:40.001764 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.001728 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8"] Jul 17 06:28:40.002126 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.002107 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2282024-e221-4907-8e2d-2a9775f525d7" containerName="revoke-keys" Jul 17 06:28:40.002219 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.002128 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2282024-e221-4907-8e2d-2a9775f525d7" containerName="revoke-keys" Jul 17 06:28:40.002219 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.002143 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02398a35-77eb-4986-8220-4e0af150ad7b" containerName="maas-api" Jul 17 06:28:40.002219 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.002151 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="02398a35-77eb-4986-8220-4e0af150ad7b" containerName="maas-api" Jul 17 06:28:40.002219 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.002177 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:40.002219 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.002185 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb301f8c-1e8f-444d-89c5-3161a7e48ffa" containerName="maas-api" Jul 17 06:28:40.002489 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.002252 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="02398a35-77eb-4986-8220-4e0af150ad7b" containerName="maas-api" Jul 17 06:28:40.002489 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.002266 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2282024-e221-4907-8e2d-2a9775f525d7" containerName="revoke-keys" Jul 17 06:28:40.006508 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.006485 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" Jul 17 06:28:40.010432 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.010409 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:28:40.011093 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.011078 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 06:28:40.016307 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.016287 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8"] Jul 17 06:28:40.056049 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.056005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:40.189460 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.189421 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03f6876-027c-49f9-ac0b-1fa72a788de9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8\" (UID: \"e03f6876-027c-49f9-ac0b-1fa72a788de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" Jul 17 06:28:40.195347 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.195318 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:40.290839 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.290745 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03f6876-027c-49f9-ac0b-1fa72a788de9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8\" (UID: \"e03f6876-027c-49f9-ac0b-1fa72a788de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" Jul 17 06:28:40.291414 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.291370 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03f6876-027c-49f9-ac0b-1fa72a788de9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8\" (UID: \"e03f6876-027c-49f9-ac0b-1fa72a788de9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" Jul 17 06:28:40.316577 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.316544 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" Jul 17 06:28:40.440604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:40.440558 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8"] Jul 17 06:28:40.442933 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:28:40.442907 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03f6876_027c_49f9_ac0b_1fa72a788de9.slice/crio-3dd5a6a0b4883f3f7f68d5f23d4d9491278629bc0461b904706a9e90d1d6dd98 WatchSource:0}: Error finding container 3dd5a6a0b4883f3f7f68d5f23d4d9491278629bc0461b904706a9e90d1d6dd98: Status 404 returned error can't find the container with id 3dd5a6a0b4883f3f7f68d5f23d4d9491278629bc0461b904706a9e90d1d6dd98 Jul 17 06:28:41.065023 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:41.064995 2580 generic.go:356] "Generic (PLEG): container finished" podID="e03f6876-027c-49f9-ac0b-1fa72a788de9" containerID="657df80751a8a08c7333070c2ada20599fb599443188786f738854eb0206a158" exitCode=6 Jul 17 06:28:41.065169 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:41.065079 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" event={"ID":"e03f6876-027c-49f9-ac0b-1fa72a788de9","Type":"ContainerDied","Data":"657df80751a8a08c7333070c2ada20599fb599443188786f738854eb0206a158"} Jul 17 06:28:41.065169 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:41.065142 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" event={"ID":"e03f6876-027c-49f9-ac0b-1fa72a788de9","Type":"ContainerStarted","Data":"3dd5a6a0b4883f3f7f68d5f23d4d9491278629bc0461b904706a9e90d1d6dd98"} Jul 17 06:28:41.065248 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:41.065238 2580 scope.go:117] "RemoveContainer" containerID="657df80751a8a08c7333070c2ada20599fb599443188786f738854eb0206a158" Jul 17 06:28:42.070949 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:42.070863 2580 generic.go:356] "Generic (PLEG): container finished" podID="e03f6876-027c-49f9-ac0b-1fa72a788de9" containerID="d30599d55419e1d6a7b24bc9423e46e1b7777629befae26b0e79d5c12a90ff1d" exitCode=6 Jul 17 06:28:42.070949 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:42.070934 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" event={"ID":"e03f6876-027c-49f9-ac0b-1fa72a788de9","Type":"ContainerDied","Data":"d30599d55419e1d6a7b24bc9423e46e1b7777629befae26b0e79d5c12a90ff1d"} Jul 17 06:28:42.071458 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:42.070977 2580 scope.go:117] "RemoveContainer" containerID="657df80751a8a08c7333070c2ada20599fb599443188786f738854eb0206a158" Jul 17 06:28:42.071458 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:42.071186 2580 scope.go:117] "RemoveContainer" containerID="d30599d55419e1d6a7b24bc9423e46e1b7777629befae26b0e79d5c12a90ff1d" Jul 17 06:28:42.071666 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:28:42.071632 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8_odh-ai-gateway-infra(e03f6876-027c-49f9-ac0b-1fa72a788de9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" podUID="e03f6876-027c-49f9-ac0b-1fa72a788de9" Jul 17 06:28:43.077503 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:43.077473 2580 scope.go:117] "RemoveContainer" containerID="d30599d55419e1d6a7b24bc9423e46e1b7777629befae26b0e79d5c12a90ff1d" Jul 17 06:28:43.077899 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:28:43.077683 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8_odh-ai-gateway-infra(e03f6876-027c-49f9-ac0b-1fa72a788de9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" podUID="e03f6876-027c-49f9-ac0b-1fa72a788de9" Jul 17 06:28:44.026845 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:44.026817 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn"] Jul 17 06:28:44.031536 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:44.031516 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" Jul 17 06:28:44.036589 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:44.036564 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn"] Jul 17 06:28:44.227224 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:44.227182 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b4e63d-b4e0-4dce-a4b9-5acb4022d838-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-4clxn\" (UID: \"56b4e63d-b4e0-4dce-a4b9-5acb4022d838\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" Jul 17 06:28:44.327850 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:44.327749 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b4e63d-b4e0-4dce-a4b9-5acb4022d838-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-4clxn\" (UID: \"56b4e63d-b4e0-4dce-a4b9-5acb4022d838\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" Jul 17 06:28:44.328430 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:44.328378 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b4e63d-b4e0-4dce-a4b9-5acb4022d838-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-4clxn\" (UID: \"56b4e63d-b4e0-4dce-a4b9-5acb4022d838\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" Jul 17 06:28:44.342725 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:44.342698 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" Jul 17 06:28:44.677448 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:44.677272 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn"] Jul 17 06:28:44.679580 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:28:44.679554 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b4e63d_b4e0_4dce_a4b9_5acb4022d838.slice/crio-94e8889c2e6ef1775dd61a46e8213d613e7c13ab6bfcd5d6adea3bb9199b369f WatchSource:0}: Error finding container 94e8889c2e6ef1775dd61a46e8213d613e7c13ab6bfcd5d6adea3bb9199b369f: Status 404 returned error can't find the container with id 94e8889c2e6ef1775dd61a46e8213d613e7c13ab6bfcd5d6adea3bb9199b369f Jul 17 06:28:45.087458 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:45.087351 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" event={"ID":"56b4e63d-b4e0-4dce-a4b9-5acb4022d838","Type":"ContainerStarted","Data":"bba6b6d643fe89c0b3cd446a7b577fbb147c4dab5078ab34106f94005c248ac8"} Jul 17 06:28:45.087458 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:45.087424 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" event={"ID":"56b4e63d-b4e0-4dce-a4b9-5acb4022d838","Type":"ContainerStarted","Data":"94e8889c2e6ef1775dd61a46e8213d613e7c13ab6bfcd5d6adea3bb9199b369f"} Jul 17 06:28:45.103075 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:45.103022 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" podStartSLOduration=1.103005934 podStartE2EDuration="1.103005934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:28:45.10168841 +0000 UTC m=+2849.846367749" watchObservedRunningTime="2026-07-17 06:28:45.103005934 +0000 UTC m=+2849.847685271" Jul 17 06:28:45.841878 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:45.841840 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:45.915619 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:45.915585 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:46.004625 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:46.004586 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:46.093691 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:46.093601 2580 generic.go:356] "Generic (PLEG): container finished" podID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" containerID="bba6b6d643fe89c0b3cd446a7b577fbb147c4dab5078ab34106f94005c248ac8" exitCode=6 Jul 17 06:28:46.093847 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:46.093693 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" event={"ID":"56b4e63d-b4e0-4dce-a4b9-5acb4022d838","Type":"ContainerDied","Data":"bba6b6d643fe89c0b3cd446a7b577fbb147c4dab5078ab34106f94005c248ac8"} Jul 17 06:28:46.093928 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:46.093912 2580 scope.go:117] "RemoveContainer" containerID="bba6b6d643fe89c0b3cd446a7b577fbb147c4dab5078ab34106f94005c248ac8" Jul 17 06:28:46.102273 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:46.102237 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:46.204221 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:46.204179 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:47.099608 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:47.099518 2580 generic.go:356] "Generic (PLEG): container finished" podID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" containerID="83aff69f8eaff359f056078e7aa0a4ac5c223b2bd6123094481a1c5e53e019ea" exitCode=6 Jul 17 06:28:47.099608 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:47.099588 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" event={"ID":"56b4e63d-b4e0-4dce-a4b9-5acb4022d838","Type":"ContainerDied","Data":"83aff69f8eaff359f056078e7aa0a4ac5c223b2bd6123094481a1c5e53e019ea"} Jul 17 06:28:47.100074 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:47.099644 2580 scope.go:117] "RemoveContainer" containerID="bba6b6d643fe89c0b3cd446a7b577fbb147c4dab5078ab34106f94005c248ac8" Jul 17 06:28:47.100074 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:47.099850 2580 scope.go:117] "RemoveContainer" containerID="83aff69f8eaff359f056078e7aa0a4ac5c223b2bd6123094481a1c5e53e019ea" Jul 17 06:28:47.100186 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:28:47.100083 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-4clxn_odh-ai-gateway-infra(56b4e63d-b4e0-4dce-a4b9-5acb4022d838)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" podUID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" Jul 17 06:28:48.106317 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:48.106292 2580 scope.go:117] "RemoveContainer" containerID="83aff69f8eaff359f056078e7aa0a4ac5c223b2bd6123094481a1c5e53e019ea" Jul 17 06:28:48.106703 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:28:48.106504 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-4clxn_odh-ai-gateway-infra(56b4e63d-b4e0-4dce-a4b9-5acb4022d838)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" podUID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" Jul 17 06:28:51.871128 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:51.871091 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:51.972557 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:51.972522 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:57.884665 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:57.884630 2580 scope.go:117] "RemoveContainer" containerID="d30599d55419e1d6a7b24bc9423e46e1b7777629befae26b0e79d5c12a90ff1d" Jul 17 06:28:58.153323 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:58.153295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" event={"ID":"e03f6876-027c-49f9-ac0b-1fa72a788de9","Type":"ContainerStarted","Data":"e51622dca69eafde877c9391e6489c31fbc2acf8b18dbcbc6fa5fbdc379d9184"} Jul 17 06:28:58.169551 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:58.169500 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" podStartSLOduration=19.169484618 podStartE2EDuration="19.169484618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:28:58.168380929 +0000 UTC m=+2862.913060259" watchObservedRunningTime="2026-07-17 06:28:58.169484618 +0000 UTC m=+2862.914164000" Jul 17 06:28:58.596397 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:58.596356 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:58.913421 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:58.913372 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:28:59.160039 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:59.160006 2580 generic.go:356] "Generic (PLEG): container finished" podID="e03f6876-027c-49f9-ac0b-1fa72a788de9" containerID="e51622dca69eafde877c9391e6489c31fbc2acf8b18dbcbc6fa5fbdc379d9184" exitCode=6 Jul 17 06:28:59.160229 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:59.160083 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" event={"ID":"e03f6876-027c-49f9-ac0b-1fa72a788de9","Type":"ContainerDied","Data":"e51622dca69eafde877c9391e6489c31fbc2acf8b18dbcbc6fa5fbdc379d9184"} Jul 17 06:28:59.160229 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:59.160154 2580 scope.go:117] "RemoveContainer" containerID="d30599d55419e1d6a7b24bc9423e46e1b7777629befae26b0e79d5c12a90ff1d" Jul 17 06:28:59.160488 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:59.160463 2580 scope.go:117] "RemoveContainer" containerID="e51622dca69eafde877c9391e6489c31fbc2acf8b18dbcbc6fa5fbdc379d9184" Jul 17 06:28:59.160710 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:28:59.160688 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8_odh-ai-gateway-infra(e03f6876-027c-49f9-ac0b-1fa72a788de9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" podUID="e03f6876-027c-49f9-ac0b-1fa72a788de9" Jul 17 06:28:59.184957 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:59.184930 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8"] Jul 17 06:28:59.241522 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:28:59.241482 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:29:00.295013 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:00.294987 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" Jul 17 06:29:00.364402 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:00.364363 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03f6876-027c-49f9-ac0b-1fa72a788de9-maas-api-service-ca\") pod \"e03f6876-027c-49f9-ac0b-1fa72a788de9\" (UID: \"e03f6876-027c-49f9-ac0b-1fa72a788de9\") " Jul 17 06:29:00.364773 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:00.364750 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e03f6876-027c-49f9-ac0b-1fa72a788de9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e03f6876-027c-49f9-ac0b-1fa72a788de9" (UID: "e03f6876-027c-49f9-ac0b-1fa72a788de9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:29:00.465888 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:00.465808 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03f6876-027c-49f9-ac0b-1fa72a788de9-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:29:01.171403 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.171308 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" Jul 17 06:29:01.171537 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.171316 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8" event={"ID":"e03f6876-027c-49f9-ac0b-1fa72a788de9","Type":"ContainerDied","Data":"3dd5a6a0b4883f3f7f68d5f23d4d9491278629bc0461b904706a9e90d1d6dd98"} Jul 17 06:29:01.171537 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.171455 2580 scope.go:117] "RemoveContainer" containerID="e51622dca69eafde877c9391e6489c31fbc2acf8b18dbcbc6fa5fbdc379d9184" Jul 17 06:29:01.187363 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.187340 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8"] Jul 17 06:29:01.189947 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.189926 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-zn2b8"] Jul 17 06:29:01.302638 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.302599 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:29:01.393310 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.393273 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr"] Jul 17 06:29:01.393714 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.393695 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e03f6876-027c-49f9-ac0b-1fa72a788de9" containerName="revoke-keys" Jul 17 06:29:01.393809 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.393718 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e03f6876-027c-49f9-ac0b-1fa72a788de9" containerName="revoke-keys" Jul 17 06:29:01.393809 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.393735 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e03f6876-027c-49f9-ac0b-1fa72a788de9" containerName="revoke-keys" Jul 17 06:29:01.393809 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.393744 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e03f6876-027c-49f9-ac0b-1fa72a788de9" containerName="revoke-keys" Jul 17 06:29:01.393916 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.393844 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e03f6876-027c-49f9-ac0b-1fa72a788de9" containerName="revoke-keys" Jul 17 06:29:01.393916 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.393854 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e03f6876-027c-49f9-ac0b-1fa72a788de9" containerName="revoke-keys" Jul 17 06:29:01.396923 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.396869 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:29:01.398298 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.398274 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:01.400614 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.400590 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3b164bbd-serving-cert\"" Jul 17 06:29:01.401204 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.401048 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 06:29:01.401204 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.401083 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpsw\"" Jul 17 06:29:01.410648 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.410588 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr"] Jul 17 06:29:01.472190 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.472095 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:29:01.475044 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.475016 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6cb524-b5d9-421f-9274-33be9bb4151a-openshift-service-ca\") pod \"maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr\" (UID: \"6c6cb524-b5d9-421f-9274-33be9bb4151a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:01.475171 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.475052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k24w5\" (UniqueName: \"kubernetes.io/projected/6c6cb524-b5d9-421f-9274-33be9bb4151a-kube-api-access-k24w5\") pod \"maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr\" (UID: \"6c6cb524-b5d9-421f-9274-33be9bb4151a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:01.475171 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.475148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6cb524-b5d9-421f-9274-33be9bb4151a-maas-api-tls\") pod \"maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr\" (UID: \"6c6cb524-b5d9-421f-9274-33be9bb4151a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:01.575683 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.575641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6cb524-b5d9-421f-9274-33be9bb4151a-openshift-service-ca\") pod \"maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr\" (UID: \"6c6cb524-b5d9-421f-9274-33be9bb4151a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:01.575891 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.575708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k24w5\" (UniqueName: \"kubernetes.io/projected/6c6cb524-b5d9-421f-9274-33be9bb4151a-kube-api-access-k24w5\") pod \"maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr\" (UID: \"6c6cb524-b5d9-421f-9274-33be9bb4151a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:01.575891 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.575793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6cb524-b5d9-421f-9274-33be9bb4151a-maas-api-tls\") pod \"maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr\" (UID: \"6c6cb524-b5d9-421f-9274-33be9bb4151a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:01.576012 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:29:01.575934 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-serving-cert: secret "maas-api-e2e-mt-3b164bbd-serving-cert" not found Jul 17 06:29:01.576012 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:29:01.575998 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6cb524-b5d9-421f-9274-33be9bb4151a-maas-api-tls podName:6c6cb524-b5d9-421f-9274-33be9bb4151a nodeName:}" failed. No retries permitted until 2026-07-17 06:29:02.07597828 +0000 UTC m=+2866.820657590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c6cb524-b5d9-421f-9274-33be9bb4151a-maas-api-tls") pod "maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" (UID: "6c6cb524-b5d9-421f-9274-33be9bb4151a") : secret "maas-api-e2e-mt-3b164bbd-serving-cert" not found Jul 17 06:29:01.576360 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.576335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6cb524-b5d9-421f-9274-33be9bb4151a-openshift-service-ca\") pod \"maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr\" (UID: \"6c6cb524-b5d9-421f-9274-33be9bb4151a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:01.588017 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.587984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k24w5\" (UniqueName: \"kubernetes.io/projected/6c6cb524-b5d9-421f-9274-33be9bb4151a-kube-api-access-k24w5\") pod \"maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr\" (UID: \"6c6cb524-b5d9-421f-9274-33be9bb4151a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:01.720101 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:01.720056 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:29:02.080021 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:02.079983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6cb524-b5d9-421f-9274-33be9bb4151a-maas-api-tls\") pod \"maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr\" (UID: \"6c6cb524-b5d9-421f-9274-33be9bb4151a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:02.082494 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:02.082471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6cb524-b5d9-421f-9274-33be9bb4151a-maas-api-tls\") pod \"maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr\" (UID: \"6c6cb524-b5d9-421f-9274-33be9bb4151a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:02.121234 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:02.121193 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:29:02.223378 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:02.223344 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:29:02.311595 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:02.311552 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:02.441896 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:02.441870 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr"] Jul 17 06:29:02.444202 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:29:02.444176 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6cb524_b5d9_421f_9274_33be9bb4151a.slice/crio-28cd57a893f63afe5e3d37b5fbe37a50f411475901a299f6b8b7782c96117643 WatchSource:0}: Error finding container 28cd57a893f63afe5e3d37b5fbe37a50f411475901a299f6b8b7782c96117643: Status 404 returned error can't find the container with id 28cd57a893f63afe5e3d37b5fbe37a50f411475901a299f6b8b7782c96117643 Jul 17 06:29:02.884718 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:02.884683 2580 scope.go:117] "RemoveContainer" containerID="83aff69f8eaff359f056078e7aa0a4ac5c223b2bd6123094481a1c5e53e019ea" Jul 17 06:29:02.890378 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:02.890354 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e03f6876-027c-49f9-ac0b-1fa72a788de9" path="/var/lib/kubelet/pods/e03f6876-027c-49f9-ac0b-1fa72a788de9/volumes" Jul 17 06:29:03.182197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:03.182159 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" event={"ID":"56b4e63d-b4e0-4dce-a4b9-5acb4022d838","Type":"ContainerStarted","Data":"5bf5acda053022026d9157655b61da70f6441e21e46cbaf397248084f8a00813"} Jul 17 06:29:03.183638 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:03.183608 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" event={"ID":"6c6cb524-b5d9-421f-9274-33be9bb4151a","Type":"ContainerStarted","Data":"a32fb13ea83148c83fe57ad4d6e9de69efd3c319e5754c3e10e657f43d2ce954"} Jul 17 06:29:03.183765 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:03.183650 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" event={"ID":"6c6cb524-b5d9-421f-9274-33be9bb4151a","Type":"ContainerStarted","Data":"28cd57a893f63afe5e3d37b5fbe37a50f411475901a299f6b8b7782c96117643"} Jul 17 06:29:03.183765 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:03.183699 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:03.217042 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:03.216993 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" podStartSLOduration=1.8970873670000001 podStartE2EDuration="2.216978665s" totalImagesPullingTime="319.891298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:29:03.214763549 +0000 UTC m=+2867.959442876" watchObservedRunningTime="2026-07-17 06:29:03.216978665 +0000 UTC m=+2867.961657993" Jul 17 06:29:04.191663 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:04.191629 2580 generic.go:356] "Generic (PLEG): container finished" podID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" containerID="5bf5acda053022026d9157655b61da70f6441e21e46cbaf397248084f8a00813" exitCode=6 Jul 17 06:29:04.192081 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:04.191706 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" event={"ID":"56b4e63d-b4e0-4dce-a4b9-5acb4022d838","Type":"ContainerDied","Data":"5bf5acda053022026d9157655b61da70f6441e21e46cbaf397248084f8a00813"} Jul 17 06:29:04.192081 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:04.191774 2580 scope.go:117] "RemoveContainer" containerID="83aff69f8eaff359f056078e7aa0a4ac5c223b2bd6123094481a1c5e53e019ea" Jul 17 06:29:04.192081 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:04.192066 2580 scope.go:117] "RemoveContainer" containerID="5bf5acda053022026d9157655b61da70f6441e21e46cbaf397248084f8a00813" Jul 17 06:29:04.192287 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:29:04.192265 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-4clxn_odh-ai-gateway-infra(56b4e63d-b4e0-4dce-a4b9-5acb4022d838)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" podUID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" Jul 17 06:29:04.211517 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:04.211493 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn"] Jul 17 06:29:05.326924 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:05.326899 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" Jul 17 06:29:05.410702 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:05.410662 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b4e63d-b4e0-4dce-a4b9-5acb4022d838-maas-api-service-ca\") pod \"56b4e63d-b4e0-4dce-a4b9-5acb4022d838\" (UID: \"56b4e63d-b4e0-4dce-a4b9-5acb4022d838\") " Jul 17 06:29:05.411061 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:05.411040 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b4e63d-b4e0-4dce-a4b9-5acb4022d838-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "56b4e63d-b4e0-4dce-a4b9-5acb4022d838" (UID: "56b4e63d-b4e0-4dce-a4b9-5acb4022d838"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:29:05.511962 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:05.511871 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b4e63d-b4e0-4dce-a4b9-5acb4022d838-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:29:05.773608 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:05.773512 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:29:06.203567 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:06.203534 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" Jul 17 06:29:06.203567 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:06.203559 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn" event={"ID":"56b4e63d-b4e0-4dce-a4b9-5acb4022d838","Type":"ContainerDied","Data":"94e8889c2e6ef1775dd61a46e8213d613e7c13ab6bfcd5d6adea3bb9199b369f"} Jul 17 06:29:06.203807 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:06.203618 2580 scope.go:117] "RemoveContainer" containerID="5bf5acda053022026d9157655b61da70f6441e21e46cbaf397248084f8a00813" Jul 17 06:29:06.226522 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:06.226488 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn"] Jul 17 06:29:06.230874 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:06.230853 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-4clxn"] Jul 17 06:29:06.682287 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:06.682252 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:29:06.889435 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:06.889402 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" path="/var/lib/kubelet/pods/56b4e63d-b4e0-4dce-a4b9-5acb4022d838/volumes" Jul 17 06:29:09.197274 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:09.197187 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:10.022281 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:10.022250 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:29:16.133252 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.133216 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp"] Jul 17 06:29:16.133762 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.133547 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e03f6876-027c-49f9-ac0b-1fa72a788de9" containerName="revoke-keys" Jul 17 06:29:16.133762 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.133559 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e03f6876-027c-49f9-ac0b-1fa72a788de9" containerName="revoke-keys" Jul 17 06:29:16.133762 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.133570 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" containerName="revoke-keys" Jul 17 06:29:16.133762 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.133575 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" containerName="revoke-keys" Jul 17 06:29:16.133762 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.133587 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" containerName="revoke-keys" Jul 17 06:29:16.133762 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.133592 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" containerName="revoke-keys" Jul 17 06:29:16.133762 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.133601 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" containerName="revoke-keys" Jul 17 06:29:16.133762 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.133606 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" containerName="revoke-keys" Jul 17 06:29:16.133762 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.133666 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e03f6876-027c-49f9-ac0b-1fa72a788de9" containerName="revoke-keys" Jul 17 06:29:16.133762 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.133674 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" containerName="revoke-keys" Jul 17 06:29:16.133762 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.133681 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" containerName="revoke-keys" Jul 17 06:29:16.133762 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.133688 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b4e63d-b4e0-4dce-a4b9-5acb4022d838" containerName="revoke-keys" Jul 17 06:29:16.137946 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.137928 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" Jul 17 06:29:16.140434 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.140411 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:29:16.144314 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.144289 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp"] Jul 17 06:29:16.207960 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.207923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62adf5a0-dc05-42c5-a982-a899bcc17b22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp\" (UID: \"62adf5a0-dc05-42c5-a982-a899bcc17b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" Jul 17 06:29:16.309157 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.309125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62adf5a0-dc05-42c5-a982-a899bcc17b22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp\" (UID: \"62adf5a0-dc05-42c5-a982-a899bcc17b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" Jul 17 06:29:16.309719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.309701 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62adf5a0-dc05-42c5-a982-a899bcc17b22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp\" (UID: \"62adf5a0-dc05-42c5-a982-a899bcc17b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" Jul 17 06:29:16.452221 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.452137 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:29:16.459968 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.459940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" Jul 17 06:29:16.810968 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:16.810756 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp"] Jul 17 06:29:16.813482 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:29:16.813453 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62adf5a0_dc05_42c5_a982_a899bcc17b22.slice/crio-c5b81204849efe878e3917441c30e040ac5d585a3915ceee902f5c5a661129a7 WatchSource:0}: Error finding container c5b81204849efe878e3917441c30e040ac5d585a3915ceee902f5c5a661129a7: Status 404 returned error can't find the container with id c5b81204849efe878e3917441c30e040ac5d585a3915ceee902f5c5a661129a7 Jul 17 06:29:17.257984 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:17.257946 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" event={"ID":"62adf5a0-dc05-42c5-a982-a899bcc17b22","Type":"ContainerStarted","Data":"46eea56220d05b2d27f1b3bee640f62f7ce5855c4ae7e8a6ad1a57a0c0c35708"} Jul 17 06:29:17.258435 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:17.258017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" event={"ID":"62adf5a0-dc05-42c5-a982-a899bcc17b22","Type":"ContainerStarted","Data":"c5b81204849efe878e3917441c30e040ac5d585a3915ceee902f5c5a661129a7"} Jul 17 06:29:18.264061 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:18.264024 2580 generic.go:356] "Generic (PLEG): container finished" podID="62adf5a0-dc05-42c5-a982-a899bcc17b22" containerID="46eea56220d05b2d27f1b3bee640f62f7ce5855c4ae7e8a6ad1a57a0c0c35708" exitCode=6 Jul 17 06:29:18.264543 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:18.264086 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" event={"ID":"62adf5a0-dc05-42c5-a982-a899bcc17b22","Type":"ContainerDied","Data":"46eea56220d05b2d27f1b3bee640f62f7ce5855c4ae7e8a6ad1a57a0c0c35708"} Jul 17 06:29:18.264543 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:18.264249 2580 scope.go:117] "RemoveContainer" containerID="46eea56220d05b2d27f1b3bee640f62f7ce5855c4ae7e8a6ad1a57a0c0c35708" Jul 17 06:29:19.269802 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:19.269769 2580 generic.go:356] "Generic (PLEG): container finished" podID="62adf5a0-dc05-42c5-a982-a899bcc17b22" containerID="cedc98d6b6e8eca0ef872daabb3d0678d8641b0278abd7549517208a4a1ea624" exitCode=6 Jul 17 06:29:19.270226 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:19.269847 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" event={"ID":"62adf5a0-dc05-42c5-a982-a899bcc17b22","Type":"ContainerDied","Data":"cedc98d6b6e8eca0ef872daabb3d0678d8641b0278abd7549517208a4a1ea624"} Jul 17 06:29:19.270226 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:19.269899 2580 scope.go:117] "RemoveContainer" containerID="46eea56220d05b2d27f1b3bee640f62f7ce5855c4ae7e8a6ad1a57a0c0c35708" Jul 17 06:29:19.270226 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:19.270085 2580 scope.go:117] "RemoveContainer" containerID="cedc98d6b6e8eca0ef872daabb3d0678d8641b0278abd7549517208a4a1ea624" Jul 17 06:29:19.270347 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:29:19.270266 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp_odh-ai-gateway-infra(62adf5a0-dc05-42c5-a982-a899bcc17b22)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" podUID="62adf5a0-dc05-42c5-a982-a899bcc17b22" Jul 17 06:29:20.276105 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:20.276075 2580 scope.go:117] "RemoveContainer" containerID="cedc98d6b6e8eca0ef872daabb3d0678d8641b0278abd7549517208a4a1ea624" Jul 17 06:29:20.276496 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:29:20.276275 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp_odh-ai-gateway-infra(62adf5a0-dc05-42c5-a982-a899bcc17b22)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" podUID="62adf5a0-dc05-42c5-a982-a899bcc17b22" Jul 17 06:29:22.480054 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:22.480021 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:29:30.306577 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:30.306536 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:29:32.587532 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:32.587496 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:29:33.884302 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:33.884265 2580 scope.go:117] "RemoveContainer" containerID="cedc98d6b6e8eca0ef872daabb3d0678d8641b0278abd7549517208a4a1ea624" Jul 17 06:29:34.336604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:34.336520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" event={"ID":"62adf5a0-dc05-42c5-a982-a899bcc17b22","Type":"ContainerStarted","Data":"3ca369f759ade7f3ded9f05d94aef6865faa43459816df6c78dd9afc2e00f110"} Jul 17 06:29:34.353822 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:34.353776 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" podStartSLOduration=18.353761546 podStartE2EDuration="18.353761546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:29:17.292228631 +0000 UTC m=+2882.036907971" watchObservedRunningTime="2026-07-17 06:29:34.353761546 +0000 UTC m=+2899.098440874" Jul 17 06:29:35.342940 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:35.342900 2580 generic.go:356] "Generic (PLEG): container finished" podID="62adf5a0-dc05-42c5-a982-a899bcc17b22" containerID="3ca369f759ade7f3ded9f05d94aef6865faa43459816df6c78dd9afc2e00f110" exitCode=6 Jul 17 06:29:35.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:35.342978 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" event={"ID":"62adf5a0-dc05-42c5-a982-a899bcc17b22","Type":"ContainerDied","Data":"3ca369f759ade7f3ded9f05d94aef6865faa43459816df6c78dd9afc2e00f110"} Jul 17 06:29:35.343354 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:35.343076 2580 scope.go:117] "RemoveContainer" containerID="cedc98d6b6e8eca0ef872daabb3d0678d8641b0278abd7549517208a4a1ea624" Jul 17 06:29:35.343504 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:35.343376 2580 scope.go:117] "RemoveContainer" containerID="3ca369f759ade7f3ded9f05d94aef6865faa43459816df6c78dd9afc2e00f110" Jul 17 06:29:35.343627 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:29:35.343606 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp_odh-ai-gateway-infra(62adf5a0-dc05-42c5-a982-a899bcc17b22)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" podUID="62adf5a0-dc05-42c5-a982-a899bcc17b22" Jul 17 06:29:35.368369 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:35.368345 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp"] Jul 17 06:29:36.484238 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:36.484214 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" Jul 17 06:29:36.585101 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:36.585065 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62adf5a0-dc05-42c5-a982-a899bcc17b22-maas-api-service-ca\") pod \"62adf5a0-dc05-42c5-a982-a899bcc17b22\" (UID: \"62adf5a0-dc05-42c5-a982-a899bcc17b22\") " Jul 17 06:29:36.585465 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:36.585441 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62adf5a0-dc05-42c5-a982-a899bcc17b22-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62adf5a0-dc05-42c5-a982-a899bcc17b22" (UID: "62adf5a0-dc05-42c5-a982-a899bcc17b22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:29:36.686556 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:36.686514 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62adf5a0-dc05-42c5-a982-a899bcc17b22-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:29:37.353708 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:37.353672 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" event={"ID":"62adf5a0-dc05-42c5-a982-a899bcc17b22","Type":"ContainerDied","Data":"c5b81204849efe878e3917441c30e040ac5d585a3915ceee902f5c5a661129a7"} Jul 17 06:29:37.353901 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:37.353733 2580 scope.go:117] "RemoveContainer" containerID="3ca369f759ade7f3ded9f05d94aef6865faa43459816df6c78dd9afc2e00f110" Jul 17 06:29:37.353901 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:37.353681 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp" Jul 17 06:29:37.370856 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:37.370829 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp"] Jul 17 06:29:37.374777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:37.374752 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-ffrfp"] Jul 17 06:29:38.890141 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:38.890095 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62adf5a0-dc05-42c5-a982-a899bcc17b22" path="/var/lib/kubelet/pods/62adf5a0-dc05-42c5-a982-a899bcc17b22/volumes" Jul 17 06:29:38.992559 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:38.992521 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm"] Jul 17 06:29:38.992876 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:38.992863 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62adf5a0-dc05-42c5-a982-a899bcc17b22" containerName="revoke-keys" Jul 17 06:29:38.992921 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:38.992877 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62adf5a0-dc05-42c5-a982-a899bcc17b22" containerName="revoke-keys" Jul 17 06:29:38.992921 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:38.992888 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62adf5a0-dc05-42c5-a982-a899bcc17b22" containerName="revoke-keys" Jul 17 06:29:38.992921 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:38.992893 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62adf5a0-dc05-42c5-a982-a899bcc17b22" containerName="revoke-keys" Jul 17 06:29:38.993016 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:38.992944 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="62adf5a0-dc05-42c5-a982-a899bcc17b22" containerName="revoke-keys" Jul 17 06:29:38.993016 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:38.992955 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="62adf5a0-dc05-42c5-a982-a899bcc17b22" containerName="revoke-keys" Jul 17 06:29:38.997256 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:38.997236 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm" Jul 17 06:29:38.999422 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:38.999402 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:29:39.003156 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:39.003122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed250e97-e673-43b0-9db0-25c99a2079a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm\" (UID: \"ed250e97-e673-43b0-9db0-25c99a2079a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm" Jul 17 06:29:39.003398 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:39.003364 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm"] Jul 17 06:29:39.104686 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:39.104643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed250e97-e673-43b0-9db0-25c99a2079a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm\" (UID: \"ed250e97-e673-43b0-9db0-25c99a2079a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm" Jul 17 06:29:39.105351 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:39.105320 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed250e97-e673-43b0-9db0-25c99a2079a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm\" (UID: \"ed250e97-e673-43b0-9db0-25c99a2079a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm" Jul 17 06:29:39.309343 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:39.309235 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm" Jul 17 06:29:39.448617 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:39.448588 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm"] Jul 17 06:29:39.451556 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:29:39.451524 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded250e97_e673_43b0_9db0_25c99a2079a8.slice/crio-b20e9df68d4dfc2fd47e6d8f3aaacaeec37944f4ed157c66edfa0f23e86e17ec WatchSource:0}: Error finding container b20e9df68d4dfc2fd47e6d8f3aaacaeec37944f4ed157c66edfa0f23e86e17ec: Status 404 returned error can't find the container with id b20e9df68d4dfc2fd47e6d8f3aaacaeec37944f4ed157c66edfa0f23e86e17ec Jul 17 06:29:40.370771 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:40.370729 2580 generic.go:356] "Generic (PLEG): container finished" podID="ed250e97-e673-43b0-9db0-25c99a2079a8" containerID="0ff54b64d80aee1bb081b8e72fd1d01acf8a0c4dca0ae048af1c7680e1cd25b8" exitCode=0 Jul 17 06:29:40.371164 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:40.370822 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm" event={"ID":"ed250e97-e673-43b0-9db0-25c99a2079a8","Type":"ContainerDied","Data":"0ff54b64d80aee1bb081b8e72fd1d01acf8a0c4dca0ae048af1c7680e1cd25b8"} Jul 17 06:29:40.371164 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:40.370874 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm" event={"ID":"ed250e97-e673-43b0-9db0-25c99a2079a8","Type":"ContainerStarted","Data":"b20e9df68d4dfc2fd47e6d8f3aaacaeec37944f4ed157c66edfa0f23e86e17ec"} Jul 17 06:29:41.511963 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:41.511936 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm" Jul 17 06:29:41.521208 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:41.521175 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed250e97-e673-43b0-9db0-25c99a2079a8-maas-api-service-ca\") pod \"ed250e97-e673-43b0-9db0-25c99a2079a8\" (UID: \"ed250e97-e673-43b0-9db0-25c99a2079a8\") " Jul 17 06:29:41.521546 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:41.521522 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed250e97-e673-43b0-9db0-25c99a2079a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed250e97-e673-43b0-9db0-25c99a2079a8" (UID: "ed250e97-e673-43b0-9db0-25c99a2079a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:29:41.622303 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:41.622263 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed250e97-e673-43b0-9db0-25c99a2079a8-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:29:42.381376 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:42.381340 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm" Jul 17 06:29:42.381376 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:42.381374 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm" event={"ID":"ed250e97-e673-43b0-9db0-25c99a2079a8","Type":"ContainerDied","Data":"b20e9df68d4dfc2fd47e6d8f3aaacaeec37944f4ed157c66edfa0f23e86e17ec"} Jul 17 06:29:42.381644 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:42.381455 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b20e9df68d4dfc2fd47e6d8f3aaacaeec37944f4ed157c66edfa0f23e86e17ec" Jul 17 06:29:52.788892 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:52.788849 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:29:54.476409 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:54.475878 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr"] Jul 17 06:29:54.476409 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:54.476181 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" podUID="6c6cb524-b5d9-421f-9274-33be9bb4151a" containerName="maas-api" containerID="cri-o://a32fb13ea83148c83fe57ad4d6e9de69efd3c319e5754c3e10e657f43d2ce954" gracePeriod=30 Jul 17 06:29:54.504984 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:54.504949 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:29:54.721144 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:54.721115 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:54.838203 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:54.838107 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6cb524-b5d9-421f-9274-33be9bb4151a-maas-api-tls\") pod \"6c6cb524-b5d9-421f-9274-33be9bb4151a\" (UID: \"6c6cb524-b5d9-421f-9274-33be9bb4151a\") " Jul 17 06:29:54.838203 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:54.838175 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k24w5\" (UniqueName: \"kubernetes.io/projected/6c6cb524-b5d9-421f-9274-33be9bb4151a-kube-api-access-k24w5\") pod \"6c6cb524-b5d9-421f-9274-33be9bb4151a\" (UID: \"6c6cb524-b5d9-421f-9274-33be9bb4151a\") " Jul 17 06:29:54.838448 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:54.838213 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6cb524-b5d9-421f-9274-33be9bb4151a-openshift-service-ca\") pod \"6c6cb524-b5d9-421f-9274-33be9bb4151a\" (UID: \"6c6cb524-b5d9-421f-9274-33be9bb4151a\") " Jul 17 06:29:54.838658 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:54.838627 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6cb524-b5d9-421f-9274-33be9bb4151a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c6cb524-b5d9-421f-9274-33be9bb4151a" (UID: "6c6cb524-b5d9-421f-9274-33be9bb4151a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:29:54.840345 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:54.840309 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6cb524-b5d9-421f-9274-33be9bb4151a-kube-api-access-k24w5" (OuterVolumeSpecName: "kube-api-access-k24w5") pod "6c6cb524-b5d9-421f-9274-33be9bb4151a" (UID: "6c6cb524-b5d9-421f-9274-33be9bb4151a"). InnerVolumeSpecName "kube-api-access-k24w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:29:54.840463 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:54.840401 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6cb524-b5d9-421f-9274-33be9bb4151a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c6cb524-b5d9-421f-9274-33be9bb4151a" (UID: "6c6cb524-b5d9-421f-9274-33be9bb4151a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:29:54.939817 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:54.939770 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c6cb524-b5d9-421f-9274-33be9bb4151a-maas-api-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:29:54.939817 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:54.939810 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k24w5\" (UniqueName: \"kubernetes.io/projected/6c6cb524-b5d9-421f-9274-33be9bb4151a-kube-api-access-k24w5\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:29:54.939817 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:54.939825 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c6cb524-b5d9-421f-9274-33be9bb4151a-openshift-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:29:55.440355 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:55.440321 2580 generic.go:356] "Generic (PLEG): container finished" podID="6c6cb524-b5d9-421f-9274-33be9bb4151a" containerID="a32fb13ea83148c83fe57ad4d6e9de69efd3c319e5754c3e10e657f43d2ce954" exitCode=0 Jul 17 06:29:55.440567 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:55.440366 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" Jul 17 06:29:55.440567 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:55.440371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" event={"ID":"6c6cb524-b5d9-421f-9274-33be9bb4151a","Type":"ContainerDied","Data":"a32fb13ea83148c83fe57ad4d6e9de69efd3c319e5754c3e10e657f43d2ce954"} Jul 17 06:29:55.440567 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:55.440508 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr" event={"ID":"6c6cb524-b5d9-421f-9274-33be9bb4151a","Type":"ContainerDied","Data":"28cd57a893f63afe5e3d37b5fbe37a50f411475901a299f6b8b7782c96117643"} Jul 17 06:29:55.440567 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:55.440539 2580 scope.go:117] "RemoveContainer" containerID="a32fb13ea83148c83fe57ad4d6e9de69efd3c319e5754c3e10e657f43d2ce954" Jul 17 06:29:55.449567 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:55.449545 2580 scope.go:117] "RemoveContainer" containerID="a32fb13ea83148c83fe57ad4d6e9de69efd3c319e5754c3e10e657f43d2ce954" Jul 17 06:29:55.449899 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:29:55.449870 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32fb13ea83148c83fe57ad4d6e9de69efd3c319e5754c3e10e657f43d2ce954\": container with ID starting with a32fb13ea83148c83fe57ad4d6e9de69efd3c319e5754c3e10e657f43d2ce954 not found: ID does not exist" containerID="a32fb13ea83148c83fe57ad4d6e9de69efd3c319e5754c3e10e657f43d2ce954" Jul 17 06:29:55.449994 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:55.449909 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32fb13ea83148c83fe57ad4d6e9de69efd3c319e5754c3e10e657f43d2ce954"} err="failed to get container status \"a32fb13ea83148c83fe57ad4d6e9de69efd3c319e5754c3e10e657f43d2ce954\": rpc error: code = NotFound desc = could not find container \"a32fb13ea83148c83fe57ad4d6e9de69efd3c319e5754c3e10e657f43d2ce954\": container with ID starting with a32fb13ea83148c83fe57ad4d6e9de69efd3c319e5754c3e10e657f43d2ce954 not found: ID does not exist" Jul 17 06:29:55.456453 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:55.456425 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr"] Jul 17 06:29:55.463037 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:55.463013 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b164bbd-c99f7fbfb-hbwlr"] Jul 17 06:29:56.889620 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:56.889588 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c6cb524-b5d9-421f-9274-33be9bb4151a" path="/var/lib/kubelet/pods/6c6cb524-b5d9-421f-9274-33be9bb4151a/volumes" Jul 17 06:29:59.086861 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:29:59.086815 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:30:00.142549 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.142509 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737830-2rgdb"] Jul 17 06:30:00.142955 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.142897 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6cb524-b5d9-421f-9274-33be9bb4151a" containerName="maas-api" Jul 17 06:30:00.142955 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.142909 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6cb524-b5d9-421f-9274-33be9bb4151a" containerName="maas-api" Jul 17 06:30:00.142955 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.142927 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62adf5a0-dc05-42c5-a982-a899bcc17b22" containerName="revoke-keys" Jul 17 06:30:00.142955 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.142933 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62adf5a0-dc05-42c5-a982-a899bcc17b22" containerName="revoke-keys" Jul 17 06:30:00.142955 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.142943 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed250e97-e673-43b0-9db0-25c99a2079a8" containerName="revoke-keys" Jul 17 06:30:00.142955 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.142954 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed250e97-e673-43b0-9db0-25c99a2079a8" containerName="revoke-keys" Jul 17 06:30:00.143145 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.143003 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="62adf5a0-dc05-42c5-a982-a899bcc17b22" containerName="revoke-keys" Jul 17 06:30:00.143145 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.143012 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6cb524-b5d9-421f-9274-33be9bb4151a" containerName="maas-api" Jul 17 06:30:00.143145 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.143020 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed250e97-e673-43b0-9db0-25c99a2079a8" containerName="revoke-keys" Jul 17 06:30:00.147561 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.147542 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737830-2rgdb" Jul 17 06:30:00.150161 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.150139 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:30:00.154511 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.154481 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737830-2rgdb"] Jul 17 06:30:00.158741 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.158717 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737830-2rgdb" Jul 17 06:30:00.295579 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.295419 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737830-2rgdb"] Jul 17 06:30:00.297846 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:30:00.297816 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1940eb7c_a67a_4956_89ba_8cfa548b7a10.slice/crio-77a1376d5690b86f0804a8d792614edd01b33c63e9608e5281045ddb11b12de3 WatchSource:0}: Error finding container 77a1376d5690b86f0804a8d792614edd01b33c63e9608e5281045ddb11b12de3: Status 404 returned error can't find the container with id 77a1376d5690b86f0804a8d792614edd01b33c63e9608e5281045ddb11b12de3 Jul 17 06:30:00.465566 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:00.465471 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737830-2rgdb" event={"ID":"1940eb7c-a67a-4956-89ba-8cfa548b7a10","Type":"ContainerStarted","Data":"77a1376d5690b86f0804a8d792614edd01b33c63e9608e5281045ddb11b12de3"} Jul 17 06:30:01.242525 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:01.242487 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:30:01.471617 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:01.471580 2580 generic.go:356] "Generic (PLEG): container finished" podID="1940eb7c-a67a-4956-89ba-8cfa548b7a10" containerID="2f7c84fb9cca502441945c1ecb55317188c66553224ca525d3d4adb3ac727539" exitCode=0 Jul 17 06:30:01.471828 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:01.471681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737830-2rgdb" event={"ID":"1940eb7c-a67a-4956-89ba-8cfa548b7a10","Type":"ContainerDied","Data":"2f7c84fb9cca502441945c1ecb55317188c66553224ca525d3d4adb3ac727539"} Jul 17 06:30:02.604310 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:02.604284 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737830-2rgdb" Jul 17 06:30:03.482750 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:03.482718 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737830-2rgdb" Jul 17 06:30:03.482968 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:03.482720 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737830-2rgdb" event={"ID":"1940eb7c-a67a-4956-89ba-8cfa548b7a10","Type":"ContainerDied","Data":"77a1376d5690b86f0804a8d792614edd01b33c63e9608e5281045ddb11b12de3"} Jul 17 06:30:03.482968 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:03.482856 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77a1376d5690b86f0804a8d792614edd01b33c63e9608e5281045ddb11b12de3" Jul 17 06:30:03.592359 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:03.592321 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:30:06.283545 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:06.283511 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:30:12.490655 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:12.490619 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z"] Jul 17 06:30:12.491089 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:12.490954 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1940eb7c-a67a-4956-89ba-8cfa548b7a10" containerName="cleanup" Jul 17 06:30:12.491089 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:12.490965 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1940eb7c-a67a-4956-89ba-8cfa548b7a10" containerName="cleanup" Jul 17 06:30:12.491089 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:12.491016 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1940eb7c-a67a-4956-89ba-8cfa548b7a10" containerName="cleanup" Jul 17 06:30:12.494177 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:12.494157 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" Jul 17 06:30:12.496627 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:12.496604 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:30:12.496771 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:12.496680 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 06:30:12.501101 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:12.501073 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z"] Jul 17 06:30:12.605660 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:12.605605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec992266-7ee4-4992-bd84-155c699120df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-s9t7z\" (UID: \"ec992266-7ee4-4992-bd84-155c699120df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" Jul 17 06:30:12.706979 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:12.706919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec992266-7ee4-4992-bd84-155c699120df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-s9t7z\" (UID: \"ec992266-7ee4-4992-bd84-155c699120df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" Jul 17 06:30:12.707625 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:12.707600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec992266-7ee4-4992-bd84-155c699120df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-s9t7z\" (UID: \"ec992266-7ee4-4992-bd84-155c699120df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" Jul 17 06:30:12.806872 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:12.806762 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" Jul 17 06:30:12.941734 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:12.941690 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z"] Jul 17 06:30:12.948304 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:30:12.948274 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec992266_7ee4_4992_bd84_155c699120df.slice/crio-84f4198e6f36d1ee6e86ce0d521dea8a36d39042fb3f502325013b62efa10bef WatchSource:0}: Error finding container 84f4198e6f36d1ee6e86ce0d521dea8a36d39042fb3f502325013b62efa10bef: Status 404 returned error can't find the container with id 84f4198e6f36d1ee6e86ce0d521dea8a36d39042fb3f502325013b62efa10bef Jul 17 06:30:13.530522 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:13.530416 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" event={"ID":"ec992266-7ee4-4992-bd84-155c699120df","Type":"ContainerStarted","Data":"451645c8854e2bc0408b928209b21a184e999ff88da3618d59137decfc15328b"} Jul 17 06:30:13.530522 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:13.530485 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" event={"ID":"ec992266-7ee4-4992-bd84-155c699120df","Type":"ContainerStarted","Data":"84f4198e6f36d1ee6e86ce0d521dea8a36d39042fb3f502325013b62efa10bef"} Jul 17 06:30:13.548500 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:13.548445 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" podStartSLOduration=1.548428328 podStartE2EDuration="1.548428328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:30:13.547722266 +0000 UTC m=+2938.292401598" watchObservedRunningTime="2026-07-17 06:30:13.548428328 +0000 UTC m=+2938.293107656" Jul 17 06:30:14.537670 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:14.537634 2580 generic.go:356] "Generic (PLEG): container finished" podID="ec992266-7ee4-4992-bd84-155c699120df" containerID="451645c8854e2bc0408b928209b21a184e999ff88da3618d59137decfc15328b" exitCode=6 Jul 17 06:30:14.538169 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:14.537705 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" event={"ID":"ec992266-7ee4-4992-bd84-155c699120df","Type":"ContainerDied","Data":"451645c8854e2bc0408b928209b21a184e999ff88da3618d59137decfc15328b"} Jul 17 06:30:14.538169 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:14.537944 2580 scope.go:117] "RemoveContainer" containerID="451645c8854e2bc0408b928209b21a184e999ff88da3618d59137decfc15328b" Jul 17 06:30:15.544670 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:15.544565 2580 generic.go:356] "Generic (PLEG): container finished" podID="ec992266-7ee4-4992-bd84-155c699120df" containerID="0e8066447deb14a2f759647b72b02890b98de13ebd0f37ecdece6f944162e5a8" exitCode=6 Jul 17 06:30:15.545135 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:15.544678 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" event={"ID":"ec992266-7ee4-4992-bd84-155c699120df","Type":"ContainerDied","Data":"0e8066447deb14a2f759647b72b02890b98de13ebd0f37ecdece6f944162e5a8"} Jul 17 06:30:15.545135 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:15.544743 2580 scope.go:117] "RemoveContainer" containerID="451645c8854e2bc0408b928209b21a184e999ff88da3618d59137decfc15328b" Jul 17 06:30:15.545135 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:15.544960 2580 scope.go:117] "RemoveContainer" containerID="0e8066447deb14a2f759647b72b02890b98de13ebd0f37ecdece6f944162e5a8" Jul 17 06:30:15.545254 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:30:15.545197 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-s9t7z_odh-ai-gateway-infra(ec992266-7ee4-4992-bd84-155c699120df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" podUID="ec992266-7ee4-4992-bd84-155c699120df" Jul 17 06:30:16.184599 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:16.184560 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8"] Jul 17 06:30:16.189012 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:16.188993 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" Jul 17 06:30:16.196457 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:16.195988 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8"] Jul 17 06:30:16.341256 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:16.341202 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f78730-8272-4a3b-a5ad-9c5312529db2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8\" (UID: \"49f78730-8272-4a3b-a5ad-9c5312529db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" Jul 17 06:30:16.441951 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:16.441846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f78730-8272-4a3b-a5ad-9c5312529db2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8\" (UID: \"49f78730-8272-4a3b-a5ad-9c5312529db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" Jul 17 06:30:16.442517 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:16.442498 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f78730-8272-4a3b-a5ad-9c5312529db2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8\" (UID: \"49f78730-8272-4a3b-a5ad-9c5312529db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" Jul 17 06:30:16.500511 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:16.500468 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" Jul 17 06:30:16.553452 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:16.553425 2580 scope.go:117] "RemoveContainer" containerID="0e8066447deb14a2f759647b72b02890b98de13ebd0f37ecdece6f944162e5a8" Jul 17 06:30:16.553840 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:30:16.553636 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-s9t7z_odh-ai-gateway-infra(ec992266-7ee4-4992-bd84-155c699120df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" podUID="ec992266-7ee4-4992-bd84-155c699120df" Jul 17 06:30:16.643936 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:16.643770 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8"] Jul 17 06:30:16.646552 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:30:16.646502 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f78730_8272_4a3b_a5ad_9c5312529db2.slice/crio-bead8e5027d2d15fc665e27f1a759327d1b52af31816d220eba1ff55f2a76097 WatchSource:0}: Error finding container bead8e5027d2d15fc665e27f1a759327d1b52af31816d220eba1ff55f2a76097: Status 404 returned error can't find the container with id bead8e5027d2d15fc665e27f1a759327d1b52af31816d220eba1ff55f2a76097 Jul 17 06:30:17.560056 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:17.559966 2580 generic.go:356] "Generic (PLEG): container finished" podID="49f78730-8272-4a3b-a5ad-9c5312529db2" containerID="2a3e091bc3e036c7d9021b0e6a74664b8d78f64733c515f6d30134e67ccb1997" exitCode=6 Jul 17 06:30:17.560056 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:17.560017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" event={"ID":"49f78730-8272-4a3b-a5ad-9c5312529db2","Type":"ContainerDied","Data":"2a3e091bc3e036c7d9021b0e6a74664b8d78f64733c515f6d30134e67ccb1997"} Jul 17 06:30:17.560056 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:17.560054 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" event={"ID":"49f78730-8272-4a3b-a5ad-9c5312529db2","Type":"ContainerStarted","Data":"bead8e5027d2d15fc665e27f1a759327d1b52af31816d220eba1ff55f2a76097"} Jul 17 06:30:17.560545 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:17.560272 2580 scope.go:117] "RemoveContainer" containerID="2a3e091bc3e036c7d9021b0e6a74664b8d78f64733c515f6d30134e67ccb1997" Jul 17 06:30:18.567748 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:18.567656 2580 generic.go:356] "Generic (PLEG): container finished" podID="49f78730-8272-4a3b-a5ad-9c5312529db2" containerID="79eca1c15be445017243307de3e07ed16a22e535a612372095b9849494fb5863" exitCode=6 Jul 17 06:30:18.567748 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:18.567705 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" event={"ID":"49f78730-8272-4a3b-a5ad-9c5312529db2","Type":"ContainerDied","Data":"79eca1c15be445017243307de3e07ed16a22e535a612372095b9849494fb5863"} Jul 17 06:30:18.568232 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:18.567770 2580 scope.go:117] "RemoveContainer" containerID="2a3e091bc3e036c7d9021b0e6a74664b8d78f64733c515f6d30134e67ccb1997" Jul 17 06:30:18.568232 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:18.567925 2580 scope.go:117] "RemoveContainer" containerID="79eca1c15be445017243307de3e07ed16a22e535a612372095b9849494fb5863" Jul 17 06:30:18.568232 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:30:18.568153 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8_odh-ai-gateway-infra(49f78730-8272-4a3b-a5ad-9c5312529db2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" podUID="49f78730-8272-4a3b-a5ad-9c5312529db2" Jul 17 06:30:19.574768 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:19.574741 2580 scope.go:117] "RemoveContainer" containerID="79eca1c15be445017243307de3e07ed16a22e535a612372095b9849494fb5863" Jul 17 06:30:19.575199 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:30:19.574928 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8_odh-ai-gateway-infra(49f78730-8272-4a3b-a5ad-9c5312529db2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" podUID="49f78730-8272-4a3b-a5ad-9c5312529db2" Jul 17 06:30:27.884569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:27.884534 2580 scope.go:117] "RemoveContainer" containerID="0e8066447deb14a2f759647b72b02890b98de13ebd0f37ecdece6f944162e5a8" Jul 17 06:30:28.617493 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:28.617456 2580 generic.go:356] "Generic (PLEG): container finished" podID="ec992266-7ee4-4992-bd84-155c699120df" containerID="52e74e338b8bd1f7f793094f6d8dcac840e3f339d422558b7bfcda8beecca486" exitCode=6 Jul 17 06:30:28.617665 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:28.617532 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" event={"ID":"ec992266-7ee4-4992-bd84-155c699120df","Type":"ContainerDied","Data":"52e74e338b8bd1f7f793094f6d8dcac840e3f339d422558b7bfcda8beecca486"} Jul 17 06:30:28.617665 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:28.617590 2580 scope.go:117] "RemoveContainer" containerID="0e8066447deb14a2f759647b72b02890b98de13ebd0f37ecdece6f944162e5a8" Jul 17 06:30:28.617897 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:28.617873 2580 scope.go:117] "RemoveContainer" containerID="52e74e338b8bd1f7f793094f6d8dcac840e3f339d422558b7bfcda8beecca486" Jul 17 06:30:28.618129 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:30:28.618104 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-s9t7z_odh-ai-gateway-infra(ec992266-7ee4-4992-bd84-155c699120df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" podUID="ec992266-7ee4-4992-bd84-155c699120df" Jul 17 06:30:29.655741 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:29.655703 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z"] Jul 17 06:30:29.795424 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:29.795368 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" Jul 17 06:30:29.859039 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:29.858994 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec992266-7ee4-4992-bd84-155c699120df-maas-api-service-ca\") pod \"ec992266-7ee4-4992-bd84-155c699120df\" (UID: \"ec992266-7ee4-4992-bd84-155c699120df\") " Jul 17 06:30:29.859411 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:29.859361 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec992266-7ee4-4992-bd84-155c699120df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec992266-7ee4-4992-bd84-155c699120df" (UID: "ec992266-7ee4-4992-bd84-155c699120df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:30:29.959687 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:29.959577 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec992266-7ee4-4992-bd84-155c699120df-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:30:30.628803 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:30.628761 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" event={"ID":"ec992266-7ee4-4992-bd84-155c699120df","Type":"ContainerDied","Data":"84f4198e6f36d1ee6e86ce0d521dea8a36d39042fb3f502325013b62efa10bef"} Jul 17 06:30:30.628803 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:30.628788 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z" Jul 17 06:30:30.629101 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:30.628840 2580 scope.go:117] "RemoveContainer" containerID="52e74e338b8bd1f7f793094f6d8dcac840e3f339d422558b7bfcda8beecca486" Jul 17 06:30:30.657694 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:30.657649 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z"] Jul 17 06:30:30.659379 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:30.659348 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-s9t7z"] Jul 17 06:30:30.884808 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:30.884717 2580 scope.go:117] "RemoveContainer" containerID="79eca1c15be445017243307de3e07ed16a22e535a612372095b9849494fb5863" Jul 17 06:30:30.890127 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:30.890092 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec992266-7ee4-4992-bd84-155c699120df" path="/var/lib/kubelet/pods/ec992266-7ee4-4992-bd84-155c699120df/volumes" Jul 17 06:30:31.636344 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:31.636307 2580 generic.go:356] "Generic (PLEG): container finished" podID="49f78730-8272-4a3b-a5ad-9c5312529db2" containerID="b9690580916937766c7e1b755c54315286bc443bba74ce4bb9da4eda0a68a3f7" exitCode=6 Jul 17 06:30:31.636547 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:31.636401 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" event={"ID":"49f78730-8272-4a3b-a5ad-9c5312529db2","Type":"ContainerDied","Data":"b9690580916937766c7e1b755c54315286bc443bba74ce4bb9da4eda0a68a3f7"} Jul 17 06:30:31.636547 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:31.636467 2580 scope.go:117] "RemoveContainer" containerID="79eca1c15be445017243307de3e07ed16a22e535a612372095b9849494fb5863" Jul 17 06:30:31.636798 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:31.636769 2580 scope.go:117] "RemoveContainer" containerID="b9690580916937766c7e1b755c54315286bc443bba74ce4bb9da4eda0a68a3f7" Jul 17 06:30:31.637055 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:30:31.637033 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8_odh-ai-gateway-infra(49f78730-8272-4a3b-a5ad-9c5312529db2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" podUID="49f78730-8272-4a3b-a5ad-9c5312529db2" Jul 17 06:30:32.661961 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:32.661924 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8"] Jul 17 06:30:32.797569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:32.797542 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" Jul 17 06:30:32.883488 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:32.883445 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f78730-8272-4a3b-a5ad-9c5312529db2-maas-api-service-ca\") pod \"49f78730-8272-4a3b-a5ad-9c5312529db2\" (UID: \"49f78730-8272-4a3b-a5ad-9c5312529db2\") " Jul 17 06:30:32.883822 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:32.883797 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f78730-8272-4a3b-a5ad-9c5312529db2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "49f78730-8272-4a3b-a5ad-9c5312529db2" (UID: "49f78730-8272-4a3b-a5ad-9c5312529db2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:30:32.984171 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:32.984060 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f78730-8272-4a3b-a5ad-9c5312529db2-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:30:33.648844 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:33.648814 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" Jul 17 06:30:33.649044 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:33.648851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8" event={"ID":"49f78730-8272-4a3b-a5ad-9c5312529db2","Type":"ContainerDied","Data":"bead8e5027d2d15fc665e27f1a759327d1b52af31816d220eba1ff55f2a76097"} Jul 17 06:30:33.649044 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:33.648912 2580 scope.go:117] "RemoveContainer" containerID="b9690580916937766c7e1b755c54315286bc443bba74ce4bb9da4eda0a68a3f7" Jul 17 06:30:33.666886 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:33.666853 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8"] Jul 17 06:30:33.674850 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:33.674821 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qb2t8"] Jul 17 06:30:34.890200 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:34.890162 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49f78730-8272-4a3b-a5ad-9c5312529db2" path="/var/lib/kubelet/pods/49f78730-8272-4a3b-a5ad-9c5312529db2/volumes" Jul 17 06:30:39.119401 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:39.119285 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:30:39.226003 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:39.225952 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:30:40.108924 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.108884 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk"] Jul 17 06:30:40.109230 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109218 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49f78730-8272-4a3b-a5ad-9c5312529db2" containerName="revoke-keys" Jul 17 06:30:40.109274 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109232 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="49f78730-8272-4a3b-a5ad-9c5312529db2" containerName="revoke-keys" Jul 17 06:30:40.109274 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109241 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec992266-7ee4-4992-bd84-155c699120df" containerName="revoke-keys" Jul 17 06:30:40.109274 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109248 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec992266-7ee4-4992-bd84-155c699120df" containerName="revoke-keys" Jul 17 06:30:40.109274 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109255 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49f78730-8272-4a3b-a5ad-9c5312529db2" containerName="revoke-keys" Jul 17 06:30:40.109274 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109261 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="49f78730-8272-4a3b-a5ad-9c5312529db2" containerName="revoke-keys" Jul 17 06:30:40.109274 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109275 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49f78730-8272-4a3b-a5ad-9c5312529db2" containerName="revoke-keys" Jul 17 06:30:40.109475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109281 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="49f78730-8272-4a3b-a5ad-9c5312529db2" containerName="revoke-keys" Jul 17 06:30:40.109475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109292 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec992266-7ee4-4992-bd84-155c699120df" containerName="revoke-keys" Jul 17 06:30:40.109475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109297 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec992266-7ee4-4992-bd84-155c699120df" containerName="revoke-keys" Jul 17 06:30:40.109475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109307 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec992266-7ee4-4992-bd84-155c699120df" containerName="revoke-keys" Jul 17 06:30:40.109475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109312 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec992266-7ee4-4992-bd84-155c699120df" containerName="revoke-keys" Jul 17 06:30:40.109475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109359 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec992266-7ee4-4992-bd84-155c699120df" containerName="revoke-keys" Jul 17 06:30:40.109475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109365 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="49f78730-8272-4a3b-a5ad-9c5312529db2" containerName="revoke-keys" Jul 17 06:30:40.109475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109372 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="49f78730-8272-4a3b-a5ad-9c5312529db2" containerName="revoke-keys" Jul 17 06:30:40.109475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.109379 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec992266-7ee4-4992-bd84-155c699120df" containerName="revoke-keys" Jul 17 06:30:40.113844 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.113820 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" Jul 17 06:30:40.117325 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.117299 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:30:40.117496 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.117366 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 06:30:40.125040 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.125013 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk"] Jul 17 06:30:40.146263 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.146227 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50269d6d-64f8-4f77-a98c-1b7a8167cc90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk\" (UID: \"50269d6d-64f8-4f77-a98c-1b7a8167cc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" Jul 17 06:30:40.247762 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.247727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50269d6d-64f8-4f77-a98c-1b7a8167cc90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk\" (UID: \"50269d6d-64f8-4f77-a98c-1b7a8167cc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" Jul 17 06:30:40.248336 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.248316 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50269d6d-64f8-4f77-a98c-1b7a8167cc90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk\" (UID: \"50269d6d-64f8-4f77-a98c-1b7a8167cc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" Jul 17 06:30:40.425231 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.425129 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" Jul 17 06:30:40.560097 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.560062 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk"] Jul 17 06:30:40.563368 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:30:40.563340 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50269d6d_64f8_4f77_a98c_1b7a8167cc90.slice/crio-f26ffb6393ff2e2c7f513988f3440835c731b96828ffcb5d10e77e4b10562492 WatchSource:0}: Error finding container f26ffb6393ff2e2c7f513988f3440835c731b96828ffcb5d10e77e4b10562492: Status 404 returned error can't find the container with id f26ffb6393ff2e2c7f513988f3440835c731b96828ffcb5d10e77e4b10562492 Jul 17 06:30:40.682846 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:40.682748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" event={"ID":"50269d6d-64f8-4f77-a98c-1b7a8167cc90","Type":"ContainerStarted","Data":"f26ffb6393ff2e2c7f513988f3440835c731b96828ffcb5d10e77e4b10562492"} Jul 17 06:30:41.689019 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:41.688980 2580 generic.go:356] "Generic (PLEG): container finished" podID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" containerID="2a061c100c100a5689f6671f8c63010a6f94d792b1914ebca5a3fe7697a0e9b2" exitCode=6 Jul 17 06:30:41.689475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:41.689069 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" event={"ID":"50269d6d-64f8-4f77-a98c-1b7a8167cc90","Type":"ContainerDied","Data":"2a061c100c100a5689f6671f8c63010a6f94d792b1914ebca5a3fe7697a0e9b2"} Jul 17 06:30:41.689475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:41.689250 2580 scope.go:117] "RemoveContainer" containerID="2a061c100c100a5689f6671f8c63010a6f94d792b1914ebca5a3fe7697a0e9b2" Jul 17 06:30:42.695230 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:42.695181 2580 generic.go:356] "Generic (PLEG): container finished" podID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" containerID="3427043329f6131338171b63232b3fe9e20387fafc449bce5e59c7f84c085a09" exitCode=6 Jul 17 06:30:42.695732 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:42.695266 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" event={"ID":"50269d6d-64f8-4f77-a98c-1b7a8167cc90","Type":"ContainerDied","Data":"3427043329f6131338171b63232b3fe9e20387fafc449bce5e59c7f84c085a09"} Jul 17 06:30:42.695732 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:42.695372 2580 scope.go:117] "RemoveContainer" containerID="2a061c100c100a5689f6671f8c63010a6f94d792b1914ebca5a3fe7697a0e9b2" Jul 17 06:30:42.695732 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:42.695579 2580 scope.go:117] "RemoveContainer" containerID="3427043329f6131338171b63232b3fe9e20387fafc449bce5e59c7f84c085a09" Jul 17 06:30:42.695871 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:30:42.695853 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk_odh-ai-gateway-infra(50269d6d-64f8-4f77-a98c-1b7a8167cc90)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" podUID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" Jul 17 06:30:43.078398 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:43.078288 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:30:43.701453 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:43.701421 2580 scope.go:117] "RemoveContainer" containerID="3427043329f6131338171b63232b3fe9e20387fafc449bce5e59c7f84c085a09" Jul 17 06:30:43.701962 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:30:43.701694 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk_odh-ai-gateway-infra(50269d6d-64f8-4f77-a98c-1b7a8167cc90)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" podUID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" Jul 17 06:30:44.121876 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:44.121839 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w"] Jul 17 06:30:44.122247 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:44.122234 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec992266-7ee4-4992-bd84-155c699120df" containerName="revoke-keys" Jul 17 06:30:44.122305 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:44.122252 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="49f78730-8272-4a3b-a5ad-9c5312529db2" containerName="revoke-keys" Jul 17 06:30:44.126500 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:44.126477 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" Jul 17 06:30:44.133275 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:44.133231 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w"] Jul 17 06:30:44.184755 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:44.184715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3a8580-169c-4f48-9bcc-6fd459e7486a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w\" (UID: \"da3a8580-169c-4f48-9bcc-6fd459e7486a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" Jul 17 06:30:44.285485 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:44.285424 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3a8580-169c-4f48-9bcc-6fd459e7486a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w\" (UID: \"da3a8580-169c-4f48-9bcc-6fd459e7486a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" Jul 17 06:30:44.286089 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:44.286067 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3a8580-169c-4f48-9bcc-6fd459e7486a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w\" (UID: \"da3a8580-169c-4f48-9bcc-6fd459e7486a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" Jul 17 06:30:44.325747 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:44.325708 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:30:44.438850 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:44.438750 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" Jul 17 06:30:44.573294 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:44.573153 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w"] Jul 17 06:30:44.576092 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:30:44.576066 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3a8580_169c_4f48_9bcc_6fd459e7486a.slice/crio-2ecb5f337ac15c71a088f7f7c6364a8a692683fadd5c5bc066940bd9bc43f918 WatchSource:0}: Error finding container 2ecb5f337ac15c71a088f7f7c6364a8a692683fadd5c5bc066940bd9bc43f918: Status 404 returned error can't find the container with id 2ecb5f337ac15c71a088f7f7c6364a8a692683fadd5c5bc066940bd9bc43f918 Jul 17 06:30:44.707211 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:44.707116 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" event={"ID":"da3a8580-169c-4f48-9bcc-6fd459e7486a","Type":"ContainerStarted","Data":"2ecb5f337ac15c71a088f7f7c6364a8a692683fadd5c5bc066940bd9bc43f918"} Jul 17 06:30:45.713321 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:45.713286 2580 generic.go:356] "Generic (PLEG): container finished" podID="da3a8580-169c-4f48-9bcc-6fd459e7486a" containerID="564dc2aaa0db1b27ee8dea16438f456c460d7e3d1793851bc16848e9a8c2419e" exitCode=6 Jul 17 06:30:45.713321 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:45.713327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" event={"ID":"da3a8580-169c-4f48-9bcc-6fd459e7486a","Type":"ContainerDied","Data":"564dc2aaa0db1b27ee8dea16438f456c460d7e3d1793851bc16848e9a8c2419e"} Jul 17 06:30:45.713774 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:45.713570 2580 scope.go:117] "RemoveContainer" containerID="564dc2aaa0db1b27ee8dea16438f456c460d7e3d1793851bc16848e9a8c2419e" Jul 17 06:30:46.722981 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:46.722940 2580 generic.go:356] "Generic (PLEG): container finished" podID="da3a8580-169c-4f48-9bcc-6fd459e7486a" containerID="a0bac8108aac445ec2f8eb3e4c19c4f26fc5de6128a2087078d6c1adf8cdfea1" exitCode=6 Jul 17 06:30:46.723477 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:46.722999 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" event={"ID":"da3a8580-169c-4f48-9bcc-6fd459e7486a","Type":"ContainerDied","Data":"a0bac8108aac445ec2f8eb3e4c19c4f26fc5de6128a2087078d6c1adf8cdfea1"} Jul 17 06:30:46.723477 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:46.723061 2580 scope.go:117] "RemoveContainer" containerID="564dc2aaa0db1b27ee8dea16438f456c460d7e3d1793851bc16848e9a8c2419e" Jul 17 06:30:46.723477 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:46.723234 2580 scope.go:117] "RemoveContainer" containerID="a0bac8108aac445ec2f8eb3e4c19c4f26fc5de6128a2087078d6c1adf8cdfea1" Jul 17 06:30:46.723602 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:30:46.723489 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w_odh-ai-gateway-infra(da3a8580-169c-4f48-9bcc-6fd459e7486a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" podUID="da3a8580-169c-4f48-9bcc-6fd459e7486a" Jul 17 06:30:47.729166 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:47.729130 2580 scope.go:117] "RemoveContainer" containerID="a0bac8108aac445ec2f8eb3e4c19c4f26fc5de6128a2087078d6c1adf8cdfea1" Jul 17 06:30:47.729606 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:30:47.729357 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w_odh-ai-gateway-infra(da3a8580-169c-4f48-9bcc-6fd459e7486a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" podUID="da3a8580-169c-4f48-9bcc-6fd459e7486a" Jul 17 06:30:49.752358 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:49.752311 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:30:55.883896 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:55.883860 2580 scope.go:117] "RemoveContainer" containerID="3427043329f6131338171b63232b3fe9e20387fafc449bce5e59c7f84c085a09" Jul 17 06:30:56.777160 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:56.777067 2580 generic.go:356] "Generic (PLEG): container finished" podID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" containerID="c7e63968258f13b277125d50f9ecb3a4187aea94e43f248552630816b0b3a291" exitCode=6 Jul 17 06:30:56.777160 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:56.777138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" event={"ID":"50269d6d-64f8-4f77-a98c-1b7a8167cc90","Type":"ContainerDied","Data":"c7e63968258f13b277125d50f9ecb3a4187aea94e43f248552630816b0b3a291"} Jul 17 06:30:56.777344 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:56.777199 2580 scope.go:117] "RemoveContainer" containerID="3427043329f6131338171b63232b3fe9e20387fafc449bce5e59c7f84c085a09" Jul 17 06:30:56.777580 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:56.777555 2580 scope.go:117] "RemoveContainer" containerID="c7e63968258f13b277125d50f9ecb3a4187aea94e43f248552630816b0b3a291" Jul 17 06:30:56.777820 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:30:56.777799 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk_odh-ai-gateway-infra(50269d6d-64f8-4f77-a98c-1b7a8167cc90)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" podUID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" Jul 17 06:30:57.804295 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:57.804254 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk"] Jul 17 06:30:57.937795 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:57.937764 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" Jul 17 06:30:58.013239 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:58.013198 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50269d6d-64f8-4f77-a98c-1b7a8167cc90-maas-api-service-ca\") pod \"50269d6d-64f8-4f77-a98c-1b7a8167cc90\" (UID: \"50269d6d-64f8-4f77-a98c-1b7a8167cc90\") " Jul 17 06:30:58.013587 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:58.013564 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50269d6d-64f8-4f77-a98c-1b7a8167cc90-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50269d6d-64f8-4f77-a98c-1b7a8167cc90" (UID: "50269d6d-64f8-4f77-a98c-1b7a8167cc90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:30:58.114879 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:58.114841 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50269d6d-64f8-4f77-a98c-1b7a8167cc90-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:30:58.790882 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:58.790852 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" Jul 17 06:30:58.791080 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:58.790851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk" event={"ID":"50269d6d-64f8-4f77-a98c-1b7a8167cc90","Type":"ContainerDied","Data":"f26ffb6393ff2e2c7f513988f3440835c731b96828ffcb5d10e77e4b10562492"} Jul 17 06:30:58.791080 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:58.790991 2580 scope.go:117] "RemoveContainer" containerID="c7e63968258f13b277125d50f9ecb3a4187aea94e43f248552630816b0b3a291" Jul 17 06:30:58.812037 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:58.812005 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk"] Jul 17 06:30:58.818262 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:58.818234 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-tb5wk"] Jul 17 06:30:58.889291 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:30:58.889258 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" path="/var/lib/kubelet/pods/50269d6d-64f8-4f77-a98c-1b7a8167cc90/volumes" Jul 17 06:31:02.534671 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.534627 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d"] Jul 17 06:31:02.535323 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.534999 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" containerName="revoke-keys" Jul 17 06:31:02.535323 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.535011 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" containerName="revoke-keys" Jul 17 06:31:02.535323 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.535024 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" containerName="revoke-keys" Jul 17 06:31:02.535323 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.535030 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" containerName="revoke-keys" Jul 17 06:31:02.535323 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.535097 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" containerName="revoke-keys" Jul 17 06:31:02.535323 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.535107 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" containerName="revoke-keys" Jul 17 06:31:02.539506 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.539478 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" Jul 17 06:31:02.546066 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.546035 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d"] Jul 17 06:31:02.654062 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.654014 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1dd308e-b2cb-490d-8134-391caf43bf2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-nmn2d\" (UID: \"a1dd308e-b2cb-490d-8134-391caf43bf2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" Jul 17 06:31:02.755602 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.755561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1dd308e-b2cb-490d-8134-391caf43bf2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-nmn2d\" (UID: \"a1dd308e-b2cb-490d-8134-391caf43bf2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" Jul 17 06:31:02.756232 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.756211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1dd308e-b2cb-490d-8134-391caf43bf2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-nmn2d\" (UID: \"a1dd308e-b2cb-490d-8134-391caf43bf2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" Jul 17 06:31:02.851187 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.851089 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" Jul 17 06:31:02.884755 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.884719 2580 scope.go:117] "RemoveContainer" containerID="a0bac8108aac445ec2f8eb3e4c19c4f26fc5de6128a2087078d6c1adf8cdfea1" Jul 17 06:31:02.991484 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:02.991458 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d"] Jul 17 06:31:02.993541 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:31:02.993505 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1dd308e_b2cb_490d_8134_391caf43bf2e.slice/crio-3a76ae4288b92afc1aaaf17c4e6e054261d3e4871afc9350e90d507a5ad806c5 WatchSource:0}: Error finding container 3a76ae4288b92afc1aaaf17c4e6e054261d3e4871afc9350e90d507a5ad806c5: Status 404 returned error can't find the container with id 3a76ae4288b92afc1aaaf17c4e6e054261d3e4871afc9350e90d507a5ad806c5 Jul 17 06:31:03.817253 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:03.817146 2580 generic.go:356] "Generic (PLEG): container finished" podID="a1dd308e-b2cb-490d-8134-391caf43bf2e" containerID="613d47ae1b34854d9dc163bc65899a97569e89143b6372aef56845a16577bb4f" exitCode=6 Jul 17 06:31:03.817253 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:03.817235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" event={"ID":"a1dd308e-b2cb-490d-8134-391caf43bf2e","Type":"ContainerDied","Data":"613d47ae1b34854d9dc163bc65899a97569e89143b6372aef56845a16577bb4f"} Jul 17 06:31:03.817811 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:03.817303 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" event={"ID":"a1dd308e-b2cb-490d-8134-391caf43bf2e","Type":"ContainerStarted","Data":"3a76ae4288b92afc1aaaf17c4e6e054261d3e4871afc9350e90d507a5ad806c5"} Jul 17 06:31:03.817811 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:03.817473 2580 scope.go:117] "RemoveContainer" containerID="613d47ae1b34854d9dc163bc65899a97569e89143b6372aef56845a16577bb4f" Jul 17 06:31:03.819432 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:03.819402 2580 generic.go:356] "Generic (PLEG): container finished" podID="da3a8580-169c-4f48-9bcc-6fd459e7486a" containerID="99ca57e2fa7989bd8f426e41f0fb8e1fb3c279eedb47d24f17f27d66c24c8bb4" exitCode=6 Jul 17 06:31:03.819432 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:03.819414 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" event={"ID":"da3a8580-169c-4f48-9bcc-6fd459e7486a","Type":"ContainerDied","Data":"99ca57e2fa7989bd8f426e41f0fb8e1fb3c279eedb47d24f17f27d66c24c8bb4"} Jul 17 06:31:03.819722 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:03.819469 2580 scope.go:117] "RemoveContainer" containerID="a0bac8108aac445ec2f8eb3e4c19c4f26fc5de6128a2087078d6c1adf8cdfea1" Jul 17 06:31:03.819989 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:03.819837 2580 scope.go:117] "RemoveContainer" containerID="99ca57e2fa7989bd8f426e41f0fb8e1fb3c279eedb47d24f17f27d66c24c8bb4" Jul 17 06:31:03.820070 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:31:03.820051 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w_odh-ai-gateway-infra(da3a8580-169c-4f48-9bcc-6fd459e7486a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" podUID="da3a8580-169c-4f48-9bcc-6fd459e7486a" Jul 17 06:31:04.827635 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:04.827596 2580 generic.go:356] "Generic (PLEG): container finished" podID="a1dd308e-b2cb-490d-8134-391caf43bf2e" containerID="c8b9e54e9d379173fa211eb37b3d8d622a7a28a45afb96bd95dad7c0197fc9b3" exitCode=6 Jul 17 06:31:04.828123 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:04.827664 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" event={"ID":"a1dd308e-b2cb-490d-8134-391caf43bf2e","Type":"ContainerDied","Data":"c8b9e54e9d379173fa211eb37b3d8d622a7a28a45afb96bd95dad7c0197fc9b3"} Jul 17 06:31:04.828123 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:04.827733 2580 scope.go:117] "RemoveContainer" containerID="613d47ae1b34854d9dc163bc65899a97569e89143b6372aef56845a16577bb4f" Jul 17 06:31:04.828123 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:04.827948 2580 scope.go:117] "RemoveContainer" containerID="c8b9e54e9d379173fa211eb37b3d8d622a7a28a45afb96bd95dad7c0197fc9b3" Jul 17 06:31:04.828263 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:31:04.828169 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-nmn2d_odh-ai-gateway-infra(a1dd308e-b2cb-490d-8134-391caf43bf2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" podUID="a1dd308e-b2cb-490d-8134-391caf43bf2e" Jul 17 06:31:04.860698 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:04.860666 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w"] Jul 17 06:31:05.000295 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:05.000261 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" Jul 17 06:31:05.180164 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:05.180123 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3a8580-169c-4f48-9bcc-6fd459e7486a-maas-api-service-ca\") pod \"da3a8580-169c-4f48-9bcc-6fd459e7486a\" (UID: \"da3a8580-169c-4f48-9bcc-6fd459e7486a\") " Jul 17 06:31:05.180559 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:05.180536 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3a8580-169c-4f48-9bcc-6fd459e7486a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da3a8580-169c-4f48-9bcc-6fd459e7486a" (UID: "da3a8580-169c-4f48-9bcc-6fd459e7486a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:31:05.281118 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:05.281070 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3a8580-169c-4f48-9bcc-6fd459e7486a-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:31:05.833972 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:05.833943 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" Jul 17 06:31:05.834510 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:05.833995 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w" event={"ID":"da3a8580-169c-4f48-9bcc-6fd459e7486a","Type":"ContainerDied","Data":"2ecb5f337ac15c71a088f7f7c6364a8a692683fadd5c5bc066940bd9bc43f918"} Jul 17 06:31:05.834510 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:05.834067 2580 scope.go:117] "RemoveContainer" containerID="99ca57e2fa7989bd8f426e41f0fb8e1fb3c279eedb47d24f17f27d66c24c8bb4" Jul 17 06:31:05.836084 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:05.836067 2580 scope.go:117] "RemoveContainer" containerID="c8b9e54e9d379173fa211eb37b3d8d622a7a28a45afb96bd95dad7c0197fc9b3" Jul 17 06:31:05.836315 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:31:05.836295 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-nmn2d_odh-ai-gateway-infra(a1dd308e-b2cb-490d-8134-391caf43bf2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" podUID="a1dd308e-b2cb-490d-8134-391caf43bf2e" Jul 17 06:31:05.866625 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:05.866588 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w"] Jul 17 06:31:05.872428 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:05.872378 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-m8l4w"] Jul 17 06:31:06.224156 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.224121 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt"] Jul 17 06:31:06.224510 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.224496 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3a8580-169c-4f48-9bcc-6fd459e7486a" containerName="revoke-keys" Jul 17 06:31:06.224510 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.224511 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3a8580-169c-4f48-9bcc-6fd459e7486a" containerName="revoke-keys" Jul 17 06:31:06.224594 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.224519 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3a8580-169c-4f48-9bcc-6fd459e7486a" containerName="revoke-keys" Jul 17 06:31:06.224594 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.224525 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3a8580-169c-4f48-9bcc-6fd459e7486a" containerName="revoke-keys" Jul 17 06:31:06.224594 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.224537 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3a8580-169c-4f48-9bcc-6fd459e7486a" containerName="revoke-keys" Jul 17 06:31:06.224594 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.224543 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3a8580-169c-4f48-9bcc-6fd459e7486a" containerName="revoke-keys" Jul 17 06:31:06.224594 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.224551 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" containerName="revoke-keys" Jul 17 06:31:06.224594 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.224556 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" containerName="revoke-keys" Jul 17 06:31:06.224773 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.224625 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3a8580-169c-4f48-9bcc-6fd459e7486a" containerName="revoke-keys" Jul 17 06:31:06.224773 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.224633 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3a8580-169c-4f48-9bcc-6fd459e7486a" containerName="revoke-keys" Jul 17 06:31:06.224773 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.224646 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="50269d6d-64f8-4f77-a98c-1b7a8167cc90" containerName="revoke-keys" Jul 17 06:31:06.229062 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.229041 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" Jul 17 06:31:06.235441 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.235407 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt"] Jul 17 06:31:06.392413 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.392351 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019b7abc-95ef-4ddf-8357-398d6b453a04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt\" (UID: \"019b7abc-95ef-4ddf-8357-398d6b453a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" Jul 17 06:31:06.493447 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.493290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019b7abc-95ef-4ddf-8357-398d6b453a04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt\" (UID: \"019b7abc-95ef-4ddf-8357-398d6b453a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" Jul 17 06:31:06.493992 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.493970 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019b7abc-95ef-4ddf-8357-398d6b453a04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt\" (UID: \"019b7abc-95ef-4ddf-8357-398d6b453a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" Jul 17 06:31:06.549359 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.549322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" Jul 17 06:31:06.677197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.677168 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt"] Jul 17 06:31:06.679861 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:31:06.679822 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019b7abc_95ef_4ddf_8357_398d6b453a04.slice/crio-56aded9c2eae1b3a0b0fcb8c23cb7bdbe71e679aa38e8c84d4b0feeb39826ff4 WatchSource:0}: Error finding container 56aded9c2eae1b3a0b0fcb8c23cb7bdbe71e679aa38e8c84d4b0feeb39826ff4: Status 404 returned error can't find the container with id 56aded9c2eae1b3a0b0fcb8c23cb7bdbe71e679aa38e8c84d4b0feeb39826ff4 Jul 17 06:31:06.842412 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.842363 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" event={"ID":"019b7abc-95ef-4ddf-8357-398d6b453a04","Type":"ContainerStarted","Data":"56aded9c2eae1b3a0b0fcb8c23cb7bdbe71e679aa38e8c84d4b0feeb39826ff4"} Jul 17 06:31:06.889509 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:06.889471 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3a8580-169c-4f48-9bcc-6fd459e7486a" path="/var/lib/kubelet/pods/da3a8580-169c-4f48-9bcc-6fd459e7486a/volumes" Jul 17 06:31:07.849654 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:07.849615 2580 generic.go:356] "Generic (PLEG): container finished" podID="019b7abc-95ef-4ddf-8357-398d6b453a04" containerID="d025cb0563848505e6f98ddeec6bbbe88a598494b7d0f5dafa3ddf1be529bc08" exitCode=6 Jul 17 06:31:07.850087 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:07.849710 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" event={"ID":"019b7abc-95ef-4ddf-8357-398d6b453a04","Type":"ContainerDied","Data":"d025cb0563848505e6f98ddeec6bbbe88a598494b7d0f5dafa3ddf1be529bc08"} Jul 17 06:31:07.850087 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:07.849905 2580 scope.go:117] "RemoveContainer" containerID="d025cb0563848505e6f98ddeec6bbbe88a598494b7d0f5dafa3ddf1be529bc08" Jul 17 06:31:08.859119 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:08.859083 2580 generic.go:356] "Generic (PLEG): container finished" podID="019b7abc-95ef-4ddf-8357-398d6b453a04" containerID="8ee30f4a5b430bea32429caadb712a462ad59b227f2d306d1f38fe6a3cda9f80" exitCode=6 Jul 17 06:31:08.859590 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:08.859137 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" event={"ID":"019b7abc-95ef-4ddf-8357-398d6b453a04","Type":"ContainerDied","Data":"8ee30f4a5b430bea32429caadb712a462ad59b227f2d306d1f38fe6a3cda9f80"} Jul 17 06:31:08.859590 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:08.859184 2580 scope.go:117] "RemoveContainer" containerID="d025cb0563848505e6f98ddeec6bbbe88a598494b7d0f5dafa3ddf1be529bc08" Jul 17 06:31:08.859590 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:08.859367 2580 scope.go:117] "RemoveContainer" containerID="8ee30f4a5b430bea32429caadb712a462ad59b227f2d306d1f38fe6a3cda9f80" Jul 17 06:31:08.859764 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:31:08.859619 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt_odh-ai-gateway-infra(019b7abc-95ef-4ddf-8357-398d6b453a04)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" podUID="019b7abc-95ef-4ddf-8357-398d6b453a04" Jul 17 06:31:09.865478 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:09.865447 2580 scope.go:117] "RemoveContainer" containerID="8ee30f4a5b430bea32429caadb712a462ad59b227f2d306d1f38fe6a3cda9f80" Jul 17 06:31:09.865882 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:31:09.865654 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt_odh-ai-gateway-infra(019b7abc-95ef-4ddf-8357-398d6b453a04)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" podUID="019b7abc-95ef-4ddf-8357-398d6b453a04" Jul 17 06:31:13.394082 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:13.394032 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:13.495490 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:13.495422 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:13.748966 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:13.748846 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:13.958638 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:13.958599 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:15.012285 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:15.012245 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:15.108108 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:15.108066 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:15.205919 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:15.205886 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:15.931139 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:15.931073 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:16.013198 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:16.013156 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:16.117478 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:16.117442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:17.400405 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:17.400352 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:31:17.429076 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:17.429032 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:20.884934 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:20.884897 2580 scope.go:117] "RemoveContainer" containerID="c8b9e54e9d379173fa211eb37b3d8d622a7a28a45afb96bd95dad7c0197fc9b3" Jul 17 06:31:21.925546 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:21.925512 2580 generic.go:356] "Generic (PLEG): container finished" podID="a1dd308e-b2cb-490d-8134-391caf43bf2e" containerID="836029c9050f316e05c4ae712912aa4cce136d4d6dd85e97cdc01bd3c5dc3b81" exitCode=6 Jul 17 06:31:21.925966 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:21.925570 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" event={"ID":"a1dd308e-b2cb-490d-8134-391caf43bf2e","Type":"ContainerDied","Data":"836029c9050f316e05c4ae712912aa4cce136d4d6dd85e97cdc01bd3c5dc3b81"} Jul 17 06:31:21.925966 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:21.925648 2580 scope.go:117] "RemoveContainer" containerID="c8b9e54e9d379173fa211eb37b3d8d622a7a28a45afb96bd95dad7c0197fc9b3" Jul 17 06:31:21.925966 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:21.925956 2580 scope.go:117] "RemoveContainer" containerID="836029c9050f316e05c4ae712912aa4cce136d4d6dd85e97cdc01bd3c5dc3b81" Jul 17 06:31:21.926172 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:31:21.926153 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-nmn2d_odh-ai-gateway-infra(a1dd308e-b2cb-490d-8134-391caf43bf2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" podUID="a1dd308e-b2cb-490d-8134-391caf43bf2e" Jul 17 06:31:22.956199 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:22.956160 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d"] Jul 17 06:31:23.094241 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:23.094213 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" Jul 17 06:31:23.135319 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:23.130102 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1dd308e-b2cb-490d-8134-391caf43bf2e-maas-api-service-ca\") pod \"a1dd308e-b2cb-490d-8134-391caf43bf2e\" (UID: \"a1dd308e-b2cb-490d-8134-391caf43bf2e\") " Jul 17 06:31:23.135319 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:23.132335 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1dd308e-b2cb-490d-8134-391caf43bf2e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a1dd308e-b2cb-490d-8134-391caf43bf2e" (UID: "a1dd308e-b2cb-490d-8134-391caf43bf2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:31:23.231315 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:23.231210 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1dd308e-b2cb-490d-8134-391caf43bf2e-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:31:23.486166 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:23.486079 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:23.586473 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:23.586430 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:23.937483 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:23.937452 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" Jul 17 06:31:23.937483 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:23.937469 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d" event={"ID":"a1dd308e-b2cb-490d-8134-391caf43bf2e","Type":"ContainerDied","Data":"3a76ae4288b92afc1aaaf17c4e6e054261d3e4871afc9350e90d507a5ad806c5"} Jul 17 06:31:23.937742 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:23.937534 2580 scope.go:117] "RemoveContainer" containerID="836029c9050f316e05c4ae712912aa4cce136d4d6dd85e97cdc01bd3c5dc3b81" Jul 17 06:31:23.958129 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:23.958100 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d"] Jul 17 06:31:23.961682 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:23.961661 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-nmn2d"] Jul 17 06:31:24.129571 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:24.129530 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:24.884223 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:24.884185 2580 scope.go:117] "RemoveContainer" containerID="8ee30f4a5b430bea32429caadb712a462ad59b227f2d306d1f38fe6a3cda9f80" Jul 17 06:31:24.889529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:24.889503 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1dd308e-b2cb-490d-8134-391caf43bf2e" path="/var/lib/kubelet/pods/a1dd308e-b2cb-490d-8134-391caf43bf2e/volumes" Jul 17 06:31:25.950205 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:25.950166 2580 generic.go:356] "Generic (PLEG): container finished" podID="019b7abc-95ef-4ddf-8357-398d6b453a04" containerID="bf59be298480542de4e38324a9b767e0706325fa8671347ddf6db957e04d2e90" exitCode=6 Jul 17 06:31:25.950698 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:25.950284 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" event={"ID":"019b7abc-95ef-4ddf-8357-398d6b453a04","Type":"ContainerDied","Data":"bf59be298480542de4e38324a9b767e0706325fa8671347ddf6db957e04d2e90"} Jul 17 06:31:25.950698 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:25.950354 2580 scope.go:117] "RemoveContainer" containerID="8ee30f4a5b430bea32429caadb712a462ad59b227f2d306d1f38fe6a3cda9f80" Jul 17 06:31:25.951061 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:25.951021 2580 scope.go:117] "RemoveContainer" containerID="bf59be298480542de4e38324a9b767e0706325fa8671347ddf6db957e04d2e90" Jul 17 06:31:25.951363 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:31:25.951339 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt_odh-ai-gateway-infra(019b7abc-95ef-4ddf-8357-398d6b453a04)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" podUID="019b7abc-95ef-4ddf-8357-398d6b453a04" Jul 17 06:31:26.980008 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:26.979974 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt"] Jul 17 06:31:27.115253 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:27.115224 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" Jul 17 06:31:27.164171 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:27.164136 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019b7abc-95ef-4ddf-8357-398d6b453a04-maas-api-service-ca\") pod \"019b7abc-95ef-4ddf-8357-398d6b453a04\" (UID: \"019b7abc-95ef-4ddf-8357-398d6b453a04\") " Jul 17 06:31:27.164543 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:27.164520 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/019b7abc-95ef-4ddf-8357-398d6b453a04-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "019b7abc-95ef-4ddf-8357-398d6b453a04" (UID: "019b7abc-95ef-4ddf-8357-398d6b453a04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:31:27.265819 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:27.265716 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/019b7abc-95ef-4ddf-8357-398d6b453a04-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:31:27.962005 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:27.961976 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" Jul 17 06:31:27.962005 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:27.961989 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt" event={"ID":"019b7abc-95ef-4ddf-8357-398d6b453a04","Type":"ContainerDied","Data":"56aded9c2eae1b3a0b0fcb8c23cb7bdbe71e679aa38e8c84d4b0feeb39826ff4"} Jul 17 06:31:27.962289 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:27.962045 2580 scope.go:117] "RemoveContainer" containerID="bf59be298480542de4e38324a9b767e0706325fa8671347ddf6db957e04d2e90" Jul 17 06:31:27.985190 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:27.985151 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt"] Jul 17 06:31:27.991912 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:27.991882 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-vtqjt"] Jul 17 06:31:28.889990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:28.889946 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="019b7abc-95ef-4ddf-8357-398d6b453a04" path="/var/lib/kubelet/pods/019b7abc-95ef-4ddf-8357-398d6b453a04/volumes" Jul 17 06:31:29.881771 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:29.881735 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:31:30.531559 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:30.531512 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:30.636951 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:30.636914 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:30.739111 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:30.739076 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:41.227685 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:41.227650 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:31:41.596143 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:31:41.596050 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:02.573341 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:02.573295 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:10.173728 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:10.173688 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:10.283413 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:10.283357 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:10.593065 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:10.592975 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:10.927110 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:10.927059 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:11.087419 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:11.087358 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:13.586010 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.585970 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:13.684094 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.684059 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8"] Jul 17 06:32:13.684559 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.684537 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1dd308e-b2cb-490d-8134-391caf43bf2e" containerName="revoke-keys" Jul 17 06:32:13.684559 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.684559 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1dd308e-b2cb-490d-8134-391caf43bf2e" containerName="revoke-keys" Jul 17 06:32:13.684777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.684572 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1dd308e-b2cb-490d-8134-391caf43bf2e" containerName="revoke-keys" Jul 17 06:32:13.684777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.684582 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1dd308e-b2cb-490d-8134-391caf43bf2e" containerName="revoke-keys" Jul 17 06:32:13.684777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.684593 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="019b7abc-95ef-4ddf-8357-398d6b453a04" containerName="revoke-keys" Jul 17 06:32:13.684777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.684604 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="019b7abc-95ef-4ddf-8357-398d6b453a04" containerName="revoke-keys" Jul 17 06:32:13.684777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.684617 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="019b7abc-95ef-4ddf-8357-398d6b453a04" containerName="revoke-keys" Jul 17 06:32:13.684777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.684625 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="019b7abc-95ef-4ddf-8357-398d6b453a04" containerName="revoke-keys" Jul 17 06:32:13.684777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.684714 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="019b7abc-95ef-4ddf-8357-398d6b453a04" containerName="revoke-keys" Jul 17 06:32:13.684777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.684722 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3a8580-169c-4f48-9bcc-6fd459e7486a" containerName="revoke-keys" Jul 17 06:32:13.684777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.684736 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1dd308e-b2cb-490d-8134-391caf43bf2e" containerName="revoke-keys" Jul 17 06:32:13.684777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.684745 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1dd308e-b2cb-490d-8134-391caf43bf2e" containerName="revoke-keys" Jul 17 06:32:13.684777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.684753 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="019b7abc-95ef-4ddf-8357-398d6b453a04" containerName="revoke-keys" Jul 17 06:32:13.688077 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.688055 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:32:13.695695 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.695220 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpsw\"" Jul 17 06:32:13.696243 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.695992 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e77d3d-serving-cert\"" Jul 17 06:32:13.696243 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.696015 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:13.696243 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.695999 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 06:32:13.696672 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.696649 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 06:32:13.706362 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.706332 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8"] Jul 17 06:32:13.773403 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.773349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/551c0fba-fca9-46fc-9d67-647d86c07c28-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8\" (UID: \"551c0fba-fca9-46fc-9d67-647d86c07c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:32:13.773601 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.773443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45br\" (UniqueName: \"kubernetes.io/projected/551c0fba-fca9-46fc-9d67-647d86c07c28-kube-api-access-g45br\") pod \"maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8\" (UID: \"551c0fba-fca9-46fc-9d67-647d86c07c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:32:13.773601 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.773493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/551c0fba-fca9-46fc-9d67-647d86c07c28-maas-api-tls\") pod \"maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8\" (UID: \"551c0fba-fca9-46fc-9d67-647d86c07c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:32:13.807058 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.807018 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:13.874650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.874618 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/551c0fba-fca9-46fc-9d67-647d86c07c28-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8\" (UID: \"551c0fba-fca9-46fc-9d67-647d86c07c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:32:13.874846 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.874666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45br\" (UniqueName: \"kubernetes.io/projected/551c0fba-fca9-46fc-9d67-647d86c07c28-kube-api-access-g45br\") pod \"maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8\" (UID: \"551c0fba-fca9-46fc-9d67-647d86c07c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:32:13.874846 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.874699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/551c0fba-fca9-46fc-9d67-647d86c07c28-maas-api-tls\") pod \"maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8\" (UID: \"551c0fba-fca9-46fc-9d67-647d86c07c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:32:13.874846 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:32:13.874795 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-serving-cert: secret "maas-api-e2e-inf-a-e77d3d-serving-cert" not found Jul 17 06:32:13.875034 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:32:13.874858 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551c0fba-fca9-46fc-9d67-647d86c07c28-maas-api-tls podName:551c0fba-fca9-46fc-9d67-647d86c07c28 nodeName:}" failed. No retries permitted until 2026-07-17 06:32:14.374841169 +0000 UTC m=+3059.119520475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/551c0fba-fca9-46fc-9d67-647d86c07c28-maas-api-tls") pod "maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" (UID: "551c0fba-fca9-46fc-9d67-647d86c07c28") : secret "maas-api-e2e-inf-a-e77d3d-serving-cert" not found Jul 17 06:32:13.875271 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.875249 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/551c0fba-fca9-46fc-9d67-647d86c07c28-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8\" (UID: \"551c0fba-fca9-46fc-9d67-647d86c07c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:32:13.885018 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:13.884982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45br\" (UniqueName: \"kubernetes.io/projected/551c0fba-fca9-46fc-9d67-647d86c07c28-kube-api-access-g45br\") pod \"maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8\" (UID: \"551c0fba-fca9-46fc-9d67-647d86c07c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:32:14.347452 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:14.347339 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:14.378504 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:14.378463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/551c0fba-fca9-46fc-9d67-647d86c07c28-maas-api-tls\") pod \"maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8\" (UID: \"551c0fba-fca9-46fc-9d67-647d86c07c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:32:14.380927 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:14.380907 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/551c0fba-fca9-46fc-9d67-647d86c07c28-maas-api-tls\") pod \"maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8\" (UID: \"551c0fba-fca9-46fc-9d67-647d86c07c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:32:14.505946 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:14.505906 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:14.600502 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:14.600371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:32:14.622402 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:14.622348 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:14.745290 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:14.745262 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8"] Jul 17 06:32:14.748460 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:32:14.748431 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551c0fba_fca9_46fc_9d67_647d86c07c28.slice/crio-94b9c2e4a9160753b2353187a244f821075497949e150084c7ecf636b631a251 WatchSource:0}: Error finding container 94b9c2e4a9160753b2353187a244f821075497949e150084c7ecf636b631a251: Status 404 returned error can't find the container with id 94b9c2e4a9160753b2353187a244f821075497949e150084c7ecf636b631a251 Jul 17 06:32:15.191124 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:15.191088 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" event={"ID":"551c0fba-fca9-46fc-9d67-647d86c07c28","Type":"ContainerStarted","Data":"94b9c2e4a9160753b2353187a244f821075497949e150084c7ecf636b631a251"} Jul 17 06:32:16.196696 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:16.196658 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" event={"ID":"551c0fba-fca9-46fc-9d67-647d86c07c28","Type":"ContainerStarted","Data":"a91d65cd61ec036627ee7f434e1c0673df4cd958ec89739ef893ab2c152be21a"} Jul 17 06:32:16.197091 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:16.196763 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:32:16.217134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:16.217080 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" podStartSLOduration=2.5690735460000003 podStartE2EDuration="3.217062646s" totalImagesPullingTime="647.9891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:32:16.215891552 +0000 UTC m=+3060.960570885" watchObservedRunningTime="2026-07-17 06:32:16.217062646 +0000 UTC m=+3060.961741968" Jul 17 06:32:18.012945 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:18.012911 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:18.355406 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:18.355343 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:18.654668 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:18.654562 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:18.955238 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:18.955205 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:21.623996 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.623944 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:21.696601 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.696560 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55"] Jul 17 06:32:21.696938 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.696926 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1dd308e-b2cb-490d-8134-391caf43bf2e" containerName="revoke-keys" Jul 17 06:32:21.696983 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.696940 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1dd308e-b2cb-490d-8134-391caf43bf2e" containerName="revoke-keys" Jul 17 06:32:21.696983 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.696952 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="019b7abc-95ef-4ddf-8357-398d6b453a04" containerName="revoke-keys" Jul 17 06:32:21.696983 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.696958 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="019b7abc-95ef-4ddf-8357-398d6b453a04" containerName="revoke-keys" Jul 17 06:32:21.697081 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.697012 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1dd308e-b2cb-490d-8134-391caf43bf2e" containerName="revoke-keys" Jul 17 06:32:21.697081 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.697021 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="019b7abc-95ef-4ddf-8357-398d6b453a04" containerName="revoke-keys" Jul 17 06:32:21.701468 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.701442 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:32:21.703992 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.703966 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-142438-serving-cert\"" Jul 17 06:32:21.719299 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.719266 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55"] Jul 17 06:32:21.731638 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.731602 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:21.734088 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.734063 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94czh\" (UniqueName: \"kubernetes.io/projected/bb2c4647-7de5-4693-8975-1b1f671b80d5-kube-api-access-94czh\") pod \"maas-api-e2e-inf-b-142438-6798dd7fc-9vw55\" (UID: \"bb2c4647-7de5-4693-8975-1b1f671b80d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:32:21.734208 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.734118 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2c4647-7de5-4693-8975-1b1f671b80d5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-142438-6798dd7fc-9vw55\" (UID: \"bb2c4647-7de5-4693-8975-1b1f671b80d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:32:21.734208 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.734184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb2c4647-7de5-4693-8975-1b1f671b80d5-maas-api-tls\") pod \"maas-api-e2e-inf-b-142438-6798dd7fc-9vw55\" (UID: \"bb2c4647-7de5-4693-8975-1b1f671b80d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:32:21.824573 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.824533 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:21.834790 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.834757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb2c4647-7de5-4693-8975-1b1f671b80d5-maas-api-tls\") pod \"maas-api-e2e-inf-b-142438-6798dd7fc-9vw55\" (UID: \"bb2c4647-7de5-4693-8975-1b1f671b80d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:32:21.834983 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.834848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94czh\" (UniqueName: \"kubernetes.io/projected/bb2c4647-7de5-4693-8975-1b1f671b80d5-kube-api-access-94czh\") pod \"maas-api-e2e-inf-b-142438-6798dd7fc-9vw55\" (UID: \"bb2c4647-7de5-4693-8975-1b1f671b80d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:32:21.834983 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.834882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2c4647-7de5-4693-8975-1b1f671b80d5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-142438-6798dd7fc-9vw55\" (UID: \"bb2c4647-7de5-4693-8975-1b1f671b80d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:32:21.834983 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:32:21.834930 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-serving-cert: secret "maas-api-e2e-inf-b-142438-serving-cert" not found Jul 17 06:32:21.835167 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:32:21.835010 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb2c4647-7de5-4693-8975-1b1f671b80d5-maas-api-tls podName:bb2c4647-7de5-4693-8975-1b1f671b80d5 nodeName:}" failed. No retries permitted until 2026-07-17 06:32:22.334988784 +0000 UTC m=+3067.079668097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bb2c4647-7de5-4693-8975-1b1f671b80d5-maas-api-tls") pod "maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" (UID: "bb2c4647-7de5-4693-8975-1b1f671b80d5") : secret "maas-api-e2e-inf-b-142438-serving-cert" not found Jul 17 06:32:21.835564 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.835537 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2c4647-7de5-4693-8975-1b1f671b80d5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-142438-6798dd7fc-9vw55\" (UID: \"bb2c4647-7de5-4693-8975-1b1f671b80d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:32:21.847823 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:21.847784 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94czh\" (UniqueName: \"kubernetes.io/projected/bb2c4647-7de5-4693-8975-1b1f671b80d5-kube-api-access-94czh\") pod \"maas-api-e2e-inf-b-142438-6798dd7fc-9vw55\" (UID: \"bb2c4647-7de5-4693-8975-1b1f671b80d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:32:22.016469 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:22.016356 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:22.206765 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:22.206735 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:32:22.339656 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:22.339554 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb2c4647-7de5-4693-8975-1b1f671b80d5-maas-api-tls\") pod \"maas-api-e2e-inf-b-142438-6798dd7fc-9vw55\" (UID: \"bb2c4647-7de5-4693-8975-1b1f671b80d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:32:22.342069 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:22.342040 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb2c4647-7de5-4693-8975-1b1f671b80d5-maas-api-tls\") pod \"maas-api-e2e-inf-b-142438-6798dd7fc-9vw55\" (UID: \"bb2c4647-7de5-4693-8975-1b1f671b80d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:32:22.388484 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:22.388442 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:32:22.420648 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:22.420617 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:22.524530 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:22.524491 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:22.612087 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:22.612045 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:32:22.754498 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:22.754467 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55"] Jul 17 06:32:22.758224 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:32:22.758190 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2c4647_7de5_4693_8975_1b1f671b80d5.slice/crio-4d3063523c0812cf5d5842eec85701d8b0dd111f358322a89c945388ddfc84b5 WatchSource:0}: Error finding container 4d3063523c0812cf5d5842eec85701d8b0dd111f358322a89c945388ddfc84b5: Status 404 returned error can't find the container with id 4d3063523c0812cf5d5842eec85701d8b0dd111f358322a89c945388ddfc84b5 Jul 17 06:32:23.233333 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:23.233291 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" event={"ID":"bb2c4647-7de5-4693-8975-1b1f671b80d5","Type":"ContainerStarted","Data":"bcb01c44936bc617efb205e1b45108e684150d0564038e10bc257a5d3e4fb035"} Jul 17 06:32:23.233539 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:23.233345 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" event={"ID":"bb2c4647-7de5-4693-8975-1b1f671b80d5","Type":"ContainerStarted","Data":"4d3063523c0812cf5d5842eec85701d8b0dd111f358322a89c945388ddfc84b5"} Jul 17 06:32:23.233539 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:23.233411 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:32:23.259364 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:23.259224 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" podStartSLOduration=1.97563694 podStartE2EDuration="2.25919929s" totalImagesPullingTime="283.56235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:32:23.258031091 +0000 UTC m=+3068.002710419" watchObservedRunningTime="2026-07-17 06:32:23.25919929 +0000 UTC m=+3068.003878620" Jul 17 06:32:26.076481 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:26.076442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:29.243815 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:29.243785 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:32:36.602592 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:36.602557 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:32:36.692130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:36.692098 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:36.793703 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:36.793662 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:37.021812 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:37.021729 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:44.233251 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:44.233216 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9"] Jul 17 06:32:44.237033 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:44.236998 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" Jul 17 06:32:44.239473 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:44.239447 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:32:44.245771 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:44.245691 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9"] Jul 17 06:32:44.349316 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:44.349273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c79cbe1-3388-48b0-9072-08862cebe8c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-2smn9\" (UID: \"4c79cbe1-3388-48b0-9072-08862cebe8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" Jul 17 06:32:44.450107 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:44.450065 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c79cbe1-3388-48b0-9072-08862cebe8c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-2smn9\" (UID: \"4c79cbe1-3388-48b0-9072-08862cebe8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" Jul 17 06:32:44.450766 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:44.450746 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c79cbe1-3388-48b0-9072-08862cebe8c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-2smn9\" (UID: \"4c79cbe1-3388-48b0-9072-08862cebe8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" Jul 17 06:32:44.550008 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:44.549916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" Jul 17 06:32:44.667725 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:44.667577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:44.689598 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:44.689573 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9"] Jul 17 06:32:44.692419 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:32:44.692370 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c79cbe1_3388_48b0_9072_08862cebe8c4.slice/crio-8d0742f62faa8759b59c2e0a1c54d608edc09b91aee0f7e9b13f34be973a2ca8 WatchSource:0}: Error finding container 8d0742f62faa8759b59c2e0a1c54d608edc09b91aee0f7e9b13f34be973a2ca8: Status 404 returned error can't find the container with id 8d0742f62faa8759b59c2e0a1c54d608edc09b91aee0f7e9b13f34be973a2ca8 Jul 17 06:32:44.767604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:44.767565 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:32:45.337641 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:45.337537 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" event={"ID":"4c79cbe1-3388-48b0-9072-08862cebe8c4","Type":"ContainerStarted","Data":"aee0f4b93a2aacc10682fc3ad1db4efffc4ebcfcbf15f2ad5c7e5ba704d9fe99"} Jul 17 06:32:45.337641 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:45.337603 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" event={"ID":"4c79cbe1-3388-48b0-9072-08862cebe8c4","Type":"ContainerStarted","Data":"8d0742f62faa8759b59c2e0a1c54d608edc09b91aee0f7e9b13f34be973a2ca8"} Jul 17 06:32:45.356438 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:45.356365 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" podStartSLOduration=1.356348037 podStartE2EDuration="1.356348037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:32:45.354149803 +0000 UTC m=+3090.098829131" watchObservedRunningTime="2026-07-17 06:32:45.356348037 +0000 UTC m=+3090.101027365" Jul 17 06:32:46.343938 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:46.343900 2580 generic.go:356] "Generic (PLEG): container finished" podID="4c79cbe1-3388-48b0-9072-08862cebe8c4" containerID="aee0f4b93a2aacc10682fc3ad1db4efffc4ebcfcbf15f2ad5c7e5ba704d9fe99" exitCode=6 Jul 17 06:32:46.344420 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:46.343959 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" event={"ID":"4c79cbe1-3388-48b0-9072-08862cebe8c4","Type":"ContainerDied","Data":"aee0f4b93a2aacc10682fc3ad1db4efffc4ebcfcbf15f2ad5c7e5ba704d9fe99"} Jul 17 06:32:46.344420 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:46.344186 2580 scope.go:117] "RemoveContainer" containerID="aee0f4b93a2aacc10682fc3ad1db4efffc4ebcfcbf15f2ad5c7e5ba704d9fe99" Jul 17 06:32:47.350960 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:47.350863 2580 generic.go:356] "Generic (PLEG): container finished" podID="4c79cbe1-3388-48b0-9072-08862cebe8c4" containerID="21cef3c8a0d77299f735ca2afa70f959d912aa9ae1756684951dbda5de59becd" exitCode=6 Jul 17 06:32:47.351347 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:47.350954 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" event={"ID":"4c79cbe1-3388-48b0-9072-08862cebe8c4","Type":"ContainerDied","Data":"21cef3c8a0d77299f735ca2afa70f959d912aa9ae1756684951dbda5de59becd"} Jul 17 06:32:47.351347 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:47.351029 2580 scope.go:117] "RemoveContainer" containerID="aee0f4b93a2aacc10682fc3ad1db4efffc4ebcfcbf15f2ad5c7e5ba704d9fe99" Jul 17 06:32:47.351347 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:47.351249 2580 scope.go:117] "RemoveContainer" containerID="21cef3c8a0d77299f735ca2afa70f959d912aa9ae1756684951dbda5de59becd" Jul 17 06:32:47.351502 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:32:47.351481 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-2smn9_odh-ai-gateway-infra(4c79cbe1-3388-48b0-9072-08862cebe8c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" podUID="4c79cbe1-3388-48b0-9072-08862cebe8c4" Jul 17 06:32:48.357693 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:48.357663 2580 scope.go:117] "RemoveContainer" containerID="21cef3c8a0d77299f735ca2afa70f959d912aa9ae1756684951dbda5de59becd" Jul 17 06:32:48.358155 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:32:48.357952 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-2smn9_odh-ai-gateway-infra(4c79cbe1-3388-48b0-9072-08862cebe8c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" podUID="4c79cbe1-3388-48b0-9072-08862cebe8c4" Jul 17 06:32:48.993597 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:48.993566 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:32:55.058671 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:32:55.058625 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:33:01.884414 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:01.884346 2580 scope.go:117] "RemoveContainer" containerID="21cef3c8a0d77299f735ca2afa70f959d912aa9ae1756684951dbda5de59becd" Jul 17 06:33:02.422691 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:02.422656 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" event={"ID":"4c79cbe1-3388-48b0-9072-08862cebe8c4","Type":"ContainerStarted","Data":"f45423cb758ca19b171c8649f0ce66d63e2ad55b0ebe7c71daa2523b3d37dd3f"} Jul 17 06:33:02.641841 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:02.641807 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt"] Jul 17 06:33:02.645134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:02.645111 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" Jul 17 06:33:02.654134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:02.654107 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt"] Jul 17 06:33:02.818445 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:02.818322 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a875354b-233f-462c-8d11-41c15df46883-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-5hrgt\" (UID: \"a875354b-233f-462c-8d11-41c15df46883\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" Jul 17 06:33:02.919513 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:02.919468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a875354b-233f-462c-8d11-41c15df46883-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-5hrgt\" (UID: \"a875354b-233f-462c-8d11-41c15df46883\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" Jul 17 06:33:02.920156 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:02.920135 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a875354b-233f-462c-8d11-41c15df46883-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-5hrgt\" (UID: \"a875354b-233f-462c-8d11-41c15df46883\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" Jul 17 06:33:02.956498 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:02.956468 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" Jul 17 06:33:03.093326 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:03.093248 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt"] Jul 17 06:33:03.096422 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:33:03.096354 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda875354b_233f_462c_8d11_41c15df46883.slice/crio-3fd7c8fcd20a9880caeb8f907de6e740acda367d342c0e8ccdcbe473609f8762 WatchSource:0}: Error finding container 3fd7c8fcd20a9880caeb8f907de6e740acda367d342c0e8ccdcbe473609f8762: Status 404 returned error can't find the container with id 3fd7c8fcd20a9880caeb8f907de6e740acda367d342c0e8ccdcbe473609f8762 Jul 17 06:33:03.429167 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:03.429129 2580 generic.go:356] "Generic (PLEG): container finished" podID="4c79cbe1-3388-48b0-9072-08862cebe8c4" containerID="f45423cb758ca19b171c8649f0ce66d63e2ad55b0ebe7c71daa2523b3d37dd3f" exitCode=6 Jul 17 06:33:03.429355 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:03.429203 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" event={"ID":"4c79cbe1-3388-48b0-9072-08862cebe8c4","Type":"ContainerDied","Data":"f45423cb758ca19b171c8649f0ce66d63e2ad55b0ebe7c71daa2523b3d37dd3f"} Jul 17 06:33:03.429355 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:03.429266 2580 scope.go:117] "RemoveContainer" containerID="21cef3c8a0d77299f735ca2afa70f959d912aa9ae1756684951dbda5de59becd" Jul 17 06:33:03.429583 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:03.429563 2580 scope.go:117] "RemoveContainer" containerID="f45423cb758ca19b171c8649f0ce66d63e2ad55b0ebe7c71daa2523b3d37dd3f" Jul 17 06:33:03.429838 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:03.429813 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-2smn9_odh-ai-gateway-infra(4c79cbe1-3388-48b0-9072-08862cebe8c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" podUID="4c79cbe1-3388-48b0-9072-08862cebe8c4" Jul 17 06:33:03.430953 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:03.430927 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" event={"ID":"a875354b-233f-462c-8d11-41c15df46883","Type":"ContainerStarted","Data":"64158bd10dc139b37dffa72ca3e3500d016ea7f1723c22679690aa44f976f801"} Jul 17 06:33:03.431037 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:03.430982 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" event={"ID":"a875354b-233f-462c-8d11-41c15df46883","Type":"ContainerStarted","Data":"3fd7c8fcd20a9880caeb8f907de6e740acda367d342c0e8ccdcbe473609f8762"} Jul 17 06:33:03.467115 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:03.467054 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" podStartSLOduration=1.467036505 podStartE2EDuration="1.467036505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:33:03.466842052 +0000 UTC m=+3108.211521398" watchObservedRunningTime="2026-07-17 06:33:03.467036505 +0000 UTC m=+3108.211715866" Jul 17 06:33:03.477961 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:03.477917 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9"] Jul 17 06:33:04.438655 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:04.438621 2580 generic.go:356] "Generic (PLEG): container finished" podID="a875354b-233f-462c-8d11-41c15df46883" containerID="64158bd10dc139b37dffa72ca3e3500d016ea7f1723c22679690aa44f976f801" exitCode=6 Jul 17 06:33:04.438937 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:04.438697 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" event={"ID":"a875354b-233f-462c-8d11-41c15df46883","Type":"ContainerDied","Data":"64158bd10dc139b37dffa72ca3e3500d016ea7f1723c22679690aa44f976f801"} Jul 17 06:33:04.438937 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:04.438848 2580 scope.go:117] "RemoveContainer" containerID="64158bd10dc139b37dffa72ca3e3500d016ea7f1723c22679690aa44f976f801" Jul 17 06:33:04.574422 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:04.574378 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" Jul 17 06:33:04.634204 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:04.634166 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c79cbe1-3388-48b0-9072-08862cebe8c4-maas-api-service-ca\") pod \"4c79cbe1-3388-48b0-9072-08862cebe8c4\" (UID: \"4c79cbe1-3388-48b0-9072-08862cebe8c4\") " Jul 17 06:33:04.634647 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:04.634619 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c79cbe1-3388-48b0-9072-08862cebe8c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c79cbe1-3388-48b0-9072-08862cebe8c4" (UID: "4c79cbe1-3388-48b0-9072-08862cebe8c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:33:04.735234 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:04.735134 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c79cbe1-3388-48b0-9072-08862cebe8c4-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:33:05.448285 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:05.448249 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" Jul 17 06:33:05.448769 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:05.448274 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9" event={"ID":"4c79cbe1-3388-48b0-9072-08862cebe8c4","Type":"ContainerDied","Data":"8d0742f62faa8759b59c2e0a1c54d608edc09b91aee0f7e9b13f34be973a2ca8"} Jul 17 06:33:05.448769 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:05.448358 2580 scope.go:117] "RemoveContainer" containerID="f45423cb758ca19b171c8649f0ce66d63e2ad55b0ebe7c71daa2523b3d37dd3f" Jul 17 06:33:05.450315 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:05.450288 2580 generic.go:356] "Generic (PLEG): container finished" podID="a875354b-233f-462c-8d11-41c15df46883" containerID="5037cee48e561a2f6174f1dc920b2deece570d18c6afdbb93e9dba6f2e871552" exitCode=6 Jul 17 06:33:05.450572 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:05.450424 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" event={"ID":"a875354b-233f-462c-8d11-41c15df46883","Type":"ContainerDied","Data":"5037cee48e561a2f6174f1dc920b2deece570d18c6afdbb93e9dba6f2e871552"} Jul 17 06:33:05.450657 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:05.450605 2580 scope.go:117] "RemoveContainer" containerID="5037cee48e561a2f6174f1dc920b2deece570d18c6afdbb93e9dba6f2e871552" Jul 17 06:33:05.450881 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:05.450859 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-5hrgt_odh-ai-gateway-infra(a875354b-233f-462c-8d11-41c15df46883)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" podUID="a875354b-233f-462c-8d11-41c15df46883" Jul 17 06:33:05.457022 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:05.457006 2580 scope.go:117] "RemoveContainer" containerID="64158bd10dc139b37dffa72ca3e3500d016ea7f1723c22679690aa44f976f801" Jul 17 06:33:05.464905 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:05.464880 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9"] Jul 17 06:33:05.470981 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:05.470960 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-2smn9"] Jul 17 06:33:06.337659 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.337620 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm"] Jul 17 06:33:06.337988 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.337974 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c79cbe1-3388-48b0-9072-08862cebe8c4" containerName="revoke-keys" Jul 17 06:33:06.338035 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.337989 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c79cbe1-3388-48b0-9072-08862cebe8c4" containerName="revoke-keys" Jul 17 06:33:06.338035 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.338000 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c79cbe1-3388-48b0-9072-08862cebe8c4" containerName="revoke-keys" Jul 17 06:33:06.338035 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.338006 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c79cbe1-3388-48b0-9072-08862cebe8c4" containerName="revoke-keys" Jul 17 06:33:06.338127 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.338076 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c79cbe1-3388-48b0-9072-08862cebe8c4" containerName="revoke-keys" Jul 17 06:33:06.338127 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.338085 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c79cbe1-3388-48b0-9072-08862cebe8c4" containerName="revoke-keys" Jul 17 06:33:06.338127 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.338092 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c79cbe1-3388-48b0-9072-08862cebe8c4" containerName="revoke-keys" Jul 17 06:33:06.342144 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.342124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" Jul 17 06:33:06.346444 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.346397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dc5500-7bc0-4067-a129-33c98463efac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm\" (UID: \"b7dc5500-7bc0-4067-a129-33c98463efac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" Jul 17 06:33:06.352177 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.352147 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm"] Jul 17 06:33:06.447090 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.447048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dc5500-7bc0-4067-a129-33c98463efac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm\" (UID: \"b7dc5500-7bc0-4067-a129-33c98463efac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" Jul 17 06:33:06.447719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.447699 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dc5500-7bc0-4067-a129-33c98463efac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm\" (UID: \"b7dc5500-7bc0-4067-a129-33c98463efac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" Jul 17 06:33:06.457766 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.457742 2580 scope.go:117] "RemoveContainer" containerID="5037cee48e561a2f6174f1dc920b2deece570d18c6afdbb93e9dba6f2e871552" Jul 17 06:33:06.458130 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:06.458001 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-5hrgt_odh-ai-gateway-infra(a875354b-233f-462c-8d11-41c15df46883)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" podUID="a875354b-233f-462c-8d11-41c15df46883" Jul 17 06:33:06.653429 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.653364 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" Jul 17 06:33:06.792951 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.792921 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm"] Jul 17 06:33:06.795218 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:33:06.795190 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7dc5500_7bc0_4067_a129_33c98463efac.slice/crio-66e1b2cebf2411e7b8aa1252818288cedfeb3c4cd9571fabedae8125a478e71b WatchSource:0}: Error finding container 66e1b2cebf2411e7b8aa1252818288cedfeb3c4cd9571fabedae8125a478e71b: Status 404 returned error can't find the container with id 66e1b2cebf2411e7b8aa1252818288cedfeb3c4cd9571fabedae8125a478e71b Jul 17 06:33:06.890040 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:06.889996 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c79cbe1-3388-48b0-9072-08862cebe8c4" path="/var/lib/kubelet/pods/4c79cbe1-3388-48b0-9072-08862cebe8c4/volumes" Jul 17 06:33:07.463757 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:07.463712 2580 generic.go:356] "Generic (PLEG): container finished" podID="b7dc5500-7bc0-4067-a129-33c98463efac" containerID="ac78517459d3aa25d63a4cfeb9accd4d14d276f054fd0a86cda115b246e5d494" exitCode=6 Jul 17 06:33:07.464248 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:07.463795 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" event={"ID":"b7dc5500-7bc0-4067-a129-33c98463efac","Type":"ContainerDied","Data":"ac78517459d3aa25d63a4cfeb9accd4d14d276f054fd0a86cda115b246e5d494"} Jul 17 06:33:07.464248 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:07.463849 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" event={"ID":"b7dc5500-7bc0-4067-a129-33c98463efac","Type":"ContainerStarted","Data":"66e1b2cebf2411e7b8aa1252818288cedfeb3c4cd9571fabedae8125a478e71b"} Jul 17 06:33:07.464248 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:07.464016 2580 scope.go:117] "RemoveContainer" containerID="ac78517459d3aa25d63a4cfeb9accd4d14d276f054fd0a86cda115b246e5d494" Jul 17 06:33:08.213791 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:08.213754 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:33:08.470775 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:08.470687 2580 generic.go:356] "Generic (PLEG): container finished" podID="b7dc5500-7bc0-4067-a129-33c98463efac" containerID="5b2e6f367c3da2603f760d37bef539625177c7d49366a0d32228fdc828267bfa" exitCode=6 Jul 17 06:33:08.471186 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:08.470776 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" event={"ID":"b7dc5500-7bc0-4067-a129-33c98463efac","Type":"ContainerDied","Data":"5b2e6f367c3da2603f760d37bef539625177c7d49366a0d32228fdc828267bfa"} Jul 17 06:33:08.471186 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:08.470869 2580 scope.go:117] "RemoveContainer" containerID="ac78517459d3aa25d63a4cfeb9accd4d14d276f054fd0a86cda115b246e5d494" Jul 17 06:33:08.471186 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:08.471036 2580 scope.go:117] "RemoveContainer" containerID="5b2e6f367c3da2603f760d37bef539625177c7d49366a0d32228fdc828267bfa" Jul 17 06:33:08.471322 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:08.471274 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm_odh-ai-gateway-infra(b7dc5500-7bc0-4067-a129-33c98463efac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" podUID="b7dc5500-7bc0-4067-a129-33c98463efac" Jul 17 06:33:09.477488 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:09.477462 2580 scope.go:117] "RemoveContainer" containerID="5b2e6f367c3da2603f760d37bef539625177c7d49366a0d32228fdc828267bfa" Jul 17 06:33:09.477873 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:09.477648 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm_odh-ai-gateway-infra(b7dc5500-7bc0-4067-a129-33c98463efac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" podUID="b7dc5500-7bc0-4067-a129-33c98463efac" Jul 17 06:33:10.273932 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:10.273900 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw"] Jul 17 06:33:10.274291 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:10.274278 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c79cbe1-3388-48b0-9072-08862cebe8c4" containerName="revoke-keys" Jul 17 06:33:10.274337 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:10.274294 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c79cbe1-3388-48b0-9072-08862cebe8c4" containerName="revoke-keys" Jul 17 06:33:10.281332 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:10.281154 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" Jul 17 06:33:10.292964 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:10.292934 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw"] Jul 17 06:33:10.384129 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:10.384084 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c63ae2a-46dd-4714-af98-b6d040850623-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw\" (UID: \"1c63ae2a-46dd-4714-af98-b6d040850623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" Jul 17 06:33:10.484741 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:10.484702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c63ae2a-46dd-4714-af98-b6d040850623-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw\" (UID: \"1c63ae2a-46dd-4714-af98-b6d040850623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" Jul 17 06:33:10.485343 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:10.485323 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c63ae2a-46dd-4714-af98-b6d040850623-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw\" (UID: \"1c63ae2a-46dd-4714-af98-b6d040850623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" Jul 17 06:33:10.592102 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:10.592001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" Jul 17 06:33:10.726556 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:10.726522 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw"] Jul 17 06:33:10.729620 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:33:10.729588 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c63ae2a_46dd_4714_af98_b6d040850623.slice/crio-7d4cf6757f064df2bc9c16104b3afc38da6f71f92ff87958ec1210762e05f89f WatchSource:0}: Error finding container 7d4cf6757f064df2bc9c16104b3afc38da6f71f92ff87958ec1210762e05f89f: Status 404 returned error can't find the container with id 7d4cf6757f064df2bc9c16104b3afc38da6f71f92ff87958ec1210762e05f89f Jul 17 06:33:11.488272 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:11.488235 2580 generic.go:356] "Generic (PLEG): container finished" podID="1c63ae2a-46dd-4714-af98-b6d040850623" containerID="af179ef1d165fcc675412ce141e4d80986ac2049e8a05c1228484dcd031505b6" exitCode=6 Jul 17 06:33:11.488272 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:11.488276 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" event={"ID":"1c63ae2a-46dd-4714-af98-b6d040850623","Type":"ContainerDied","Data":"af179ef1d165fcc675412ce141e4d80986ac2049e8a05c1228484dcd031505b6"} Jul 17 06:33:11.488729 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:11.488314 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" event={"ID":"1c63ae2a-46dd-4714-af98-b6d040850623","Type":"ContainerStarted","Data":"7d4cf6757f064df2bc9c16104b3afc38da6f71f92ff87958ec1210762e05f89f"} Jul 17 06:33:11.488729 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:11.488495 2580 scope.go:117] "RemoveContainer" containerID="af179ef1d165fcc675412ce141e4d80986ac2049e8a05c1228484dcd031505b6" Jul 17 06:33:12.494634 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:12.494595 2580 generic.go:356] "Generic (PLEG): container finished" podID="1c63ae2a-46dd-4714-af98-b6d040850623" containerID="d4ffe9cad7cf45b25cb86955a41cd58fd75c5c0e5f25744fae040b098705c801" exitCode=6 Jul 17 06:33:12.495091 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:12.494657 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" event={"ID":"1c63ae2a-46dd-4714-af98-b6d040850623","Type":"ContainerDied","Data":"d4ffe9cad7cf45b25cb86955a41cd58fd75c5c0e5f25744fae040b098705c801"} Jul 17 06:33:12.495091 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:12.494706 2580 scope.go:117] "RemoveContainer" containerID="af179ef1d165fcc675412ce141e4d80986ac2049e8a05c1228484dcd031505b6" Jul 17 06:33:12.495091 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:12.494900 2580 scope.go:117] "RemoveContainer" containerID="d4ffe9cad7cf45b25cb86955a41cd58fd75c5c0e5f25744fae040b098705c801" Jul 17 06:33:12.495236 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:12.495148 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw_odh-ai-gateway-infra(1c63ae2a-46dd-4714-af98-b6d040850623)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" podUID="1c63ae2a-46dd-4714-af98-b6d040850623" Jul 17 06:33:13.500686 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:13.500661 2580 scope.go:117] "RemoveContainer" containerID="d4ffe9cad7cf45b25cb86955a41cd58fd75c5c0e5f25744fae040b098705c801" Jul 17 06:33:13.501094 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:13.500864 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw_odh-ai-gateway-infra(1c63ae2a-46dd-4714-af98-b6d040850623)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" podUID="1c63ae2a-46dd-4714-af98-b6d040850623" Jul 17 06:33:15.281676 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:15.281634 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:33:17.478173 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:17.478134 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:33:17.884438 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:17.884371 2580 scope.go:117] "RemoveContainer" containerID="5037cee48e561a2f6174f1dc920b2deece570d18c6afdbb93e9dba6f2e871552" Jul 17 06:33:18.527591 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:18.527495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" event={"ID":"a875354b-233f-462c-8d11-41c15df46883","Type":"ContainerStarted","Data":"0cd4739a793cf1a12707a4c076dc1b9e18a869979afeaec9623c74ddd480d311"} Jul 17 06:33:19.194907 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:19.194816 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:33:19.534162 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:19.534077 2580 generic.go:356] "Generic (PLEG): container finished" podID="a875354b-233f-462c-8d11-41c15df46883" containerID="0cd4739a793cf1a12707a4c076dc1b9e18a869979afeaec9623c74ddd480d311" exitCode=6 Jul 17 06:33:19.534162 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:19.534153 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" event={"ID":"a875354b-233f-462c-8d11-41c15df46883","Type":"ContainerDied","Data":"0cd4739a793cf1a12707a4c076dc1b9e18a869979afeaec9623c74ddd480d311"} Jul 17 06:33:19.534653 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:19.534219 2580 scope.go:117] "RemoveContainer" containerID="5037cee48e561a2f6174f1dc920b2deece570d18c6afdbb93e9dba6f2e871552" Jul 17 06:33:19.534653 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:19.534549 2580 scope.go:117] "RemoveContainer" containerID="0cd4739a793cf1a12707a4c076dc1b9e18a869979afeaec9623c74ddd480d311" Jul 17 06:33:19.534799 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:19.534773 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-5hrgt_odh-ai-gateway-infra(a875354b-233f-462c-8d11-41c15df46883)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" podUID="a875354b-233f-462c-8d11-41c15df46883" Jul 17 06:33:19.558299 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:19.558266 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt"] Jul 17 06:33:20.668377 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:20.668344 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" Jul 17 06:33:20.778889 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:20.778851 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a875354b-233f-462c-8d11-41c15df46883-maas-api-service-ca\") pod \"a875354b-233f-462c-8d11-41c15df46883\" (UID: \"a875354b-233f-462c-8d11-41c15df46883\") " Jul 17 06:33:20.779242 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:20.779217 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a875354b-233f-462c-8d11-41c15df46883-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a875354b-233f-462c-8d11-41c15df46883" (UID: "a875354b-233f-462c-8d11-41c15df46883"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:33:20.880094 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:20.880054 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a875354b-233f-462c-8d11-41c15df46883-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:33:20.884313 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:20.884292 2580 scope.go:117] "RemoveContainer" containerID="5b2e6f367c3da2603f760d37bef539625177c7d49366a0d32228fdc828267bfa" Jul 17 06:33:21.548257 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:21.548225 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" Jul 17 06:33:21.548475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:21.548219 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt" event={"ID":"a875354b-233f-462c-8d11-41c15df46883","Type":"ContainerDied","Data":"3fd7c8fcd20a9880caeb8f907de6e740acda367d342c0e8ccdcbe473609f8762"} Jul 17 06:33:21.548475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:21.548378 2580 scope.go:117] "RemoveContainer" containerID="0cd4739a793cf1a12707a4c076dc1b9e18a869979afeaec9623c74ddd480d311" Jul 17 06:33:21.550286 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:21.550255 2580 generic.go:356] "Generic (PLEG): container finished" podID="b7dc5500-7bc0-4067-a129-33c98463efac" containerID="cbccead78db6aae6946bb79e6d663eb0d3e601d0bf2196173cb8c305fa689931" exitCode=6 Jul 17 06:33:21.550447 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:21.550286 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" event={"ID":"b7dc5500-7bc0-4067-a129-33c98463efac","Type":"ContainerDied","Data":"cbccead78db6aae6946bb79e6d663eb0d3e601d0bf2196173cb8c305fa689931"} Jul 17 06:33:21.550707 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:21.550677 2580 scope.go:117] "RemoveContainer" containerID="cbccead78db6aae6946bb79e6d663eb0d3e601d0bf2196173cb8c305fa689931" Jul 17 06:33:21.550960 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:21.550929 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm_odh-ai-gateway-infra(b7dc5500-7bc0-4067-a129-33c98463efac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" podUID="b7dc5500-7bc0-4067-a129-33c98463efac" Jul 17 06:33:21.557121 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:21.557098 2580 scope.go:117] "RemoveContainer" containerID="5b2e6f367c3da2603f760d37bef539625177c7d49366a0d32228fdc828267bfa" Jul 17 06:33:21.584437 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:21.584403 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt"] Jul 17 06:33:21.589490 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:21.589461 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-5hrgt"] Jul 17 06:33:22.581731 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:22.581696 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm"] Jul 17 06:33:22.721412 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:22.721370 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" Jul 17 06:33:22.797318 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:22.797280 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dc5500-7bc0-4067-a129-33c98463efac-maas-api-service-ca\") pod \"b7dc5500-7bc0-4067-a129-33c98463efac\" (UID: \"b7dc5500-7bc0-4067-a129-33c98463efac\") " Jul 17 06:33:22.797698 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:22.797672 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7dc5500-7bc0-4067-a129-33c98463efac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b7dc5500-7bc0-4067-a129-33c98463efac" (UID: "b7dc5500-7bc0-4067-a129-33c98463efac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:33:22.890106 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:22.890069 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a875354b-233f-462c-8d11-41c15df46883" path="/var/lib/kubelet/pods/a875354b-233f-462c-8d11-41c15df46883/volumes" Jul 17 06:33:22.898033 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:22.898003 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7dc5500-7bc0-4067-a129-33c98463efac-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:33:23.566499 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:23.566466 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" Jul 17 06:33:23.566499 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:23.566488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm" event={"ID":"b7dc5500-7bc0-4067-a129-33c98463efac","Type":"ContainerDied","Data":"66e1b2cebf2411e7b8aa1252818288cedfeb3c4cd9571fabedae8125a478e71b"} Jul 17 06:33:23.566765 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:23.566555 2580 scope.go:117] "RemoveContainer" containerID="cbccead78db6aae6946bb79e6d663eb0d3e601d0bf2196173cb8c305fa689931" Jul 17 06:33:23.583018 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:23.582986 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm"] Jul 17 06:33:23.586709 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:23.586681 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-5c6cm"] Jul 17 06:33:24.889078 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:24.889045 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7dc5500-7bc0-4067-a129-33c98463efac" path="/var/lib/kubelet/pods/b7dc5500-7bc0-4067-a129-33c98463efac/volumes" Jul 17 06:33:26.886367 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:26.886331 2580 scope.go:117] "RemoveContainer" containerID="d4ffe9cad7cf45b25cb86955a41cd58fd75c5c0e5f25744fae040b098705c801" Jul 17 06:33:27.588158 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:27.588118 2580 generic.go:356] "Generic (PLEG): container finished" podID="1c63ae2a-46dd-4714-af98-b6d040850623" containerID="dcf6a29dce24b8b3d82270aaeb3588f6e95e2ac23cf5e55cc1d5caa9f87306d1" exitCode=6 Jul 17 06:33:27.588342 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:27.588168 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" event={"ID":"1c63ae2a-46dd-4714-af98-b6d040850623","Type":"ContainerDied","Data":"dcf6a29dce24b8b3d82270aaeb3588f6e95e2ac23cf5e55cc1d5caa9f87306d1"} Jul 17 06:33:27.588342 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:27.588218 2580 scope.go:117] "RemoveContainer" containerID="d4ffe9cad7cf45b25cb86955a41cd58fd75c5c0e5f25744fae040b098705c801" Jul 17 06:33:27.588588 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:27.588568 2580 scope.go:117] "RemoveContainer" containerID="dcf6a29dce24b8b3d82270aaeb3588f6e95e2ac23cf5e55cc1d5caa9f87306d1" Jul 17 06:33:27.588827 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:27.588804 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw_odh-ai-gateway-infra(1c63ae2a-46dd-4714-af98-b6d040850623)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" podUID="1c63ae2a-46dd-4714-af98-b6d040850623" Jul 17 06:33:28.628471 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:28.628435 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw"] Jul 17 06:33:28.766678 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:28.766646 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" Jul 17 06:33:28.852276 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:28.852233 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c63ae2a-46dd-4714-af98-b6d040850623-maas-api-service-ca\") pod \"1c63ae2a-46dd-4714-af98-b6d040850623\" (UID: \"1c63ae2a-46dd-4714-af98-b6d040850623\") " Jul 17 06:33:28.852598 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:28.852575 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c63ae2a-46dd-4714-af98-b6d040850623-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c63ae2a-46dd-4714-af98-b6d040850623" (UID: "1c63ae2a-46dd-4714-af98-b6d040850623"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:33:28.953615 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:28.953566 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c63ae2a-46dd-4714-af98-b6d040850623-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:33:29.601777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:29.601744 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" event={"ID":"1c63ae2a-46dd-4714-af98-b6d040850623","Type":"ContainerDied","Data":"7d4cf6757f064df2bc9c16104b3afc38da6f71f92ff87958ec1210762e05f89f"} Jul 17 06:33:29.601777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:29.601775 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw" Jul 17 06:33:29.602020 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:29.601804 2580 scope.go:117] "RemoveContainer" containerID="dcf6a29dce24b8b3d82270aaeb3588f6e95e2ac23cf5e55cc1d5caa9f87306d1" Jul 17 06:33:29.619994 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:29.619963 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw"] Jul 17 06:33:29.624307 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:29.624276 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mh9gw"] Jul 17 06:33:30.889881 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:30.889848 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c63ae2a-46dd-4714-af98-b6d040850623" path="/var/lib/kubelet/pods/1c63ae2a-46dd-4714-af98-b6d040850623/volumes" Jul 17 06:33:33.694300 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:33.694266 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:33:44.288262 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288166 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh"] Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288530 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c63ae2a-46dd-4714-af98-b6d040850623" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288542 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c63ae2a-46dd-4714-af98-b6d040850623" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288560 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7dc5500-7bc0-4067-a129-33c98463efac" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288566 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7dc5500-7bc0-4067-a129-33c98463efac" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288575 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c63ae2a-46dd-4714-af98-b6d040850623" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288581 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c63ae2a-46dd-4714-af98-b6d040850623" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288590 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a875354b-233f-462c-8d11-41c15df46883" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288596 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a875354b-233f-462c-8d11-41c15df46883" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288604 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7dc5500-7bc0-4067-a129-33c98463efac" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288610 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7dc5500-7bc0-4067-a129-33c98463efac" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288619 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a875354b-233f-462c-8d11-41c15df46883" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288624 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a875354b-233f-462c-8d11-41c15df46883" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288669 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a875354b-233f-462c-8d11-41c15df46883" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288676 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c63ae2a-46dd-4714-af98-b6d040850623" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288682 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a875354b-233f-462c-8d11-41c15df46883" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288689 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7dc5500-7bc0-4067-a129-33c98463efac" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288694 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7dc5500-7bc0-4067-a129-33c98463efac" containerName="revoke-keys" Jul 17 06:33:44.288789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.288701 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c63ae2a-46dd-4714-af98-b6d040850623" containerName="revoke-keys" Jul 17 06:33:44.293131 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.293108 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" Jul 17 06:33:44.297466 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.297440 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:33:44.303202 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.303171 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh"] Jul 17 06:33:44.410535 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.410501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7dc2aa7-31be-4672-956f-2760b404747f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh\" (UID: \"a7dc2aa7-31be-4672-956f-2760b404747f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" Jul 17 06:33:44.511879 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.511838 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7dc2aa7-31be-4672-956f-2760b404747f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh\" (UID: \"a7dc2aa7-31be-4672-956f-2760b404747f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" Jul 17 06:33:44.512555 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.512535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7dc2aa7-31be-4672-956f-2760b404747f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh\" (UID: \"a7dc2aa7-31be-4672-956f-2760b404747f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" Jul 17 06:33:44.604258 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.604223 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" Jul 17 06:33:44.755116 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:44.755079 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh"] Jul 17 06:33:44.758956 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:33:44.758917 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7dc2aa7_31be_4672_956f_2760b404747f.slice/crio-b7c015d78f657dc209275ccdcb41087f513f4a0f588cffe8da6c7b2644bd4575 WatchSource:0}: Error finding container b7c015d78f657dc209275ccdcb41087f513f4a0f588cffe8da6c7b2644bd4575: Status 404 returned error can't find the container with id b7c015d78f657dc209275ccdcb41087f513f4a0f588cffe8da6c7b2644bd4575 Jul 17 06:33:45.188166 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:45.188070 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:33:45.679799 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:45.679760 2580 generic.go:356] "Generic (PLEG): container finished" podID="a7dc2aa7-31be-4672-956f-2760b404747f" containerID="3a216dd096ec06064f77f0b6c3b67e952313d84a57981b970265b413e2b493e9" exitCode=6 Jul 17 06:33:45.680217 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:45.679825 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" event={"ID":"a7dc2aa7-31be-4672-956f-2760b404747f","Type":"ContainerDied","Data":"3a216dd096ec06064f77f0b6c3b67e952313d84a57981b970265b413e2b493e9"} Jul 17 06:33:45.680217 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:45.679869 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" event={"ID":"a7dc2aa7-31be-4672-956f-2760b404747f","Type":"ContainerStarted","Data":"b7c015d78f657dc209275ccdcb41087f513f4a0f588cffe8da6c7b2644bd4575"} Jul 17 06:33:45.680217 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:45.680048 2580 scope.go:117] "RemoveContainer" containerID="3a216dd096ec06064f77f0b6c3b67e952313d84a57981b970265b413e2b493e9" Jul 17 06:33:46.076580 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.076474 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:33:46.207130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.207085 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99"] Jul 17 06:33:46.207550 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.207530 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a875354b-233f-462c-8d11-41c15df46883" containerName="revoke-keys" Jul 17 06:33:46.207550 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.207552 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a875354b-233f-462c-8d11-41c15df46883" containerName="revoke-keys" Jul 17 06:33:46.207719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.207583 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7dc5500-7bc0-4067-a129-33c98463efac" containerName="revoke-keys" Jul 17 06:33:46.207719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.207592 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7dc5500-7bc0-4067-a129-33c98463efac" containerName="revoke-keys" Jul 17 06:33:46.207719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.207608 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c63ae2a-46dd-4714-af98-b6d040850623" containerName="revoke-keys" Jul 17 06:33:46.207719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.207617 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c63ae2a-46dd-4714-af98-b6d040850623" containerName="revoke-keys" Jul 17 06:33:46.207719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.207707 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c63ae2a-46dd-4714-af98-b6d040850623" containerName="revoke-keys" Jul 17 06:33:46.207977 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.207721 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a875354b-233f-462c-8d11-41c15df46883" containerName="revoke-keys" Jul 17 06:33:46.207977 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.207731 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7dc5500-7bc0-4067-a129-33c98463efac" containerName="revoke-keys" Jul 17 06:33:46.210705 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.210683 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99" Jul 17 06:33:46.222592 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.222553 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99"] Jul 17 06:33:46.330727 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.330633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1784b9e9-4a76-4357-917a-c214afd77cf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99\" (UID: \"1784b9e9-4a76-4357-917a-c214afd77cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99" Jul 17 06:33:46.432002 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.431960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1784b9e9-4a76-4357-917a-c214afd77cf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99\" (UID: \"1784b9e9-4a76-4357-917a-c214afd77cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99" Jul 17 06:33:46.432644 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.432625 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1784b9e9-4a76-4357-917a-c214afd77cf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99\" (UID: \"1784b9e9-4a76-4357-917a-c214afd77cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99" Jul 17 06:33:46.521317 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.521275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99" Jul 17 06:33:46.664314 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.664280 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99"] Jul 17 06:33:46.666956 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:33:46.666918 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1784b9e9_4a76_4357_917a_c214afd77cf0.slice/crio-8aa528011ef039e7b541e4fff8d215aff268a9b8320f121bcfec6ce00468cb65 WatchSource:0}: Error finding container 8aa528011ef039e7b541e4fff8d215aff268a9b8320f121bcfec6ce00468cb65: Status 404 returned error can't find the container with id 8aa528011ef039e7b541e4fff8d215aff268a9b8320f121bcfec6ce00468cb65 Jul 17 06:33:46.687240 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.687213 2580 generic.go:356] "Generic (PLEG): container finished" podID="a7dc2aa7-31be-4672-956f-2760b404747f" containerID="b3225a3b7b2a2f10b69740c152338e499945940531b6599bf879d908cee8e105" exitCode=6 Jul 17 06:33:46.687656 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.687296 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" event={"ID":"a7dc2aa7-31be-4672-956f-2760b404747f","Type":"ContainerDied","Data":"b3225a3b7b2a2f10b69740c152338e499945940531b6599bf879d908cee8e105"} Jul 17 06:33:46.687656 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.687368 2580 scope.go:117] "RemoveContainer" containerID="3a216dd096ec06064f77f0b6c3b67e952313d84a57981b970265b413e2b493e9" Jul 17 06:33:46.687656 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.687638 2580 scope.go:117] "RemoveContainer" containerID="b3225a3b7b2a2f10b69740c152338e499945940531b6599bf879d908cee8e105" Jul 17 06:33:46.687915 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:46.687886 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh_odh-ai-gateway-infra(a7dc2aa7-31be-4672-956f-2760b404747f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" podUID="a7dc2aa7-31be-4672-956f-2760b404747f" Jul 17 06:33:46.688902 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.688872 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99" event={"ID":"1784b9e9-4a76-4357-917a-c214afd77cf0","Type":"ContainerStarted","Data":"8aa528011ef039e7b541e4fff8d215aff268a9b8320f121bcfec6ce00468cb65"} Jul 17 06:33:46.870745 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.870628 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:33:46.967606 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:46.967568 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:33:47.695454 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:47.695421 2580 generic.go:356] "Generic (PLEG): container finished" podID="1784b9e9-4a76-4357-917a-c214afd77cf0" containerID="0dadeebc09fb20f5ad7b46a4f54a6c0219516865e1035ed2a551d0edfa3a5161" exitCode=0 Jul 17 06:33:47.695941 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:47.695517 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99" event={"ID":"1784b9e9-4a76-4357-917a-c214afd77cf0","Type":"ContainerDied","Data":"0dadeebc09fb20f5ad7b46a4f54a6c0219516865e1035ed2a551d0edfa3a5161"} Jul 17 06:33:47.697499 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:47.697480 2580 scope.go:117] "RemoveContainer" containerID="b3225a3b7b2a2f10b69740c152338e499945940531b6599bf879d908cee8e105" Jul 17 06:33:47.697702 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:47.697685 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh_odh-ai-gateway-infra(a7dc2aa7-31be-4672-956f-2760b404747f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" podUID="a7dc2aa7-31be-4672-956f-2760b404747f" Jul 17 06:33:48.831816 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:48.831790 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99" Jul 17 06:33:48.951127 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:48.951091 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1784b9e9-4a76-4357-917a-c214afd77cf0-maas-api-service-ca\") pod \"1784b9e9-4a76-4357-917a-c214afd77cf0\" (UID: \"1784b9e9-4a76-4357-917a-c214afd77cf0\") " Jul 17 06:33:48.951453 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:48.951428 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1784b9e9-4a76-4357-917a-c214afd77cf0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1784b9e9-4a76-4357-917a-c214afd77cf0" (UID: "1784b9e9-4a76-4357-917a-c214afd77cf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:33:49.052353 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:49.052257 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1784b9e9-4a76-4357-917a-c214afd77cf0-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:33:49.710590 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:49.710555 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99" event={"ID":"1784b9e9-4a76-4357-917a-c214afd77cf0","Type":"ContainerDied","Data":"8aa528011ef039e7b541e4fff8d215aff268a9b8320f121bcfec6ce00468cb65"} Jul 17 06:33:49.710590 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:49.710580 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99" Jul 17 06:33:49.710806 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:49.710608 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aa528011ef039e7b541e4fff8d215aff268a9b8320f121bcfec6ce00468cb65" Jul 17 06:33:50.403645 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:50.403606 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:33:52.694504 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:52.694467 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr"] Jul 17 06:33:52.694909 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:52.694837 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1784b9e9-4a76-4357-917a-c214afd77cf0" containerName="revoke-keys" Jul 17 06:33:52.694909 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:52.694850 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1784b9e9-4a76-4357-917a-c214afd77cf0" containerName="revoke-keys" Jul 17 06:33:52.694986 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:52.694938 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1784b9e9-4a76-4357-917a-c214afd77cf0" containerName="revoke-keys" Jul 17 06:33:52.698343 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:52.698322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" Jul 17 06:33:52.706188 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:52.706165 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr"] Jul 17 06:33:52.780743 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:52.780696 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3392da-339f-4078-af69-2396558c523c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-j68zr\" (UID: \"4c3392da-339f-4078-af69-2396558c523c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" Jul 17 06:33:52.882054 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:52.882010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3392da-339f-4078-af69-2396558c523c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-j68zr\" (UID: \"4c3392da-339f-4078-af69-2396558c523c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" Jul 17 06:33:52.882741 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:52.882721 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3392da-339f-4078-af69-2396558c523c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-j68zr\" (UID: \"4c3392da-339f-4078-af69-2396558c523c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" Jul 17 06:33:53.010201 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:53.010114 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" Jul 17 06:33:53.146198 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:53.146172 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr"] Jul 17 06:33:53.149137 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:33:53.149109 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3392da_339f_4078_af69_2396558c523c.slice/crio-a64dbc2bc5a3b3a8f07e4d15fb416f738466bcdf1d3b6bbc3af9172bdd1d51c9 WatchSource:0}: Error finding container a64dbc2bc5a3b3a8f07e4d15fb416f738466bcdf1d3b6bbc3af9172bdd1d51c9: Status 404 returned error can't find the container with id a64dbc2bc5a3b3a8f07e4d15fb416f738466bcdf1d3b6bbc3af9172bdd1d51c9 Jul 17 06:33:53.731238 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:53.731123 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" event={"ID":"4c3392da-339f-4078-af69-2396558c523c","Type":"ContainerStarted","Data":"5de9cd3a70c455342595d9dd8211b1b4d0b754f3aebc28ac0a8c69e01937d246"} Jul 17 06:33:53.731238 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:53.731200 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" event={"ID":"4c3392da-339f-4078-af69-2396558c523c","Type":"ContainerStarted","Data":"a64dbc2bc5a3b3a8f07e4d15fb416f738466bcdf1d3b6bbc3af9172bdd1d51c9"} Jul 17 06:33:54.737708 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:54.737669 2580 generic.go:356] "Generic (PLEG): container finished" podID="4c3392da-339f-4078-af69-2396558c523c" containerID="5de9cd3a70c455342595d9dd8211b1b4d0b754f3aebc28ac0a8c69e01937d246" exitCode=6 Jul 17 06:33:54.738230 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:54.737753 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" event={"ID":"4c3392da-339f-4078-af69-2396558c523c","Type":"ContainerDied","Data":"5de9cd3a70c455342595d9dd8211b1b4d0b754f3aebc28ac0a8c69e01937d246"} Jul 17 06:33:54.738230 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:54.737981 2580 scope.go:117] "RemoveContainer" containerID="5de9cd3a70c455342595d9dd8211b1b4d0b754f3aebc28ac0a8c69e01937d246" Jul 17 06:33:55.744579 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:55.744481 2580 generic.go:356] "Generic (PLEG): container finished" podID="4c3392da-339f-4078-af69-2396558c523c" containerID="a04f980dc5b5377821ce6743cba6654224b5bc952117263fa9f13f24452d81a7" exitCode=6 Jul 17 06:33:55.744579 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:55.744561 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" event={"ID":"4c3392da-339f-4078-af69-2396558c523c","Type":"ContainerDied","Data":"a04f980dc5b5377821ce6743cba6654224b5bc952117263fa9f13f24452d81a7"} Jul 17 06:33:55.745079 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:55.744627 2580 scope.go:117] "RemoveContainer" containerID="5de9cd3a70c455342595d9dd8211b1b4d0b754f3aebc28ac0a8c69e01937d246" Jul 17 06:33:55.745079 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:55.744806 2580 scope.go:117] "RemoveContainer" containerID="a04f980dc5b5377821ce6743cba6654224b5bc952117263fa9f13f24452d81a7" Jul 17 06:33:55.745079 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:55.745060 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-j68zr_odh-ai-gateway-infra(4c3392da-339f-4078-af69-2396558c523c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" podUID="4c3392da-339f-4078-af69-2396558c523c" Jul 17 06:33:56.387493 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:56.387455 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m"] Jul 17 06:33:56.390820 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:56.390801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" Jul 17 06:33:56.398950 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:56.398919 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m"] Jul 17 06:33:56.413797 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:56.413765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce66f64-2733-4506-b0e1-c2420cc60fae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m\" (UID: \"cce66f64-2733-4506-b0e1-c2420cc60fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" Jul 17 06:33:56.514476 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:56.514432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce66f64-2733-4506-b0e1-c2420cc60fae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m\" (UID: \"cce66f64-2733-4506-b0e1-c2420cc60fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" Jul 17 06:33:56.515058 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:56.515038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce66f64-2733-4506-b0e1-c2420cc60fae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m\" (UID: \"cce66f64-2733-4506-b0e1-c2420cc60fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" Jul 17 06:33:56.702824 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:56.702712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" Jul 17 06:33:56.751237 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:56.751210 2580 scope.go:117] "RemoveContainer" containerID="a04f980dc5b5377821ce6743cba6654224b5bc952117263fa9f13f24452d81a7" Jul 17 06:33:56.751690 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:56.751461 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-j68zr_odh-ai-gateway-infra(4c3392da-339f-4078-af69-2396558c523c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" podUID="4c3392da-339f-4078-af69-2396558c523c" Jul 17 06:33:56.844072 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:56.844043 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m"] Jul 17 06:33:56.845915 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:33:56.845886 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce66f64_2733_4506_b0e1_c2420cc60fae.slice/crio-254da7b247eaf20199e3294e830ae87415b0dd5eeaf4ffbf33c278e3a35d2e19 WatchSource:0}: Error finding container 254da7b247eaf20199e3294e830ae87415b0dd5eeaf4ffbf33c278e3a35d2e19: Status 404 returned error can't find the container with id 254da7b247eaf20199e3294e830ae87415b0dd5eeaf4ffbf33c278e3a35d2e19 Jul 17 06:33:56.926882 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:56.926847 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8"] Jul 17 06:33:56.927087 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:56.927059 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" podUID="551c0fba-fca9-46fc-9d67-647d86c07c28" containerName="maas-api" containerID="cri-o://a91d65cd61ec036627ee7f434e1c0673df4cd958ec89739ef893ab2c152be21a" gracePeriod=30 Jul 17 06:33:56.968120 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:56.968028 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:33:57.188624 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.188599 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:33:57.221573 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.221493 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/551c0fba-fca9-46fc-9d67-647d86c07c28-maas-api-tls\") pod \"551c0fba-fca9-46fc-9d67-647d86c07c28\" (UID: \"551c0fba-fca9-46fc-9d67-647d86c07c28\") " Jul 17 06:33:57.221719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.221581 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/551c0fba-fca9-46fc-9d67-647d86c07c28-openshift-service-ca\") pod \"551c0fba-fca9-46fc-9d67-647d86c07c28\" (UID: \"551c0fba-fca9-46fc-9d67-647d86c07c28\") " Jul 17 06:33:57.221719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.221617 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g45br\" (UniqueName: \"kubernetes.io/projected/551c0fba-fca9-46fc-9d67-647d86c07c28-kube-api-access-g45br\") pod \"551c0fba-fca9-46fc-9d67-647d86c07c28\" (UID: \"551c0fba-fca9-46fc-9d67-647d86c07c28\") " Jul 17 06:33:57.222006 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.221973 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551c0fba-fca9-46fc-9d67-647d86c07c28-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "551c0fba-fca9-46fc-9d67-647d86c07c28" (UID: "551c0fba-fca9-46fc-9d67-647d86c07c28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:33:57.223823 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.223797 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551c0fba-fca9-46fc-9d67-647d86c07c28-kube-api-access-g45br" (OuterVolumeSpecName: "kube-api-access-g45br") pod "551c0fba-fca9-46fc-9d67-647d86c07c28" (UID: "551c0fba-fca9-46fc-9d67-647d86c07c28"). InnerVolumeSpecName "kube-api-access-g45br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:33:57.223823 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.223803 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551c0fba-fca9-46fc-9d67-647d86c07c28-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "551c0fba-fca9-46fc-9d67-647d86c07c28" (UID: "551c0fba-fca9-46fc-9d67-647d86c07c28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:33:57.322707 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.322666 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45br\" (UniqueName: \"kubernetes.io/projected/551c0fba-fca9-46fc-9d67-647d86c07c28-kube-api-access-g45br\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:33:57.322707 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.322700 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/551c0fba-fca9-46fc-9d67-647d86c07c28-maas-api-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:33:57.322707 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.322709 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/551c0fba-fca9-46fc-9d67-647d86c07c28-openshift-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:33:57.757737 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.757695 2580 generic.go:356] "Generic (PLEG): container finished" podID="cce66f64-2733-4506-b0e1-c2420cc60fae" containerID="0784ce7b30483c3d3774c943fe67bdb74839e20d54d6cc6742c5c5d485cb2745" exitCode=6 Jul 17 06:33:57.758202 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.757789 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" event={"ID":"cce66f64-2733-4506-b0e1-c2420cc60fae","Type":"ContainerDied","Data":"0784ce7b30483c3d3774c943fe67bdb74839e20d54d6cc6742c5c5d485cb2745"} Jul 17 06:33:57.758202 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.757860 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" event={"ID":"cce66f64-2733-4506-b0e1-c2420cc60fae","Type":"ContainerStarted","Data":"254da7b247eaf20199e3294e830ae87415b0dd5eeaf4ffbf33c278e3a35d2e19"} Jul 17 06:33:57.758202 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.758011 2580 scope.go:117] "RemoveContainer" containerID="0784ce7b30483c3d3774c943fe67bdb74839e20d54d6cc6742c5c5d485cb2745" Jul 17 06:33:57.759296 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.759267 2580 generic.go:356] "Generic (PLEG): container finished" podID="551c0fba-fca9-46fc-9d67-647d86c07c28" containerID="a91d65cd61ec036627ee7f434e1c0673df4cd958ec89739ef893ab2c152be21a" exitCode=0 Jul 17 06:33:57.759422 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.759298 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" event={"ID":"551c0fba-fca9-46fc-9d67-647d86c07c28","Type":"ContainerDied","Data":"a91d65cd61ec036627ee7f434e1c0673df4cd958ec89739ef893ab2c152be21a"} Jul 17 06:33:57.759422 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.759309 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" Jul 17 06:33:57.759422 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.759346 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8" event={"ID":"551c0fba-fca9-46fc-9d67-647d86c07c28","Type":"ContainerDied","Data":"94b9c2e4a9160753b2353187a244f821075497949e150084c7ecf636b631a251"} Jul 17 06:33:57.759422 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.759368 2580 scope.go:117] "RemoveContainer" containerID="a91d65cd61ec036627ee7f434e1c0673df4cd958ec89739ef893ab2c152be21a" Jul 17 06:33:57.774598 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.774571 2580 scope.go:117] "RemoveContainer" containerID="a91d65cd61ec036627ee7f434e1c0673df4cd958ec89739ef893ab2c152be21a" Jul 17 06:33:57.774982 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:57.774889 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91d65cd61ec036627ee7f434e1c0673df4cd958ec89739ef893ab2c152be21a\": container with ID starting with a91d65cd61ec036627ee7f434e1c0673df4cd958ec89739ef893ab2c152be21a not found: ID does not exist" containerID="a91d65cd61ec036627ee7f434e1c0673df4cd958ec89739ef893ab2c152be21a" Jul 17 06:33:57.774982 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.774924 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91d65cd61ec036627ee7f434e1c0673df4cd958ec89739ef893ab2c152be21a"} err="failed to get container status \"a91d65cd61ec036627ee7f434e1c0673df4cd958ec89739ef893ab2c152be21a\": rpc error: code = NotFound desc = could not find container \"a91d65cd61ec036627ee7f434e1c0673df4cd958ec89739ef893ab2c152be21a\": container with ID starting with a91d65cd61ec036627ee7f434e1c0673df4cd958ec89739ef893ab2c152be21a not found: ID does not exist" Jul 17 06:33:57.786870 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.786797 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8"] Jul 17 06:33:57.790487 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:57.790461 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e77d3d-66b8f945db-v8kl8"] Jul 17 06:33:58.766561 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:58.766523 2580 generic.go:356] "Generic (PLEG): container finished" podID="cce66f64-2733-4506-b0e1-c2420cc60fae" containerID="ca83f01a9869fb7e45568568cbc9ef2b81c8309666d2212f5901e1ebb4ac5b9f" exitCode=6 Jul 17 06:33:58.767024 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:58.766576 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" event={"ID":"cce66f64-2733-4506-b0e1-c2420cc60fae","Type":"ContainerDied","Data":"ca83f01a9869fb7e45568568cbc9ef2b81c8309666d2212f5901e1ebb4ac5b9f"} Jul 17 06:33:58.767024 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:58.766627 2580 scope.go:117] "RemoveContainer" containerID="0784ce7b30483c3d3774c943fe67bdb74839e20d54d6cc6742c5c5d485cb2745" Jul 17 06:33:58.767024 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:58.766779 2580 scope.go:117] "RemoveContainer" containerID="ca83f01a9869fb7e45568568cbc9ef2b81c8309666d2212f5901e1ebb4ac5b9f" Jul 17 06:33:58.767125 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:58.767042 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m_odh-ai-gateway-infra(cce66f64-2733-4506-b0e1-c2420cc60fae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" podUID="cce66f64-2733-4506-b0e1-c2420cc60fae" Jul 17 06:33:58.884508 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:58.884474 2580 scope.go:117] "RemoveContainer" containerID="b3225a3b7b2a2f10b69740c152338e499945940531b6599bf879d908cee8e105" Jul 17 06:33:58.889727 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:58.889698 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="551c0fba-fca9-46fc-9d67-647d86c07c28" path="/var/lib/kubelet/pods/551c0fba-fca9-46fc-9d67-647d86c07c28/volumes" Jul 17 06:33:59.778263 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:59.778159 2580 scope.go:117] "RemoveContainer" containerID="ca83f01a9869fb7e45568568cbc9ef2b81c8309666d2212f5901e1ebb4ac5b9f" Jul 17 06:33:59.778706 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:59.778428 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m_odh-ai-gateway-infra(cce66f64-2733-4506-b0e1-c2420cc60fae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" podUID="cce66f64-2733-4506-b0e1-c2420cc60fae" Jul 17 06:33:59.780055 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:59.780032 2580 generic.go:356] "Generic (PLEG): container finished" podID="a7dc2aa7-31be-4672-956f-2760b404747f" containerID="41884f94c8f7ccf27d0ae814a9f898529d682b874778baa1491fc62aa9daa6aa" exitCode=6 Jul 17 06:33:59.780158 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:59.780064 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" event={"ID":"a7dc2aa7-31be-4672-956f-2760b404747f","Type":"ContainerDied","Data":"41884f94c8f7ccf27d0ae814a9f898529d682b874778baa1491fc62aa9daa6aa"} Jul 17 06:33:59.780158 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:59.780133 2580 scope.go:117] "RemoveContainer" containerID="b3225a3b7b2a2f10b69740c152338e499945940531b6599bf879d908cee8e105" Jul 17 06:33:59.780408 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:33:59.780375 2580 scope.go:117] "RemoveContainer" containerID="41884f94c8f7ccf27d0ae814a9f898529d682b874778baa1491fc62aa9daa6aa" Jul 17 06:33:59.780601 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:33:59.780579 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh_odh-ai-gateway-infra(a7dc2aa7-31be-4672-956f-2760b404747f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" podUID="a7dc2aa7-31be-4672-956f-2760b404747f" Jul 17 06:34:00.293048 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.293002 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:34:00.308282 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.308245 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j"] Jul 17 06:34:00.308626 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.308611 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="551c0fba-fca9-46fc-9d67-647d86c07c28" containerName="maas-api" Jul 17 06:34:00.308674 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.308627 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="551c0fba-fca9-46fc-9d67-647d86c07c28" containerName="maas-api" Jul 17 06:34:00.308706 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.308699 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="551c0fba-fca9-46fc-9d67-647d86c07c28" containerName="maas-api" Jul 17 06:34:00.312783 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.312766 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" Jul 17 06:34:00.322157 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.322128 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j"] Jul 17 06:34:00.347424 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.347367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab2d2d-7c06-4d62-9a60-1b92b8e123fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-9765j\" (UID: \"6bab2d2d-7c06-4d62-9a60-1b92b8e123fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" Jul 17 06:34:00.448518 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.448475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab2d2d-7c06-4d62-9a60-1b92b8e123fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-9765j\" (UID: \"6bab2d2d-7c06-4d62-9a60-1b92b8e123fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" Jul 17 06:34:00.449143 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.449122 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab2d2d-7c06-4d62-9a60-1b92b8e123fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-9765j\" (UID: \"6bab2d2d-7c06-4d62-9a60-1b92b8e123fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" Jul 17 06:34:00.623657 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.623608 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" Jul 17 06:34:00.757623 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.757596 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j"] Jul 17 06:34:00.759590 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:34:00.759558 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bab2d2d_7c06_4d62_9a60_1b92b8e123fe.slice/crio-1e2006ae7c6a4588f567db1446d07b2536ae7cc29feb76bbc7a706198fff8309 WatchSource:0}: Error finding container 1e2006ae7c6a4588f567db1446d07b2536ae7cc29feb76bbc7a706198fff8309: Status 404 returned error can't find the container with id 1e2006ae7c6a4588f567db1446d07b2536ae7cc29feb76bbc7a706198fff8309 Jul 17 06:34:00.789185 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.789152 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" event={"ID":"6bab2d2d-7c06-4d62-9a60-1b92b8e123fe","Type":"ContainerStarted","Data":"1e2006ae7c6a4588f567db1446d07b2536ae7cc29feb76bbc7a706198fff8309"} Jul 17 06:34:00.822734 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.822699 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh"] Jul 17 06:34:00.961923 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:00.961896 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" Jul 17 06:34:01.054289 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:01.054257 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7dc2aa7-31be-4672-956f-2760b404747f-maas-api-service-ca\") pod \"a7dc2aa7-31be-4672-956f-2760b404747f\" (UID: \"a7dc2aa7-31be-4672-956f-2760b404747f\") " Jul 17 06:34:01.054696 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:01.054663 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7dc2aa7-31be-4672-956f-2760b404747f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a7dc2aa7-31be-4672-956f-2760b404747f" (UID: "a7dc2aa7-31be-4672-956f-2760b404747f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:34:01.155725 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:01.155622 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7dc2aa7-31be-4672-956f-2760b404747f-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:34:01.410136 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:01.410026 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:01.751056 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:01.750965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:01.796272 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:01.796235 2580 generic.go:356] "Generic (PLEG): container finished" podID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" containerID="fb7e29256eb0b23eaa47b0f8bc794c78cc35d12296e8faeee0ac979270ceed00" exitCode=6 Jul 17 06:34:01.796771 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:01.796403 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" event={"ID":"6bab2d2d-7c06-4d62-9a60-1b92b8e123fe","Type":"ContainerDied","Data":"fb7e29256eb0b23eaa47b0f8bc794c78cc35d12296e8faeee0ac979270ceed00"} Jul 17 06:34:01.796771 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:01.796624 2580 scope.go:117] "RemoveContainer" containerID="fb7e29256eb0b23eaa47b0f8bc794c78cc35d12296e8faeee0ac979270ceed00" Jul 17 06:34:01.797927 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:01.797901 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" event={"ID":"a7dc2aa7-31be-4672-956f-2760b404747f","Type":"ContainerDied","Data":"b7c015d78f657dc209275ccdcb41087f513f4a0f588cffe8da6c7b2644bd4575"} Jul 17 06:34:01.798045 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:01.797961 2580 scope.go:117] "RemoveContainer" containerID="41884f94c8f7ccf27d0ae814a9f898529d682b874778baa1491fc62aa9daa6aa" Jul 17 06:34:01.798045 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:01.797965 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh" Jul 17 06:34:01.831983 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:01.831955 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh"] Jul 17 06:34:01.839886 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:01.839860 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-j5qgh"] Jul 17 06:34:02.805303 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:02.805268 2580 generic.go:356] "Generic (PLEG): container finished" podID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" containerID="7d1f6315468b2a6cfa6510f6ccc6f21dc10088641d35292e5ed12efd0db460df" exitCode=6 Jul 17 06:34:02.805303 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:02.805310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" event={"ID":"6bab2d2d-7c06-4d62-9a60-1b92b8e123fe","Type":"ContainerDied","Data":"7d1f6315468b2a6cfa6510f6ccc6f21dc10088641d35292e5ed12efd0db460df"} Jul 17 06:34:02.805824 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:02.805371 2580 scope.go:117] "RemoveContainer" containerID="fb7e29256eb0b23eaa47b0f8bc794c78cc35d12296e8faeee0ac979270ceed00" Jul 17 06:34:02.805824 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:02.805608 2580 scope.go:117] "RemoveContainer" containerID="7d1f6315468b2a6cfa6510f6ccc6f21dc10088641d35292e5ed12efd0db460df" Jul 17 06:34:02.805922 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:02.805856 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-9765j_odh-ai-gateway-infra(6bab2d2d-7c06-4d62-9a60-1b92b8e123fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" podUID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" Jul 17 06:34:02.889992 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:02.889961 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7dc2aa7-31be-4672-956f-2760b404747f" path="/var/lib/kubelet/pods/a7dc2aa7-31be-4672-956f-2760b404747f/volumes" Jul 17 06:34:03.812376 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:03.812346 2580 scope.go:117] "RemoveContainer" containerID="7d1f6315468b2a6cfa6510f6ccc6f21dc10088641d35292e5ed12efd0db460df" Jul 17 06:34:03.812831 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:03.812594 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-9765j_odh-ai-gateway-infra(6bab2d2d-7c06-4d62-9a60-1b92b8e123fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" podUID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" Jul 17 06:34:08.299876 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:08.299808 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:08.422778 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:08.422737 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:08.636481 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:08.636447 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:08.799355 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:08.799313 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:08.923877 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:08.923791 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs"] Jul 17 06:34:08.924159 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:08.924146 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7dc2aa7-31be-4672-956f-2760b404747f" containerName="revoke-keys" Jul 17 06:34:08.924208 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:08.924161 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7dc2aa7-31be-4672-956f-2760b404747f" containerName="revoke-keys" Jul 17 06:34:08.924208 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:08.924172 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7dc2aa7-31be-4672-956f-2760b404747f" containerName="revoke-keys" Jul 17 06:34:08.924208 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:08.924179 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7dc2aa7-31be-4672-956f-2760b404747f" containerName="revoke-keys" Jul 17 06:34:08.924298 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:08.924250 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7dc2aa7-31be-4672-956f-2760b404747f" containerName="revoke-keys" Jul 17 06:34:08.924298 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:08.924258 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7dc2aa7-31be-4672-956f-2760b404747f" containerName="revoke-keys" Jul 17 06:34:08.928708 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:08.928683 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs" Jul 17 06:34:08.944991 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:08.944963 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs"] Jul 17 06:34:09.024487 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:09.024451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19cdc854-30c1-4bb1-a199-2693b4b7c69e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-142438-d8tzs\" (UID: \"19cdc854-30c1-4bb1-a199-2693b4b7c69e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs" Jul 17 06:34:09.125513 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:09.125471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19cdc854-30c1-4bb1-a199-2693b4b7c69e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-142438-d8tzs\" (UID: \"19cdc854-30c1-4bb1-a199-2693b4b7c69e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs" Jul 17 06:34:09.126182 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:09.126159 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19cdc854-30c1-4bb1-a199-2693b4b7c69e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-142438-d8tzs\" (UID: \"19cdc854-30c1-4bb1-a199-2693b4b7c69e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs" Jul 17 06:34:09.238744 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:09.238650 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs" Jul 17 06:34:09.384844 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:09.384812 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs"] Jul 17 06:34:09.387462 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:34:09.387421 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19cdc854_30c1_4bb1_a199_2693b4b7c69e.slice/crio-323e570c0ae38c046dc793f69dc5c720ea23eb954a78773b02ae781378840374 WatchSource:0}: Error finding container 323e570c0ae38c046dc793f69dc5c720ea23eb954a78773b02ae781378840374: Status 404 returned error can't find the container with id 323e570c0ae38c046dc793f69dc5c720ea23eb954a78773b02ae781378840374 Jul 17 06:34:09.622609 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:09.622575 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:09.721942 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:09.721906 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:09.843342 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:09.843247 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs" event={"ID":"19cdc854-30c1-4bb1-a199-2693b4b7c69e","Type":"ContainerStarted","Data":"f40c2a75706f6a9966aac9ad16d46636e260927d0ac3274e90350bc68bd1e508"} Jul 17 06:34:09.843342 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:09.843303 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs" event={"ID":"19cdc854-30c1-4bb1-a199-2693b4b7c69e","Type":"ContainerStarted","Data":"323e570c0ae38c046dc793f69dc5c720ea23eb954a78773b02ae781378840374"} Jul 17 06:34:09.873786 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:09.873726 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs" podStartSLOduration=1.8737078820000002 podStartE2EDuration="1.873707882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:34:09.871799362 +0000 UTC m=+3174.616478688" watchObservedRunningTime="2026-07-17 06:34:09.873707882 +0000 UTC m=+3174.618387209" Jul 17 06:34:09.884027 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:09.883992 2580 scope.go:117] "RemoveContainer" containerID="a04f980dc5b5377821ce6743cba6654224b5bc952117263fa9f13f24452d81a7" Jul 17 06:34:10.849657 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:10.849562 2580 generic.go:356] "Generic (PLEG): container finished" podID="4c3392da-339f-4078-af69-2396558c523c" containerID="d13449ca1c863ca2acf61236ca5a3da742af29bbc8ab5363b3189487fd6da82a" exitCode=6 Jul 17 06:34:10.849657 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:10.849633 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" event={"ID":"4c3392da-339f-4078-af69-2396558c523c","Type":"ContainerDied","Data":"d13449ca1c863ca2acf61236ca5a3da742af29bbc8ab5363b3189487fd6da82a"} Jul 17 06:34:10.850185 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:10.849696 2580 scope.go:117] "RemoveContainer" containerID="a04f980dc5b5377821ce6743cba6654224b5bc952117263fa9f13f24452d81a7" Jul 17 06:34:10.850185 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:10.850021 2580 scope.go:117] "RemoveContainer" containerID="d13449ca1c863ca2acf61236ca5a3da742af29bbc8ab5363b3189487fd6da82a" Jul 17 06:34:10.850309 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:10.850282 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-j68zr_odh-ai-gateway-infra(4c3392da-339f-4078-af69-2396558c523c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" podUID="4c3392da-339f-4078-af69-2396558c523c" Jul 17 06:34:10.851290 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:10.851268 2580 generic.go:356] "Generic (PLEG): container finished" podID="19cdc854-30c1-4bb1-a199-2693b4b7c69e" containerID="f40c2a75706f6a9966aac9ad16d46636e260927d0ac3274e90350bc68bd1e508" exitCode=0 Jul 17 06:34:10.851417 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:10.851297 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs" event={"ID":"19cdc854-30c1-4bb1-a199-2693b4b7c69e","Type":"ContainerDied","Data":"f40c2a75706f6a9966aac9ad16d46636e260927d0ac3274e90350bc68bd1e508"} Jul 17 06:34:11.883278 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:11.883235 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr"] Jul 17 06:34:11.884047 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:11.884029 2580 scope.go:117] "RemoveContainer" containerID="ca83f01a9869fb7e45568568cbc9ef2b81c8309666d2212f5901e1ebb4ac5b9f" Jul 17 06:34:12.010871 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.010842 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs" Jul 17 06:34:12.053277 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.053247 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19cdc854-30c1-4bb1-a199-2693b4b7c69e-maas-api-service-ca\") pod \"19cdc854-30c1-4bb1-a199-2693b4b7c69e\" (UID: \"19cdc854-30c1-4bb1-a199-2693b4b7c69e\") " Jul 17 06:34:12.053710 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.053681 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19cdc854-30c1-4bb1-a199-2693b4b7c69e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "19cdc854-30c1-4bb1-a199-2693b4b7c69e" (UID: "19cdc854-30c1-4bb1-a199-2693b4b7c69e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:34:12.058537 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.058508 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" Jul 17 06:34:12.154204 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.154178 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3392da-339f-4078-af69-2396558c523c-maas-api-service-ca\") pod \"4c3392da-339f-4078-af69-2396558c523c\" (UID: \"4c3392da-339f-4078-af69-2396558c523c\") " Jul 17 06:34:12.154559 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.154539 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19cdc854-30c1-4bb1-a199-2693b4b7c69e-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:34:12.154600 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.154587 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3392da-339f-4078-af69-2396558c523c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c3392da-339f-4078-af69-2396558c523c" (UID: "4c3392da-339f-4078-af69-2396558c523c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:34:12.255551 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.255507 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3392da-339f-4078-af69-2396558c523c-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:34:12.863494 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.863459 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs" Jul 17 06:34:12.863706 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.863495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-142438-d8tzs" event={"ID":"19cdc854-30c1-4bb1-a199-2693b4b7c69e","Type":"ContainerDied","Data":"323e570c0ae38c046dc793f69dc5c720ea23eb954a78773b02ae781378840374"} Jul 17 06:34:12.863706 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.863547 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="323e570c0ae38c046dc793f69dc5c720ea23eb954a78773b02ae781378840374" Jul 17 06:34:12.864815 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.864782 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" Jul 17 06:34:12.864956 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.864784 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr" event={"ID":"4c3392da-339f-4078-af69-2396558c523c","Type":"ContainerDied","Data":"a64dbc2bc5a3b3a8f07e4d15fb416f738466bcdf1d3b6bbc3af9172bdd1d51c9"} Jul 17 06:34:12.864956 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.864924 2580 scope.go:117] "RemoveContainer" containerID="d13449ca1c863ca2acf61236ca5a3da742af29bbc8ab5363b3189487fd6da82a" Jul 17 06:34:12.867000 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.866973 2580 generic.go:356] "Generic (PLEG): container finished" podID="cce66f64-2733-4506-b0e1-c2420cc60fae" containerID="8548079070ab99b9bcf296acc79333c43e25a65084015b0506ecc479aa260ae4" exitCode=6 Jul 17 06:34:12.867136 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.867015 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" event={"ID":"cce66f64-2733-4506-b0e1-c2420cc60fae","Type":"ContainerDied","Data":"8548079070ab99b9bcf296acc79333c43e25a65084015b0506ecc479aa260ae4"} Jul 17 06:34:12.867354 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.867336 2580 scope.go:117] "RemoveContainer" containerID="8548079070ab99b9bcf296acc79333c43e25a65084015b0506ecc479aa260ae4" Jul 17 06:34:12.867606 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:12.867583 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m_odh-ai-gateway-infra(cce66f64-2733-4506-b0e1-c2420cc60fae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" podUID="cce66f64-2733-4506-b0e1-c2420cc60fae" Jul 17 06:34:12.874831 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.874812 2580 scope.go:117] "RemoveContainer" containerID="ca83f01a9869fb7e45568568cbc9ef2b81c8309666d2212f5901e1ebb4ac5b9f" Jul 17 06:34:12.926002 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.925963 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr"] Jul 17 06:34:12.940321 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:12.940285 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-j68zr"] Jul 17 06:34:13.928615 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:13.928576 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m"] Jul 17 06:34:14.067206 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:14.067178 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" Jul 17 06:34:14.172613 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:14.172572 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce66f64-2733-4506-b0e1-c2420cc60fae-maas-api-service-ca\") pod \"cce66f64-2733-4506-b0e1-c2420cc60fae\" (UID: \"cce66f64-2733-4506-b0e1-c2420cc60fae\") " Jul 17 06:34:14.172974 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:14.172950 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce66f64-2733-4506-b0e1-c2420cc60fae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cce66f64-2733-4506-b0e1-c2420cc60fae" (UID: "cce66f64-2733-4506-b0e1-c2420cc60fae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:34:14.273827 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:14.273732 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce66f64-2733-4506-b0e1-c2420cc60fae-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:34:14.886963 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:14.886930 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" Jul 17 06:34:14.891949 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:14.891912 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c3392da-339f-4078-af69-2396558c523c" path="/var/lib/kubelet/pods/4c3392da-339f-4078-af69-2396558c523c/volumes" Jul 17 06:34:14.892492 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:14.892469 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m" event={"ID":"cce66f64-2733-4506-b0e1-c2420cc60fae","Type":"ContainerDied","Data":"254da7b247eaf20199e3294e830ae87415b0dd5eeaf4ffbf33c278e3a35d2e19"} Jul 17 06:34:14.892538 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:14.892531 2580 scope.go:117] "RemoveContainer" containerID="8548079070ab99b9bcf296acc79333c43e25a65084015b0506ecc479aa260ae4" Jul 17 06:34:14.913134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:14.913102 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m"] Jul 17 06:34:14.920139 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:14.920106 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2pm2m"] Jul 17 06:34:16.889944 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:16.889900 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cce66f64-2733-4506-b0e1-c2420cc60fae" path="/var/lib/kubelet/pods/cce66f64-2733-4506-b0e1-c2420cc60fae/volumes" Jul 17 06:34:17.884018 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:17.883986 2580 scope.go:117] "RemoveContainer" containerID="7d1f6315468b2a6cfa6510f6ccc6f21dc10088641d35292e5ed12efd0db460df" Jul 17 06:34:18.909618 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:18.909582 2580 generic.go:356] "Generic (PLEG): container finished" podID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" containerID="fed00ed1466ca35f085b60612ddfcc4ac009ab5c515b848d19a1e6b3b611b5e4" exitCode=6 Jul 17 06:34:18.910036 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:18.909658 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" event={"ID":"6bab2d2d-7c06-4d62-9a60-1b92b8e123fe","Type":"ContainerDied","Data":"fed00ed1466ca35f085b60612ddfcc4ac009ab5c515b848d19a1e6b3b611b5e4"} Jul 17 06:34:18.910036 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:18.909720 2580 scope.go:117] "RemoveContainer" containerID="7d1f6315468b2a6cfa6510f6ccc6f21dc10088641d35292e5ed12efd0db460df" Jul 17 06:34:18.910036 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:18.909984 2580 scope.go:117] "RemoveContainer" containerID="fed00ed1466ca35f085b60612ddfcc4ac009ab5c515b848d19a1e6b3b611b5e4" Jul 17 06:34:18.910219 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:18.910195 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-9765j_odh-ai-gateway-infra(6bab2d2d-7c06-4d62-9a60-1b92b8e123fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" podUID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" Jul 17 06:34:19.508591 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.508549 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55"] Jul 17 06:34:19.508806 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.508764 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" podUID="bb2c4647-7de5-4693-8975-1b1f671b80d5" containerName="maas-api" containerID="cri-o://bcb01c44936bc617efb205e1b45108e684150d0564038e10bc257a5d3e4fb035" gracePeriod=30 Jul 17 06:34:19.560472 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.560432 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:19.760801 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.760731 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:34:19.821332 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.821289 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94czh\" (UniqueName: \"kubernetes.io/projected/bb2c4647-7de5-4693-8975-1b1f671b80d5-kube-api-access-94czh\") pod \"bb2c4647-7de5-4693-8975-1b1f671b80d5\" (UID: \"bb2c4647-7de5-4693-8975-1b1f671b80d5\") " Jul 17 06:34:19.821549 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.821414 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2c4647-7de5-4693-8975-1b1f671b80d5-openshift-service-ca\") pod \"bb2c4647-7de5-4693-8975-1b1f671b80d5\" (UID: \"bb2c4647-7de5-4693-8975-1b1f671b80d5\") " Jul 17 06:34:19.821549 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.821489 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb2c4647-7de5-4693-8975-1b1f671b80d5-maas-api-tls\") pod \"bb2c4647-7de5-4693-8975-1b1f671b80d5\" (UID: \"bb2c4647-7de5-4693-8975-1b1f671b80d5\") " Jul 17 06:34:19.821825 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.821795 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2c4647-7de5-4693-8975-1b1f671b80d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb2c4647-7de5-4693-8975-1b1f671b80d5" (UID: "bb2c4647-7de5-4693-8975-1b1f671b80d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:34:19.823490 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.823450 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2c4647-7de5-4693-8975-1b1f671b80d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb2c4647-7de5-4693-8975-1b1f671b80d5" (UID: "bb2c4647-7de5-4693-8975-1b1f671b80d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:34:19.823549 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.823529 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2c4647-7de5-4693-8975-1b1f671b80d5-kube-api-access-94czh" (OuterVolumeSpecName: "kube-api-access-94czh") pod "bb2c4647-7de5-4693-8975-1b1f671b80d5" (UID: "bb2c4647-7de5-4693-8975-1b1f671b80d5"). InnerVolumeSpecName "kube-api-access-94czh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:34:19.915362 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.915324 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb2c4647-7de5-4693-8975-1b1f671b80d5" containerID="bcb01c44936bc617efb205e1b45108e684150d0564038e10bc257a5d3e4fb035" exitCode=0 Jul 17 06:34:19.915362 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.915368 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" Jul 17 06:34:19.915913 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.915362 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" event={"ID":"bb2c4647-7de5-4693-8975-1b1f671b80d5","Type":"ContainerDied","Data":"bcb01c44936bc617efb205e1b45108e684150d0564038e10bc257a5d3e4fb035"} Jul 17 06:34:19.915913 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.915506 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55" event={"ID":"bb2c4647-7de5-4693-8975-1b1f671b80d5","Type":"ContainerDied","Data":"4d3063523c0812cf5d5842eec85701d8b0dd111f358322a89c945388ddfc84b5"} Jul 17 06:34:19.915913 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.915535 2580 scope.go:117] "RemoveContainer" containerID="bcb01c44936bc617efb205e1b45108e684150d0564038e10bc257a5d3e4fb035" Jul 17 06:34:19.922072 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.922048 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb2c4647-7de5-4693-8975-1b1f671b80d5-maas-api-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:34:19.922072 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.922071 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94czh\" (UniqueName: \"kubernetes.io/projected/bb2c4647-7de5-4693-8975-1b1f671b80d5-kube-api-access-94czh\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:34:19.922251 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.922084 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2c4647-7de5-4693-8975-1b1f671b80d5-openshift-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:34:19.925490 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.925470 2580 scope.go:117] "RemoveContainer" containerID="bcb01c44936bc617efb205e1b45108e684150d0564038e10bc257a5d3e4fb035" Jul 17 06:34:19.925785 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:19.925765 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb01c44936bc617efb205e1b45108e684150d0564038e10bc257a5d3e4fb035\": container with ID starting with bcb01c44936bc617efb205e1b45108e684150d0564038e10bc257a5d3e4fb035 not found: ID does not exist" containerID="bcb01c44936bc617efb205e1b45108e684150d0564038e10bc257a5d3e4fb035" Jul 17 06:34:19.925833 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.925795 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb01c44936bc617efb205e1b45108e684150d0564038e10bc257a5d3e4fb035"} err="failed to get container status \"bcb01c44936bc617efb205e1b45108e684150d0564038e10bc257a5d3e4fb035\": rpc error: code = NotFound desc = could not find container \"bcb01c44936bc617efb205e1b45108e684150d0564038e10bc257a5d3e4fb035\": container with ID starting with bcb01c44936bc617efb205e1b45108e684150d0564038e10bc257a5d3e4fb035 not found: ID does not exist" Jul 17 06:34:19.941040 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.941005 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55"] Jul 17 06:34:19.948113 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.948085 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j"] Jul 17 06:34:19.960477 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:19.960442 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-142438-6798dd7fc-9vw55"] Jul 17 06:34:20.072883 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:20.072855 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" Jul 17 06:34:20.223804 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:20.223762 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab2d2d-7c06-4d62-9a60-1b92b8e123fe-maas-api-service-ca\") pod \"6bab2d2d-7c06-4d62-9a60-1b92b8e123fe\" (UID: \"6bab2d2d-7c06-4d62-9a60-1b92b8e123fe\") " Jul 17 06:34:20.224147 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:20.224123 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bab2d2d-7c06-4d62-9a60-1b92b8e123fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" (UID: "6bab2d2d-7c06-4d62-9a60-1b92b8e123fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:34:20.325250 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:20.325134 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab2d2d-7c06-4d62-9a60-1b92b8e123fe-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:34:20.889474 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:20.889437 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb2c4647-7de5-4693-8975-1b1f671b80d5" path="/var/lib/kubelet/pods/bb2c4647-7de5-4693-8975-1b1f671b80d5/volumes" Jul 17 06:34:20.924803 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:20.924765 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" event={"ID":"6bab2d2d-7c06-4d62-9a60-1b92b8e123fe","Type":"ContainerDied","Data":"1e2006ae7c6a4588f567db1446d07b2536ae7cc29feb76bbc7a706198fff8309"} Jul 17 06:34:20.924803 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:20.924795 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j" Jul 17 06:34:20.925337 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:20.924838 2580 scope.go:117] "RemoveContainer" containerID="fed00ed1466ca35f085b60612ddfcc4ac009ab5c515b848d19a1e6b3b611b5e4" Jul 17 06:34:20.946716 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:20.946676 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j"] Jul 17 06:34:20.963964 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:20.963925 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-9765j"] Jul 17 06:34:22.296955 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:22.296915 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:34:22.889217 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:22.889179 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" path="/var/lib/kubelet/pods/6bab2d2d-7c06-4d62-9a60-1b92b8e123fe/volumes" Jul 17 06:34:24.055872 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:24.055826 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:24.443676 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:24.443644 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:31.171147 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:31.171104 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:31.368400 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:31.368326 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:31.504471 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:31.504352 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:31.684660 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:31.684618 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:32.073824 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:32.073789 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:32.410177 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:32.410131 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:32.682790 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:32.682682 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:34.346956 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.346918 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx"] Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347233 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347244 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347263 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce66f64-2733-4506-b0e1-c2420cc60fae" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347269 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce66f64-2733-4506-b0e1-c2420cc60fae" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347280 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7dc2aa7-31be-4672-956f-2760b404747f" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347285 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7dc2aa7-31be-4672-956f-2760b404747f" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347292 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347297 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347306 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c3392da-339f-4078-af69-2396558c523c" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347311 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c3392da-339f-4078-af69-2396558c523c" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347318 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19cdc854-30c1-4bb1-a199-2693b4b7c69e" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347323 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19cdc854-30c1-4bb1-a199-2693b4b7c69e" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347329 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c3392da-339f-4078-af69-2396558c523c" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347333 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c3392da-339f-4078-af69-2396558c523c" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347345 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c3392da-339f-4078-af69-2396558c523c" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347350 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c3392da-339f-4078-af69-2396558c523c" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347357 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce66f64-2733-4506-b0e1-c2420cc60fae" containerName="revoke-keys" Jul 17 06:34:34.347356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347362 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce66f64-2733-4506-b0e1-c2420cc60fae" containerName="revoke-keys" Jul 17 06:34:34.348130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347369 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2c4647-7de5-4693-8975-1b1f671b80d5" containerName="maas-api" Jul 17 06:34:34.348130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347374 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2c4647-7de5-4693-8975-1b1f671b80d5" containerName="maas-api" Jul 17 06:34:34.348130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347450 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cce66f64-2733-4506-b0e1-c2420cc60fae" containerName="revoke-keys" Jul 17 06:34:34.348130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347459 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c3392da-339f-4078-af69-2396558c523c" containerName="revoke-keys" Jul 17 06:34:34.348130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347464 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cce66f64-2733-4506-b0e1-c2420cc60fae" containerName="revoke-keys" Jul 17 06:34:34.348130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347470 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" containerName="revoke-keys" Jul 17 06:34:34.348130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347475 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" containerName="revoke-keys" Jul 17 06:34:34.348130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347480 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7dc2aa7-31be-4672-956f-2760b404747f" containerName="revoke-keys" Jul 17 06:34:34.348130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347485 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c3392da-339f-4078-af69-2396558c523c" containerName="revoke-keys" Jul 17 06:34:34.348130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347490 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cce66f64-2733-4506-b0e1-c2420cc60fae" containerName="revoke-keys" Jul 17 06:34:34.348130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347496 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19cdc854-30c1-4bb1-a199-2693b4b7c69e" containerName="revoke-keys" Jul 17 06:34:34.348130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347501 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c3392da-339f-4078-af69-2396558c523c" containerName="revoke-keys" Jul 17 06:34:34.348130 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.347507 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2c4647-7de5-4693-8975-1b1f671b80d5" containerName="maas-api" Jul 17 06:34:34.350615 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.350595 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" Jul 17 06:34:34.353707 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.353681 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:34:34.354092 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.354073 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 06:34:34.360415 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.360371 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx"] Jul 17 06:34:34.453522 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.453484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26966b38-383e-4083-b623-922185ba6070-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-f45zx\" (UID: \"26966b38-383e-4083-b623-922185ba6070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" Jul 17 06:34:34.554827 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.554782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26966b38-383e-4083-b623-922185ba6070-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-f45zx\" (UID: \"26966b38-383e-4083-b623-922185ba6070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" Jul 17 06:34:34.555452 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.555434 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26966b38-383e-4083-b623-922185ba6070-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-f45zx\" (UID: \"26966b38-383e-4083-b623-922185ba6070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" Jul 17 06:34:34.661431 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.661366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" Jul 17 06:34:34.805459 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.805433 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx"] Jul 17 06:34:34.807954 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:34:34.807919 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26966b38_383e_4083_b623_922185ba6070.slice/crio-b09d20f91deef14f7a989cce758e083717e2ab487170290fb1faa468636144a2 WatchSource:0}: Error finding container b09d20f91deef14f7a989cce758e083717e2ab487170290fb1faa468636144a2: Status 404 returned error can't find the container with id b09d20f91deef14f7a989cce758e083717e2ab487170290fb1faa468636144a2 Jul 17 06:34:34.989983 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:34.989892 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" event={"ID":"26966b38-383e-4083-b623-922185ba6070","Type":"ContainerStarted","Data":"b09d20f91deef14f7a989cce758e083717e2ab487170290fb1faa468636144a2"} Jul 17 06:34:35.525136 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.525093 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:35.610574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.610530 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:35.688428 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.688361 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b"] Jul 17 06:34:35.688738 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.688723 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce66f64-2733-4506-b0e1-c2420cc60fae" containerName="revoke-keys" Jul 17 06:34:35.688803 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.688739 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce66f64-2733-4506-b0e1-c2420cc60fae" containerName="revoke-keys" Jul 17 06:34:35.688803 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.688763 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" containerName="revoke-keys" Jul 17 06:34:35.688803 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.688769 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" containerName="revoke-keys" Jul 17 06:34:35.688959 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.688823 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bab2d2d-7c06-4d62-9a60-1b92b8e123fe" containerName="revoke-keys" Jul 17 06:34:35.691884 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.691860 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:34:35.695115 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.695088 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 06:34:35.695271 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.695088 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpsw\"" Jul 17 06:34:35.695841 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.695823 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f90cce-serving-cert\"" Jul 17 06:34:35.717727 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.717693 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b"] Jul 17 06:34:35.748798 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.748760 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:35.867966 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.867917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4183e38e-894c-41d6-b737-84f682339937-maas-api-tls\") pod \"maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b\" (UID: \"4183e38e-894c-41d6-b737-84f682339937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:34:35.868160 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.868047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mjhx\" (UniqueName: \"kubernetes.io/projected/4183e38e-894c-41d6-b737-84f682339937-kube-api-access-5mjhx\") pod \"maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b\" (UID: \"4183e38e-894c-41d6-b737-84f682339937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:34:35.868160 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.868088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183e38e-894c-41d6-b737-84f682339937-openshift-service-ca\") pod \"maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b\" (UID: \"4183e38e-894c-41d6-b737-84f682339937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:34:35.969122 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.969080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4183e38e-894c-41d6-b737-84f682339937-maas-api-tls\") pod \"maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b\" (UID: \"4183e38e-894c-41d6-b737-84f682339937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:34:35.969332 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.969145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mjhx\" (UniqueName: \"kubernetes.io/projected/4183e38e-894c-41d6-b737-84f682339937-kube-api-access-5mjhx\") pod \"maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b\" (UID: \"4183e38e-894c-41d6-b737-84f682339937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:34:35.969332 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.969173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183e38e-894c-41d6-b737-84f682339937-openshift-service-ca\") pod \"maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b\" (UID: \"4183e38e-894c-41d6-b737-84f682339937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:34:35.969921 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.969900 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183e38e-894c-41d6-b737-84f682339937-openshift-service-ca\") pod \"maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b\" (UID: \"4183e38e-894c-41d6-b737-84f682339937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:34:35.971677 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.971657 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4183e38e-894c-41d6-b737-84f682339937-maas-api-tls\") pod \"maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b\" (UID: \"4183e38e-894c-41d6-b737-84f682339937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:34:35.985195 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.985159 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mjhx\" (UniqueName: \"kubernetes.io/projected/4183e38e-894c-41d6-b737-84f682339937-kube-api-access-5mjhx\") pod \"maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b\" (UID: \"4183e38e-894c-41d6-b737-84f682339937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:34:35.996679 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.996640 2580 generic.go:356] "Generic (PLEG): container finished" podID="26966b38-383e-4083-b623-922185ba6070" containerID="c1010bc43c6cafb17b716f9124e4ba6dbaf47188b07214d1db01c2361102816d" exitCode=6 Jul 17 06:34:35.996889 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.996715 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" event={"ID":"26966b38-383e-4083-b623-922185ba6070","Type":"ContainerDied","Data":"c1010bc43c6cafb17b716f9124e4ba6dbaf47188b07214d1db01c2361102816d"} Jul 17 06:34:35.996973 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:35.996961 2580 scope.go:117] "RemoveContainer" containerID="c1010bc43c6cafb17b716f9124e4ba6dbaf47188b07214d1db01c2361102816d" Jul 17 06:34:36.003523 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:36.003501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:34:36.066042 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:36.065053 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:36.151222 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:36.151133 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b"] Jul 17 06:34:36.154551 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:34:36.154523 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4183e38e_894c_41d6_b737_84f682339937.slice/crio-b2ce76fd5270af7007ad030f857f577b33ae35f557c355cbbebf26795a4e70b6 WatchSource:0}: Error finding container b2ce76fd5270af7007ad030f857f577b33ae35f557c355cbbebf26795a4e70b6: Status 404 returned error can't find the container with id b2ce76fd5270af7007ad030f857f577b33ae35f557c355cbbebf26795a4e70b6 Jul 17 06:34:36.156026 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:36.156010 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 06:34:36.623852 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:36.623816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:36.720579 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:36.720527 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:37.002782 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:37.002744 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" event={"ID":"4183e38e-894c-41d6-b737-84f682339937","Type":"ContainerStarted","Data":"ac07fbef17c6290a97ac65356e67b92e1261ce3f2170cbd1e8241ff9397fb1af"} Jul 17 06:34:37.002978 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:37.002807 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" event={"ID":"4183e38e-894c-41d6-b737-84f682339937","Type":"ContainerStarted","Data":"b2ce76fd5270af7007ad030f857f577b33ae35f557c355cbbebf26795a4e70b6"} Jul 17 06:34:37.002978 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:37.002837 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:34:37.004619 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:37.004584 2580 generic.go:356] "Generic (PLEG): container finished" podID="26966b38-383e-4083-b623-922185ba6070" containerID="063bff85618fe8ceaa7299b34c2160db41e96fb6ca64c53b3e7ea7d4e27cfab2" exitCode=6 Jul 17 06:34:37.004736 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:37.004668 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" event={"ID":"26966b38-383e-4083-b623-922185ba6070","Type":"ContainerDied","Data":"063bff85618fe8ceaa7299b34c2160db41e96fb6ca64c53b3e7ea7d4e27cfab2"} Jul 17 06:34:37.004736 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:37.004726 2580 scope.go:117] "RemoveContainer" containerID="c1010bc43c6cafb17b716f9124e4ba6dbaf47188b07214d1db01c2361102816d" Jul 17 06:34:37.004847 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:37.004833 2580 scope.go:117] "RemoveContainer" containerID="063bff85618fe8ceaa7299b34c2160db41e96fb6ca64c53b3e7ea7d4e27cfab2" Jul 17 06:34:37.005054 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:37.005028 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-f45zx_odh-ai-gateway-infra(26966b38-383e-4083-b623-922185ba6070)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" podUID="26966b38-383e-4083-b623-922185ba6070" Jul 17 06:34:37.030967 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:37.030902 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" podStartSLOduration=1.534712561 podStartE2EDuration="2.030879407s" totalImagesPullingTime="496.166846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:34:37.030337794 +0000 UTC m=+3201.775017122" watchObservedRunningTime="2026-07-17 06:34:37.030879407 +0000 UTC m=+3201.775558736" Jul 17 06:34:38.011338 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:38.011303 2580 scope.go:117] "RemoveContainer" containerID="063bff85618fe8ceaa7299b34c2160db41e96fb6ca64c53b3e7ea7d4e27cfab2" Jul 17 06:34:38.011768 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:38.011569 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-f45zx_odh-ai-gateway-infra(26966b38-383e-4083-b623-922185ba6070)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" podUID="26966b38-383e-4083-b623-922185ba6070" Jul 17 06:34:38.899476 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:38.899435 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:34:39.949873 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:39.949834 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:40.436630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:40.436598 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:40.863241 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:40.863213 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:41.347811 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:41.347707 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:42.198429 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:42.198378 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:43.017013 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.016979 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:34:43.648510 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.648476 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:43.757250 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.757207 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p"] Jul 17 06:34:43.762026 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.762004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:34:43.765033 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.765006 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1b3bd1-serving-cert\"" Jul 17 06:34:43.783590 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.783380 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:43.786166 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.783850 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p"] Jul 17 06:34:43.826791 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.826747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/978c0240-3a07-400d-b2ee-247dd5b43bc2-maas-api-tls\") pod \"maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p\" (UID: \"978c0240-3a07-400d-b2ee-247dd5b43bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:34:43.826791 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.826791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/978c0240-3a07-400d-b2ee-247dd5b43bc2-openshift-service-ca\") pod \"maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p\" (UID: \"978c0240-3a07-400d-b2ee-247dd5b43bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:34:43.827017 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.826858 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgm8k\" (UniqueName: \"kubernetes.io/projected/978c0240-3a07-400d-b2ee-247dd5b43bc2-kube-api-access-lgm8k\") pod \"maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p\" (UID: \"978c0240-3a07-400d-b2ee-247dd5b43bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:34:43.891659 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.891618 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:43.928070 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.927966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/978c0240-3a07-400d-b2ee-247dd5b43bc2-maas-api-tls\") pod \"maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p\" (UID: \"978c0240-3a07-400d-b2ee-247dd5b43bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:34:43.928070 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.928009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/978c0240-3a07-400d-b2ee-247dd5b43bc2-openshift-service-ca\") pod \"maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p\" (UID: \"978c0240-3a07-400d-b2ee-247dd5b43bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:34:43.928070 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.928048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgm8k\" (UniqueName: \"kubernetes.io/projected/978c0240-3a07-400d-b2ee-247dd5b43bc2-kube-api-access-lgm8k\") pod \"maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p\" (UID: \"978c0240-3a07-400d-b2ee-247dd5b43bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:34:43.928376 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:43.928143 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-serving-cert: secret "maas-api-e2e-api-b-1b3bd1-serving-cert" not found Jul 17 06:34:43.928376 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:43.928245 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/978c0240-3a07-400d-b2ee-247dd5b43bc2-maas-api-tls podName:978c0240-3a07-400d-b2ee-247dd5b43bc2 nodeName:}" failed. No retries permitted until 2026-07-17 06:34:44.428220154 +0000 UTC m=+3209.172899460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/978c0240-3a07-400d-b2ee-247dd5b43bc2-maas-api-tls") pod "maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" (UID: "978c0240-3a07-400d-b2ee-247dd5b43bc2") : secret "maas-api-e2e-api-b-1b3bd1-serving-cert" not found Jul 17 06:34:43.928778 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.928758 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/978c0240-3a07-400d-b2ee-247dd5b43bc2-openshift-service-ca\") pod \"maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p\" (UID: \"978c0240-3a07-400d-b2ee-247dd5b43bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:34:43.938936 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:43.938902 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgm8k\" (UniqueName: \"kubernetes.io/projected/978c0240-3a07-400d-b2ee-247dd5b43bc2-kube-api-access-lgm8k\") pod \"maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p\" (UID: \"978c0240-3a07-400d-b2ee-247dd5b43bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:34:44.097407 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:44.097348 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:44.432154 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:44.432100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/978c0240-3a07-400d-b2ee-247dd5b43bc2-maas-api-tls\") pod \"maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p\" (UID: \"978c0240-3a07-400d-b2ee-247dd5b43bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:34:44.434826 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:44.434792 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/978c0240-3a07-400d-b2ee-247dd5b43bc2-maas-api-tls\") pod \"maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p\" (UID: \"978c0240-3a07-400d-b2ee-247dd5b43bc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:34:44.624394 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:44.624345 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:44.672223 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:44.672179 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:34:44.731351 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:44.731311 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:44.826760 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:44.826726 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p"] Jul 17 06:34:44.830874 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:34:44.830842 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978c0240_3a07_400d_b2ee_247dd5b43bc2.slice/crio-5be38cc27bf4b6d36de96107a9e1dc15db8d0379f9340f8dc0477f981011d220 WatchSource:0}: Error finding container 5be38cc27bf4b6d36de96107a9e1dc15db8d0379f9340f8dc0477f981011d220: Status 404 returned error can't find the container with id 5be38cc27bf4b6d36de96107a9e1dc15db8d0379f9340f8dc0477f981011d220 Jul 17 06:34:45.046650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:45.046547 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" event={"ID":"978c0240-3a07-400d-b2ee-247dd5b43bc2","Type":"ContainerStarted","Data":"5be38cc27bf4b6d36de96107a9e1dc15db8d0379f9340f8dc0477f981011d220"} Jul 17 06:34:46.053006 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:46.052968 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" event={"ID":"978c0240-3a07-400d-b2ee-247dd5b43bc2","Type":"ContainerStarted","Data":"d2db5f26bd66da89b88e34ebac2aabde1ad72656d4a13f7b8620dd565a61d030"} Jul 17 06:34:46.053377 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:46.053052 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:34:46.075662 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:46.075607 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" podStartSLOduration=2.722958204 podStartE2EDuration="3.075590051s" totalImagesPullingTime="352.631847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:34:46.074028382 +0000 UTC m=+3210.818707711" watchObservedRunningTime="2026-07-17 06:34:46.075590051 +0000 UTC m=+3210.820269378" Jul 17 06:34:46.459908 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:46.459873 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs"] Jul 17 06:34:46.463313 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:46.463291 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" Jul 17 06:34:46.480583 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:46.480532 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs"] Jul 17 06:34:46.548400 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:46.548347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17171f14-8e47-4634-93bb-05cbf1c947e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs\" (UID: \"17171f14-8e47-4634-93bb-05cbf1c947e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" Jul 17 06:34:46.649632 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:46.649588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17171f14-8e47-4634-93bb-05cbf1c947e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs\" (UID: \"17171f14-8e47-4634-93bb-05cbf1c947e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" Jul 17 06:34:46.650310 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:46.650285 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17171f14-8e47-4634-93bb-05cbf1c947e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs\" (UID: \"17171f14-8e47-4634-93bb-05cbf1c947e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" Jul 17 06:34:46.773983 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:46.773887 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" Jul 17 06:34:46.912170 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:46.912142 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs"] Jul 17 06:34:46.914623 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:34:46.914593 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17171f14_8e47_4634_93bb_05cbf1c947e8.slice/crio-71d04766898fa91590f1f6a8343e1202dd2a0e62d54798780ba05b8959eecc02 WatchSource:0}: Error finding container 71d04766898fa91590f1f6a8343e1202dd2a0e62d54798780ba05b8959eecc02: Status 404 returned error can't find the container with id 71d04766898fa91590f1f6a8343e1202dd2a0e62d54798780ba05b8959eecc02 Jul 17 06:34:47.058900 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:47.058805 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" event={"ID":"17171f14-8e47-4634-93bb-05cbf1c947e8","Type":"ContainerStarted","Data":"71d04766898fa91590f1f6a8343e1202dd2a0e62d54798780ba05b8959eecc02"} Jul 17 06:34:48.011521 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:48.011423 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:48.064858 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:48.064817 2580 generic.go:356] "Generic (PLEG): container finished" podID="17171f14-8e47-4634-93bb-05cbf1c947e8" containerID="701fb0ebb68ec04d458c3047aaa108ffaef6e9b855eecd28f966fe23ed1f7bb0" exitCode=6 Jul 17 06:34:48.065262 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:48.064868 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" event={"ID":"17171f14-8e47-4634-93bb-05cbf1c947e8","Type":"ContainerDied","Data":"701fb0ebb68ec04d458c3047aaa108ffaef6e9b855eecd28f966fe23ed1f7bb0"} Jul 17 06:34:48.065262 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:48.065097 2580 scope.go:117] "RemoveContainer" containerID="701fb0ebb68ec04d458c3047aaa108ffaef6e9b855eecd28f966fe23ed1f7bb0" Jul 17 06:34:49.072089 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:49.072049 2580 generic.go:356] "Generic (PLEG): container finished" podID="17171f14-8e47-4634-93bb-05cbf1c947e8" containerID="7d356ed6895948012d863143a18fdf97b7d078695b0120c35dfba99856acb8c1" exitCode=6 Jul 17 06:34:49.072601 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:49.072131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" event={"ID":"17171f14-8e47-4634-93bb-05cbf1c947e8","Type":"ContainerDied","Data":"7d356ed6895948012d863143a18fdf97b7d078695b0120c35dfba99856acb8c1"} Jul 17 06:34:49.072601 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:49.072218 2580 scope.go:117] "RemoveContainer" containerID="701fb0ebb68ec04d458c3047aaa108ffaef6e9b855eecd28f966fe23ed1f7bb0" Jul 17 06:34:49.072601 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:49.072424 2580 scope.go:117] "RemoveContainer" containerID="7d356ed6895948012d863143a18fdf97b7d078695b0120c35dfba99856acb8c1" Jul 17 06:34:49.072757 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:49.072666 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs_odh-ai-gateway-infra(17171f14-8e47-4634-93bb-05cbf1c947e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" podUID="17171f14-8e47-4634-93bb-05cbf1c947e8" Jul 17 06:34:50.078500 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:50.078471 2580 scope.go:117] "RemoveContainer" containerID="7d356ed6895948012d863143a18fdf97b7d078695b0120c35dfba99856acb8c1" Jul 17 06:34:50.078890 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:50.078661 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs_odh-ai-gateway-infra(17171f14-8e47-4634-93bb-05cbf1c947e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" podUID="17171f14-8e47-4634-93bb-05cbf1c947e8" Jul 17 06:34:50.884355 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:50.884317 2580 scope.go:117] "RemoveContainer" containerID="063bff85618fe8ceaa7299b34c2160db41e96fb6ca64c53b3e7ea7d4e27cfab2" Jul 17 06:34:51.085561 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:51.085527 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" event={"ID":"26966b38-383e-4083-b623-922185ba6070","Type":"ContainerStarted","Data":"d1793985501da0b1a8260bc994ab6b4f980429ee33c36aa4bbeda7c55b02c80c"} Jul 17 06:34:51.112093 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:51.112038 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" podStartSLOduration=17.112020487 podStartE2EDuration="17.112020487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:34:51.111210183 +0000 UTC m=+3215.855889510" watchObservedRunningTime="2026-07-17 06:34:51.112020487 +0000 UTC m=+3215.856699812" Jul 17 06:34:52.064364 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:52.064329 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:34:52.093274 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:52.093239 2580 generic.go:356] "Generic (PLEG): container finished" podID="26966b38-383e-4083-b623-922185ba6070" containerID="d1793985501da0b1a8260bc994ab6b4f980429ee33c36aa4bbeda7c55b02c80c" exitCode=6 Jul 17 06:34:52.093777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:52.093314 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" event={"ID":"26966b38-383e-4083-b623-922185ba6070","Type":"ContainerDied","Data":"d1793985501da0b1a8260bc994ab6b4f980429ee33c36aa4bbeda7c55b02c80c"} Jul 17 06:34:52.093777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:52.093411 2580 scope.go:117] "RemoveContainer" containerID="063bff85618fe8ceaa7299b34c2160db41e96fb6ca64c53b3e7ea7d4e27cfab2" Jul 17 06:34:52.093777 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:52.093740 2580 scope.go:117] "RemoveContainer" containerID="d1793985501da0b1a8260bc994ab6b4f980429ee33c36aa4bbeda7c55b02c80c" Jul 17 06:34:52.093989 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:52.093967 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-f45zx_odh-ai-gateway-infra(26966b38-383e-4083-b623-922185ba6070)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" podUID="26966b38-383e-4083-b623-922185ba6070" Jul 17 06:34:52.142769 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:52.142732 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx"] Jul 17 06:34:52.801318 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:52.801282 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb"] Jul 17 06:34:52.805940 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:52.805920 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" Jul 17 06:34:52.828663 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:52.828624 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb"] Jul 17 06:34:52.904953 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:52.904913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3236cdf-8991-47b2-aac9-4d061adfe26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-24csb\" (UID: \"d3236cdf-8991-47b2-aac9-4d061adfe26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" Jul 17 06:34:53.006155 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:53.006109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3236cdf-8991-47b2-aac9-4d061adfe26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-24csb\" (UID: \"d3236cdf-8991-47b2-aac9-4d061adfe26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" Jul 17 06:34:53.006802 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:53.006780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3236cdf-8991-47b2-aac9-4d061adfe26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-24csb\" (UID: \"d3236cdf-8991-47b2-aac9-4d061adfe26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" Jul 17 06:34:53.116221 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:53.116190 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" Jul 17 06:34:53.249643 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:53.249615 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" Jul 17 06:34:53.276958 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:53.276924 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb"] Jul 17 06:34:53.280577 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:34:53.280471 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3236cdf_8991_47b2_aac9_4d061adfe26a.slice/crio-7717d9923575dd14ff381ed8e2bbd62b15275adceb2cda88e2e790d77c797a69 WatchSource:0}: Error finding container 7717d9923575dd14ff381ed8e2bbd62b15275adceb2cda88e2e790d77c797a69: Status 404 returned error can't find the container with id 7717d9923575dd14ff381ed8e2bbd62b15275adceb2cda88e2e790d77c797a69 Jul 17 06:34:53.309259 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:53.309229 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26966b38-383e-4083-b623-922185ba6070-maas-api-service-ca\") pod \"26966b38-383e-4083-b623-922185ba6070\" (UID: \"26966b38-383e-4083-b623-922185ba6070\") " Jul 17 06:34:53.309611 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:53.309587 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26966b38-383e-4083-b623-922185ba6070-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26966b38-383e-4083-b623-922185ba6070" (UID: "26966b38-383e-4083-b623-922185ba6070"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:34:53.410853 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:53.410812 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26966b38-383e-4083-b623-922185ba6070-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:34:54.105610 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:54.105577 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" Jul 17 06:34:54.105810 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:54.105613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx" event={"ID":"26966b38-383e-4083-b623-922185ba6070","Type":"ContainerDied","Data":"b09d20f91deef14f7a989cce758e083717e2ab487170290fb1faa468636144a2"} Jul 17 06:34:54.105810 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:54.105685 2580 scope.go:117] "RemoveContainer" containerID="d1793985501da0b1a8260bc994ab6b4f980429ee33c36aa4bbeda7c55b02c80c" Jul 17 06:34:54.107303 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:54.107279 2580 generic.go:356] "Generic (PLEG): container finished" podID="d3236cdf-8991-47b2-aac9-4d061adfe26a" containerID="04e6bf435a9a3591a38749168ad030af5d0876470d26a44124bfbf097460b353" exitCode=6 Jul 17 06:34:54.107409 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:54.107339 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" event={"ID":"d3236cdf-8991-47b2-aac9-4d061adfe26a","Type":"ContainerDied","Data":"04e6bf435a9a3591a38749168ad030af5d0876470d26a44124bfbf097460b353"} Jul 17 06:34:54.107409 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:54.107396 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" event={"ID":"d3236cdf-8991-47b2-aac9-4d061adfe26a","Type":"ContainerStarted","Data":"7717d9923575dd14ff381ed8e2bbd62b15275adceb2cda88e2e790d77c797a69"} Jul 17 06:34:54.107603 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:54.107591 2580 scope.go:117] "RemoveContainer" containerID="04e6bf435a9a3591a38749168ad030af5d0876470d26a44124bfbf097460b353" Jul 17 06:34:54.157373 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:54.153174 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx"] Jul 17 06:34:54.161580 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:54.161546 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-f45zx"] Jul 17 06:34:54.889954 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:54.889919 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26966b38-383e-4083-b623-922185ba6070" path="/var/lib/kubelet/pods/26966b38-383e-4083-b623-922185ba6070/volumes" Jul 17 06:34:55.114081 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:55.114050 2580 generic.go:356] "Generic (PLEG): container finished" podID="d3236cdf-8991-47b2-aac9-4d061adfe26a" containerID="e965aa1e488be06f0431baf327bda1b7592a4052d1b8a6c1b4fa89e7b3488ae2" exitCode=6 Jul 17 06:34:55.114304 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:55.114133 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" event={"ID":"d3236cdf-8991-47b2-aac9-4d061adfe26a","Type":"ContainerDied","Data":"e965aa1e488be06f0431baf327bda1b7592a4052d1b8a6c1b4fa89e7b3488ae2"} Jul 17 06:34:55.114304 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:55.114190 2580 scope.go:117] "RemoveContainer" containerID="04e6bf435a9a3591a38749168ad030af5d0876470d26a44124bfbf097460b353" Jul 17 06:34:55.114451 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:55.114423 2580 scope.go:117] "RemoveContainer" containerID="e965aa1e488be06f0431baf327bda1b7592a4052d1b8a6c1b4fa89e7b3488ae2" Jul 17 06:34:55.114672 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:55.114652 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-24csb_odh-ai-gateway-infra(d3236cdf-8991-47b2-aac9-4d061adfe26a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" podUID="d3236cdf-8991-47b2-aac9-4d061adfe26a" Jul 17 06:34:56.121761 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:56.121734 2580 scope.go:117] "RemoveContainer" containerID="e965aa1e488be06f0431baf327bda1b7592a4052d1b8a6c1b4fa89e7b3488ae2" Jul 17 06:34:56.122151 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:34:56.121940 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-24csb_odh-ai-gateway-infra(d3236cdf-8991-47b2-aac9-4d061adfe26a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" podUID="d3236cdf-8991-47b2-aac9-4d061adfe26a" Jul 17 06:34:56.920081 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:56.920027 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:57.026435 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:57.026400 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:34:59.016360 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:34:59.016320 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:35:00.391379 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.391342 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v"] Jul 17 06:35:00.391818 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.391772 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26966b38-383e-4083-b623-922185ba6070" containerName="revoke-keys" Jul 17 06:35:00.391818 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.391785 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="26966b38-383e-4083-b623-922185ba6070" containerName="revoke-keys" Jul 17 06:35:00.391818 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.391802 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26966b38-383e-4083-b623-922185ba6070" containerName="revoke-keys" Jul 17 06:35:00.391818 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.391807 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="26966b38-383e-4083-b623-922185ba6070" containerName="revoke-keys" Jul 17 06:35:00.391818 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.391815 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26966b38-383e-4083-b623-922185ba6070" containerName="revoke-keys" Jul 17 06:35:00.391818 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.391820 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="26966b38-383e-4083-b623-922185ba6070" containerName="revoke-keys" Jul 17 06:35:00.392011 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.391869 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="26966b38-383e-4083-b623-922185ba6070" containerName="revoke-keys" Jul 17 06:35:00.392011 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.391881 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="26966b38-383e-4083-b623-922185ba6070" containerName="revoke-keys" Jul 17 06:35:00.394577 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.394558 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" Jul 17 06:35:00.409353 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.409321 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v"] Jul 17 06:35:00.478334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.478288 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb7717d-f22f-4875-9e89-dde686239a75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-ll27v\" (UID: \"3fb7717d-f22f-4875-9e89-dde686239a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" Jul 17 06:35:00.579345 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.579299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb7717d-f22f-4875-9e89-dde686239a75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-ll27v\" (UID: \"3fb7717d-f22f-4875-9e89-dde686239a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" Jul 17 06:35:00.579997 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.579975 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb7717d-f22f-4875-9e89-dde686239a75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-ll27v\" (UID: \"3fb7717d-f22f-4875-9e89-dde686239a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" Jul 17 06:35:00.705041 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.704951 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" Jul 17 06:35:00.845604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:00.845578 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v"] Jul 17 06:35:00.846897 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:35:00.846869 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb7717d_f22f_4875_9e89_dde686239a75.slice/crio-3e27cc998a7545f727d29b949fd22816e1f9c080c3222a1a6b39b2ec96e9aa4a WatchSource:0}: Error finding container 3e27cc998a7545f727d29b949fd22816e1f9c080c3222a1a6b39b2ec96e9aa4a: Status 404 returned error can't find the container with id 3e27cc998a7545f727d29b949fd22816e1f9c080c3222a1a6b39b2ec96e9aa4a Jul 17 06:35:01.145973 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:01.145937 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" event={"ID":"3fb7717d-f22f-4875-9e89-dde686239a75","Type":"ContainerStarted","Data":"3e27cc998a7545f727d29b949fd22816e1f9c080c3222a1a6b39b2ec96e9aa4a"} Jul 17 06:35:01.884136 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:01.884107 2580 scope.go:117] "RemoveContainer" containerID="7d356ed6895948012d863143a18fdf97b7d078695b0120c35dfba99856acb8c1" Jul 17 06:35:02.152585 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:02.152551 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" event={"ID":"17171f14-8e47-4634-93bb-05cbf1c947e8","Type":"ContainerStarted","Data":"be519de3ba8e85900d9b814be6330b7a47db5b94658a7089a85bada07343fd85"} Jul 17 06:35:02.154134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:02.154108 2580 generic.go:356] "Generic (PLEG): container finished" podID="3fb7717d-f22f-4875-9e89-dde686239a75" containerID="760b7dfec204d14b84da297a9f171f316d9ea025626c273a52c9321764dde1d6" exitCode=6 Jul 17 06:35:02.154243 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:02.154152 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" event={"ID":"3fb7717d-f22f-4875-9e89-dde686239a75","Type":"ContainerDied","Data":"760b7dfec204d14b84da297a9f171f316d9ea025626c273a52c9321764dde1d6"} Jul 17 06:35:02.154344 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:02.154332 2580 scope.go:117] "RemoveContainer" containerID="760b7dfec204d14b84da297a9f171f316d9ea025626c273a52c9321764dde1d6" Jul 17 06:35:02.177620 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:02.177568 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" podStartSLOduration=16.177549544 podStartE2EDuration="16.177549544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:35:02.175947873 +0000 UTC m=+3226.920627200" watchObservedRunningTime="2026-07-17 06:35:02.177549544 +0000 UTC m=+3226.922228873" Jul 17 06:35:03.160968 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:03.160931 2580 generic.go:356] "Generic (PLEG): container finished" podID="3fb7717d-f22f-4875-9e89-dde686239a75" containerID="cafd67f499de09f02038f9f9b38640370b033a491951d88becfa9e7b35aab007" exitCode=6 Jul 17 06:35:03.161477 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:03.161008 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" event={"ID":"3fb7717d-f22f-4875-9e89-dde686239a75","Type":"ContainerDied","Data":"cafd67f499de09f02038f9f9b38640370b033a491951d88becfa9e7b35aab007"} Jul 17 06:35:03.161477 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:03.161073 2580 scope.go:117] "RemoveContainer" containerID="760b7dfec204d14b84da297a9f171f316d9ea025626c273a52c9321764dde1d6" Jul 17 06:35:03.161477 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:03.161306 2580 scope.go:117] "RemoveContainer" containerID="cafd67f499de09f02038f9f9b38640370b033a491951d88becfa9e7b35aab007" Jul 17 06:35:03.161617 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:35:03.161575 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-ll27v_odh-ai-gateway-infra(3fb7717d-f22f-4875-9e89-dde686239a75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" podUID="3fb7717d-f22f-4875-9e89-dde686239a75" Jul 17 06:35:03.163026 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:03.163004 2580 generic.go:356] "Generic (PLEG): container finished" podID="17171f14-8e47-4634-93bb-05cbf1c947e8" containerID="be519de3ba8e85900d9b814be6330b7a47db5b94658a7089a85bada07343fd85" exitCode=6 Jul 17 06:35:03.163126 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:03.163080 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" event={"ID":"17171f14-8e47-4634-93bb-05cbf1c947e8","Type":"ContainerDied","Data":"be519de3ba8e85900d9b814be6330b7a47db5b94658a7089a85bada07343fd85"} Jul 17 06:35:03.163358 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:03.163345 2580 scope.go:117] "RemoveContainer" containerID="be519de3ba8e85900d9b814be6330b7a47db5b94658a7089a85bada07343fd85" Jul 17 06:35:03.163611 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:35:03.163593 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs_odh-ai-gateway-infra(17171f14-8e47-4634-93bb-05cbf1c947e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" podUID="17171f14-8e47-4634-93bb-05cbf1c947e8" Jul 17 06:35:03.172719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:03.172701 2580 scope.go:117] "RemoveContainer" containerID="7d356ed6895948012d863143a18fdf97b7d078695b0120c35dfba99856acb8c1" Jul 17 06:35:03.202653 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:03.202619 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs"] Jul 17 06:35:04.170336 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:04.170305 2580 scope.go:117] "RemoveContainer" containerID="cafd67f499de09f02038f9f9b38640370b033a491951d88becfa9e7b35aab007" Jul 17 06:35:04.170800 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:35:04.170572 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-ll27v_odh-ai-gateway-infra(3fb7717d-f22f-4875-9e89-dde686239a75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" podUID="3fb7717d-f22f-4875-9e89-dde686239a75" Jul 17 06:35:04.310787 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:04.310762 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" Jul 17 06:35:04.418249 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:04.418212 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17171f14-8e47-4634-93bb-05cbf1c947e8-maas-api-service-ca\") pod \"17171f14-8e47-4634-93bb-05cbf1c947e8\" (UID: \"17171f14-8e47-4634-93bb-05cbf1c947e8\") " Jul 17 06:35:04.418651 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:04.418622 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17171f14-8e47-4634-93bb-05cbf1c947e8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17171f14-8e47-4634-93bb-05cbf1c947e8" (UID: "17171f14-8e47-4634-93bb-05cbf1c947e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:35:04.519513 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:04.519427 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17171f14-8e47-4634-93bb-05cbf1c947e8-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:35:05.180616 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:05.180522 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" event={"ID":"17171f14-8e47-4634-93bb-05cbf1c947e8","Type":"ContainerDied","Data":"71d04766898fa91590f1f6a8343e1202dd2a0e62d54798780ba05b8959eecc02"} Jul 17 06:35:05.180616 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:05.180556 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs" Jul 17 06:35:05.180616 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:05.180595 2580 scope.go:117] "RemoveContainer" containerID="be519de3ba8e85900d9b814be6330b7a47db5b94658a7089a85bada07343fd85" Jul 17 06:35:05.201346 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:05.201314 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs"] Jul 17 06:35:05.209420 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:05.209376 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-rlfrs"] Jul 17 06:35:06.886619 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:06.886585 2580 scope.go:117] "RemoveContainer" containerID="e965aa1e488be06f0431baf327bda1b7592a4052d1b8a6c1b4fa89e7b3488ae2" Jul 17 06:35:06.889914 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:06.889870 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17171f14-8e47-4634-93bb-05cbf1c947e8" path="/var/lib/kubelet/pods/17171f14-8e47-4634-93bb-05cbf1c947e8/volumes" Jul 17 06:35:07.191723 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:07.191688 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" event={"ID":"d3236cdf-8991-47b2-aac9-4d061adfe26a","Type":"ContainerStarted","Data":"6b7a525a518eedb70d961edf008e89f157e7f99dfdabf025d2e4088f95f03008"} Jul 17 06:35:07.210234 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:07.210171 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" podStartSLOduration=15.210155491 podStartE2EDuration="15.210155491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:35:07.208478468 +0000 UTC m=+3231.953157853" watchObservedRunningTime="2026-07-17 06:35:07.210155491 +0000 UTC m=+3231.954834819" Jul 17 06:35:08.107912 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:08.107868 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:08.198228 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:08.198195 2580 generic.go:356] "Generic (PLEG): container finished" podID="d3236cdf-8991-47b2-aac9-4d061adfe26a" containerID="6b7a525a518eedb70d961edf008e89f157e7f99dfdabf025d2e4088f95f03008" exitCode=6 Jul 17 06:35:08.198446 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:08.198252 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" event={"ID":"d3236cdf-8991-47b2-aac9-4d061adfe26a","Type":"ContainerDied","Data":"6b7a525a518eedb70d961edf008e89f157e7f99dfdabf025d2e4088f95f03008"} Jul 17 06:35:08.198446 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:08.198303 2580 scope.go:117] "RemoveContainer" containerID="e965aa1e488be06f0431baf327bda1b7592a4052d1b8a6c1b4fa89e7b3488ae2" Jul 17 06:35:08.198635 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:08.198616 2580 scope.go:117] "RemoveContainer" containerID="6b7a525a518eedb70d961edf008e89f157e7f99dfdabf025d2e4088f95f03008" Jul 17 06:35:08.198865 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:35:08.198846 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-24csb_odh-ai-gateway-infra(d3236cdf-8991-47b2-aac9-4d061adfe26a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" podUID="d3236cdf-8991-47b2-aac9-4d061adfe26a" Jul 17 06:35:08.212546 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:08.212514 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:08.222358 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:08.222329 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb"] Jul 17 06:35:09.342176 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:09.342150 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" Jul 17 06:35:09.467657 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:09.467551 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3236cdf-8991-47b2-aac9-4d061adfe26a-maas-api-service-ca\") pod \"d3236cdf-8991-47b2-aac9-4d061adfe26a\" (UID: \"d3236cdf-8991-47b2-aac9-4d061adfe26a\") " Jul 17 06:35:09.467992 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:09.467965 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3236cdf-8991-47b2-aac9-4d061adfe26a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3236cdf-8991-47b2-aac9-4d061adfe26a" (UID: "d3236cdf-8991-47b2-aac9-4d061adfe26a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:35:09.569004 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:09.568958 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3236cdf-8991-47b2-aac9-4d061adfe26a-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:35:10.211030 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:10.210995 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" Jul 17 06:35:10.211030 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:10.211008 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb" event={"ID":"d3236cdf-8991-47b2-aac9-4d061adfe26a","Type":"ContainerDied","Data":"7717d9923575dd14ff381ed8e2bbd62b15275adceb2cda88e2e790d77c797a69"} Jul 17 06:35:10.211321 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:10.211074 2580 scope.go:117] "RemoveContainer" containerID="6b7a525a518eedb70d961edf008e89f157e7f99dfdabf025d2e4088f95f03008" Jul 17 06:35:10.233279 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:10.233249 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb"] Jul 17 06:35:10.238116 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:10.238089 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-24csb"] Jul 17 06:35:10.889498 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:10.889458 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3236cdf-8991-47b2-aac9-4d061adfe26a" path="/var/lib/kubelet/pods/d3236cdf-8991-47b2-aac9-4d061adfe26a/volumes" Jul 17 06:35:12.733369 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:12.733329 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b"] Jul 17 06:35:12.733859 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:12.733582 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" podUID="4183e38e-894c-41d6-b737-84f682339937" containerName="maas-api" containerID="cri-o://ac07fbef17c6290a97ac65356e67b92e1261ce3f2170cbd1e8241ff9397fb1af" gracePeriod=30 Jul 17 06:35:12.778214 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:12.778173 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:12.983789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:12.983711 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:35:13.106541 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.106500 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183e38e-894c-41d6-b737-84f682339937-openshift-service-ca\") pod \"4183e38e-894c-41d6-b737-84f682339937\" (UID: \"4183e38e-894c-41d6-b737-84f682339937\") " Jul 17 06:35:13.106709 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.106558 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4183e38e-894c-41d6-b737-84f682339937-maas-api-tls\") pod \"4183e38e-894c-41d6-b737-84f682339937\" (UID: \"4183e38e-894c-41d6-b737-84f682339937\") " Jul 17 06:35:13.106709 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.106597 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mjhx\" (UniqueName: \"kubernetes.io/projected/4183e38e-894c-41d6-b737-84f682339937-kube-api-access-5mjhx\") pod \"4183e38e-894c-41d6-b737-84f682339937\" (UID: \"4183e38e-894c-41d6-b737-84f682339937\") " Jul 17 06:35:13.106950 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.106924 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4183e38e-894c-41d6-b737-84f682339937-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4183e38e-894c-41d6-b737-84f682339937" (UID: "4183e38e-894c-41d6-b737-84f682339937"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:35:13.108846 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.108816 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4183e38e-894c-41d6-b737-84f682339937-kube-api-access-5mjhx" (OuterVolumeSpecName: "kube-api-access-5mjhx") pod "4183e38e-894c-41d6-b737-84f682339937" (UID: "4183e38e-894c-41d6-b737-84f682339937"). InnerVolumeSpecName "kube-api-access-5mjhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:35:13.108970 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.108852 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4183e38e-894c-41d6-b737-84f682339937-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4183e38e-894c-41d6-b737-84f682339937" (UID: "4183e38e-894c-41d6-b737-84f682339937"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:35:13.207442 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.207380 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183e38e-894c-41d6-b737-84f682339937-openshift-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:35:13.207442 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.207435 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4183e38e-894c-41d6-b737-84f682339937-maas-api-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:35:13.207442 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.207447 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mjhx\" (UniqueName: \"kubernetes.io/projected/4183e38e-894c-41d6-b737-84f682339937-kube-api-access-5mjhx\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:35:13.228054 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.228022 2580 generic.go:356] "Generic (PLEG): container finished" podID="4183e38e-894c-41d6-b737-84f682339937" containerID="ac07fbef17c6290a97ac65356e67b92e1261ce3f2170cbd1e8241ff9397fb1af" exitCode=0 Jul 17 06:35:13.228245 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.228074 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" event={"ID":"4183e38e-894c-41d6-b737-84f682339937","Type":"ContainerDied","Data":"ac07fbef17c6290a97ac65356e67b92e1261ce3f2170cbd1e8241ff9397fb1af"} Jul 17 06:35:13.228245 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.228081 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" Jul 17 06:35:13.228245 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.228118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b" event={"ID":"4183e38e-894c-41d6-b737-84f682339937","Type":"ContainerDied","Data":"b2ce76fd5270af7007ad030f857f577b33ae35f557c355cbbebf26795a4e70b6"} Jul 17 06:35:13.228245 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.228141 2580 scope.go:117] "RemoveContainer" containerID="ac07fbef17c6290a97ac65356e67b92e1261ce3f2170cbd1e8241ff9397fb1af" Jul 17 06:35:13.238199 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.238178 2580 scope.go:117] "RemoveContainer" containerID="ac07fbef17c6290a97ac65356e67b92e1261ce3f2170cbd1e8241ff9397fb1af" Jul 17 06:35:13.238531 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:35:13.238511 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac07fbef17c6290a97ac65356e67b92e1261ce3f2170cbd1e8241ff9397fb1af\": container with ID starting with ac07fbef17c6290a97ac65356e67b92e1261ce3f2170cbd1e8241ff9397fb1af not found: ID does not exist" containerID="ac07fbef17c6290a97ac65356e67b92e1261ce3f2170cbd1e8241ff9397fb1af" Jul 17 06:35:13.238614 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.238544 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac07fbef17c6290a97ac65356e67b92e1261ce3f2170cbd1e8241ff9397fb1af"} err="failed to get container status \"ac07fbef17c6290a97ac65356e67b92e1261ce3f2170cbd1e8241ff9397fb1af\": rpc error: code = NotFound desc = could not find container \"ac07fbef17c6290a97ac65356e67b92e1261ce3f2170cbd1e8241ff9397fb1af\": container with ID starting with ac07fbef17c6290a97ac65356e67b92e1261ce3f2170cbd1e8241ff9397fb1af not found: ID does not exist" Jul 17 06:35:13.253730 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.253699 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b"] Jul 17 06:35:13.258950 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.258917 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f90cce-5bdb4f8b7-8mn4b"] Jul 17 06:35:13.338293 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:13.338256 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:14.889986 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:14.889946 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4183e38e-894c-41d6-b737-84f682339937" path="/var/lib/kubelet/pods/4183e38e-894c-41d6-b737-84f682339937/volumes" Jul 17 06:35:16.250354 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:16.250320 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:16.650022 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:16.649981 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:16.964592 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:16.964485 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:35:17.884558 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:17.884522 2580 scope.go:117] "RemoveContainer" containerID="cafd67f499de09f02038f9f9b38640370b033a491951d88becfa9e7b35aab007" Jul 17 06:35:18.255324 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:18.255290 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" event={"ID":"3fb7717d-f22f-4875-9e89-dde686239a75","Type":"ContainerStarted","Data":"27dac4331c6db93a1f22067e9ad5c30eac53ea32d84e7dbd28ad0b23de56e87e"} Jul 17 06:35:18.277815 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:18.277758 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" podStartSLOduration=18.277739618 podStartE2EDuration="18.277739618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:35:18.275639315 +0000 UTC m=+3243.020318647" watchObservedRunningTime="2026-07-17 06:35:18.277739618 +0000 UTC m=+3243.022418945" Jul 17 06:35:19.261268 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:19.261177 2580 generic.go:356] "Generic (PLEG): container finished" podID="3fb7717d-f22f-4875-9e89-dde686239a75" containerID="27dac4331c6db93a1f22067e9ad5c30eac53ea32d84e7dbd28ad0b23de56e87e" exitCode=6 Jul 17 06:35:19.261763 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:19.261257 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" event={"ID":"3fb7717d-f22f-4875-9e89-dde686239a75","Type":"ContainerDied","Data":"27dac4331c6db93a1f22067e9ad5c30eac53ea32d84e7dbd28ad0b23de56e87e"} Jul 17 06:35:19.261763 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:19.261334 2580 scope.go:117] "RemoveContainer" containerID="cafd67f499de09f02038f9f9b38640370b033a491951d88becfa9e7b35aab007" Jul 17 06:35:19.261763 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:19.261709 2580 scope.go:117] "RemoveContainer" containerID="27dac4331c6db93a1f22067e9ad5c30eac53ea32d84e7dbd28ad0b23de56e87e" Jul 17 06:35:19.261929 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:35:19.261911 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-ll27v_odh-ai-gateway-infra(3fb7717d-f22f-4875-9e89-dde686239a75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" podUID="3fb7717d-f22f-4875-9e89-dde686239a75" Jul 17 06:35:19.298510 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:19.298476 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v"] Jul 17 06:35:20.405746 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:20.405721 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" Jul 17 06:35:20.473135 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:20.473107 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb7717d-f22f-4875-9e89-dde686239a75-maas-api-service-ca\") pod \"3fb7717d-f22f-4875-9e89-dde686239a75\" (UID: \"3fb7717d-f22f-4875-9e89-dde686239a75\") " Jul 17 06:35:20.473500 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:20.473476 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb7717d-f22f-4875-9e89-dde686239a75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3fb7717d-f22f-4875-9e89-dde686239a75" (UID: "3fb7717d-f22f-4875-9e89-dde686239a75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:35:20.574077 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:20.573974 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fb7717d-f22f-4875-9e89-dde686239a75-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:35:21.273362 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:21.273277 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" Jul 17 06:35:21.273362 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:21.273305 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v" event={"ID":"3fb7717d-f22f-4875-9e89-dde686239a75","Type":"ContainerDied","Data":"3e27cc998a7545f727d29b949fd22816e1f9c080c3222a1a6b39b2ec96e9aa4a"} Jul 17 06:35:21.273600 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:21.273371 2580 scope.go:117] "RemoveContainer" containerID="27dac4331c6db93a1f22067e9ad5c30eac53ea32d84e7dbd28ad0b23de56e87e" Jul 17 06:35:21.293245 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:21.293203 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v"] Jul 17 06:35:21.302730 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:21.302698 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-ll27v"] Jul 17 06:35:22.889364 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:22.889327 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fb7717d-f22f-4875-9e89-dde686239a75" path="/var/lib/kubelet/pods/3fb7717d-f22f-4875-9e89-dde686239a75/volumes" Jul 17 06:35:23.254991 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.254905 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:23.411376 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.411340 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:23.608617 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.608584 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:23.937604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.937508 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q"] Jul 17 06:35:23.938095 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938003 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3236cdf-8991-47b2-aac9-4d061adfe26a" containerName="revoke-keys" Jul 17 06:35:23.938095 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938024 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3236cdf-8991-47b2-aac9-4d061adfe26a" containerName="revoke-keys" Jul 17 06:35:23.938095 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938039 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4183e38e-894c-41d6-b737-84f682339937" containerName="maas-api" Jul 17 06:35:23.938095 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938047 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4183e38e-894c-41d6-b737-84f682339937" containerName="maas-api" Jul 17 06:35:23.938095 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938063 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17171f14-8e47-4634-93bb-05cbf1c947e8" containerName="revoke-keys" Jul 17 06:35:23.938095 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938071 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="17171f14-8e47-4634-93bb-05cbf1c947e8" containerName="revoke-keys" Jul 17 06:35:23.938095 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938082 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fb7717d-f22f-4875-9e89-dde686239a75" containerName="revoke-keys" Jul 17 06:35:23.938095 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938090 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fb7717d-f22f-4875-9e89-dde686239a75" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938103 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fb7717d-f22f-4875-9e89-dde686239a75" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938111 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fb7717d-f22f-4875-9e89-dde686239a75" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938125 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17171f14-8e47-4634-93bb-05cbf1c947e8" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938134 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="17171f14-8e47-4634-93bb-05cbf1c947e8" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938143 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3236cdf-8991-47b2-aac9-4d061adfe26a" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938151 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3236cdf-8991-47b2-aac9-4d061adfe26a" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938162 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17171f14-8e47-4634-93bb-05cbf1c947e8" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938170 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="17171f14-8e47-4634-93bb-05cbf1c947e8" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938250 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4183e38e-894c-41d6-b737-84f682339937" containerName="maas-api" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938264 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fb7717d-f22f-4875-9e89-dde686239a75" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938275 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fb7717d-f22f-4875-9e89-dde686239a75" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938286 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3236cdf-8991-47b2-aac9-4d061adfe26a" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938295 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="17171f14-8e47-4634-93bb-05cbf1c947e8" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938305 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="17171f14-8e47-4634-93bb-05cbf1c947e8" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938315 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="17171f14-8e47-4634-93bb-05cbf1c947e8" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938324 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3236cdf-8991-47b2-aac9-4d061adfe26a" containerName="revoke-keys" Jul 17 06:35:23.938529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.938331 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="26966b38-383e-4083-b623-922185ba6070" containerName="revoke-keys" Jul 17 06:35:23.943185 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.943160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q" Jul 17 06:35:23.946380 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.946359 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:35:23.965717 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.965682 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q"] Jul 17 06:35:23.973361 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:23.973328 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:24.004547 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:24.004507 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q\" (UID: \"48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q" Jul 17 06:35:24.106020 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:24.105981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q\" (UID: \"48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q" Jul 17 06:35:24.106699 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:24.106679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q\" (UID: \"48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q" Jul 17 06:35:24.253716 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:24.253621 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q" Jul 17 06:35:24.413482 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:24.413447 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q"] Jul 17 06:35:24.417613 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:35:24.417586 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b2bbef_6ea9_4c4f_9b48_41bc8b7a7a76.slice/crio-b58c9f93e876afe508ef5abd22a534dc2c6d33c6b64cd15be5951045b0484336 WatchSource:0}: Error finding container b58c9f93e876afe508ef5abd22a534dc2c6d33c6b64cd15be5951045b0484336: Status 404 returned error can't find the container with id b58c9f93e876afe508ef5abd22a534dc2c6d33c6b64cd15be5951045b0484336 Jul 17 06:35:25.294468 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:25.294419 2580 generic.go:356] "Generic (PLEG): container finished" podID="48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76" containerID="3cd6cd73e20bed9a6eac02ac4520f609c922c5f545896d507071f76a3b51e056" exitCode=0 Jul 17 06:35:25.294869 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:25.294489 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q" event={"ID":"48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76","Type":"ContainerDied","Data":"3cd6cd73e20bed9a6eac02ac4520f609c922c5f545896d507071f76a3b51e056"} Jul 17 06:35:25.294869 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:25.294549 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q" event={"ID":"48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76","Type":"ContainerStarted","Data":"b58c9f93e876afe508ef5abd22a534dc2c6d33c6b64cd15be5951045b0484336"} Jul 17 06:35:26.433077 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:26.433045 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q" Jul 17 06:35:26.527892 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:26.527849 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76-maas-api-service-ca\") pod \"48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76\" (UID: \"48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76\") " Jul 17 06:35:26.528280 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:26.528258 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76" (UID: "48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:35:26.629165 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:26.629127 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:35:27.304702 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:27.304621 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q" Jul 17 06:35:27.304860 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:27.304626 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q" event={"ID":"48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76","Type":"ContainerDied","Data":"b58c9f93e876afe508ef5abd22a534dc2c6d33c6b64cd15be5951045b0484336"} Jul 17 06:35:27.304860 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:27.304745 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b58c9f93e876afe508ef5abd22a534dc2c6d33c6b64cd15be5951045b0484336" Jul 17 06:35:40.194736 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.194692 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:40.297106 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.297065 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:40.462811 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.462721 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p"] Jul 17 06:35:40.462997 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.462970 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" podUID="978c0240-3a07-400d-b2ee-247dd5b43bc2" containerName="maas-api" containerID="cri-o://d2db5f26bd66da89b88e34ebac2aabde1ad72656d4a13f7b8620dd565a61d030" gracePeriod=30 Jul 17 06:35:40.502829 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.502786 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:40.710020 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.709989 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:35:40.766312 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.766209 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/978c0240-3a07-400d-b2ee-247dd5b43bc2-maas-api-tls\") pod \"978c0240-3a07-400d-b2ee-247dd5b43bc2\" (UID: \"978c0240-3a07-400d-b2ee-247dd5b43bc2\") " Jul 17 06:35:40.766312 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.766274 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgm8k\" (UniqueName: \"kubernetes.io/projected/978c0240-3a07-400d-b2ee-247dd5b43bc2-kube-api-access-lgm8k\") pod \"978c0240-3a07-400d-b2ee-247dd5b43bc2\" (UID: \"978c0240-3a07-400d-b2ee-247dd5b43bc2\") " Jul 17 06:35:40.766544 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.766320 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/978c0240-3a07-400d-b2ee-247dd5b43bc2-openshift-service-ca\") pod \"978c0240-3a07-400d-b2ee-247dd5b43bc2\" (UID: \"978c0240-3a07-400d-b2ee-247dd5b43bc2\") " Jul 17 06:35:40.766765 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.766741 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978c0240-3a07-400d-b2ee-247dd5b43bc2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "978c0240-3a07-400d-b2ee-247dd5b43bc2" (UID: "978c0240-3a07-400d-b2ee-247dd5b43bc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:35:40.768481 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.768453 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978c0240-3a07-400d-b2ee-247dd5b43bc2-kube-api-access-lgm8k" (OuterVolumeSpecName: "kube-api-access-lgm8k") pod "978c0240-3a07-400d-b2ee-247dd5b43bc2" (UID: "978c0240-3a07-400d-b2ee-247dd5b43bc2"). InnerVolumeSpecName "kube-api-access-lgm8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:35:40.768481 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.768462 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/978c0240-3a07-400d-b2ee-247dd5b43bc2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "978c0240-3a07-400d-b2ee-247dd5b43bc2" (UID: "978c0240-3a07-400d-b2ee-247dd5b43bc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:35:40.867853 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.867808 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/978c0240-3a07-400d-b2ee-247dd5b43bc2-maas-api-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:35:40.867853 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.867844 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgm8k\" (UniqueName: \"kubernetes.io/projected/978c0240-3a07-400d-b2ee-247dd5b43bc2-kube-api-access-lgm8k\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:35:40.867853 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:40.867853 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/978c0240-3a07-400d-b2ee-247dd5b43bc2-openshift-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:35:41.376885 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:41.376846 2580 generic.go:356] "Generic (PLEG): container finished" podID="978c0240-3a07-400d-b2ee-247dd5b43bc2" containerID="d2db5f26bd66da89b88e34ebac2aabde1ad72656d4a13f7b8620dd565a61d030" exitCode=0 Jul 17 06:35:41.377342 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:41.376902 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" Jul 17 06:35:41.377342 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:41.376935 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" event={"ID":"978c0240-3a07-400d-b2ee-247dd5b43bc2","Type":"ContainerDied","Data":"d2db5f26bd66da89b88e34ebac2aabde1ad72656d4a13f7b8620dd565a61d030"} Jul 17 06:35:41.377342 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:41.376997 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p" event={"ID":"978c0240-3a07-400d-b2ee-247dd5b43bc2","Type":"ContainerDied","Data":"5be38cc27bf4b6d36de96107a9e1dc15db8d0379f9340f8dc0477f981011d220"} Jul 17 06:35:41.377342 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:41.377021 2580 scope.go:117] "RemoveContainer" containerID="d2db5f26bd66da89b88e34ebac2aabde1ad72656d4a13f7b8620dd565a61d030" Jul 17 06:35:41.385633 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:41.385613 2580 scope.go:117] "RemoveContainer" containerID="d2db5f26bd66da89b88e34ebac2aabde1ad72656d4a13f7b8620dd565a61d030" Jul 17 06:35:41.385918 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:35:41.385898 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2db5f26bd66da89b88e34ebac2aabde1ad72656d4a13f7b8620dd565a61d030\": container with ID starting with d2db5f26bd66da89b88e34ebac2aabde1ad72656d4a13f7b8620dd565a61d030 not found: ID does not exist" containerID="d2db5f26bd66da89b88e34ebac2aabde1ad72656d4a13f7b8620dd565a61d030" Jul 17 06:35:41.385964 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:41.385929 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2db5f26bd66da89b88e34ebac2aabde1ad72656d4a13f7b8620dd565a61d030"} err="failed to get container status \"d2db5f26bd66da89b88e34ebac2aabde1ad72656d4a13f7b8620dd565a61d030\": rpc error: code = NotFound desc = could not find container \"d2db5f26bd66da89b88e34ebac2aabde1ad72656d4a13f7b8620dd565a61d030\": container with ID starting with d2db5f26bd66da89b88e34ebac2aabde1ad72656d4a13f7b8620dd565a61d030 not found: ID does not exist" Jul 17 06:35:41.404359 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:41.404327 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p"] Jul 17 06:35:41.409049 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:41.409018 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1b3bd1-7cdf55f4bf-b5s5p"] Jul 17 06:35:42.829154 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.829120 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk"] Jul 17 06:35:42.829569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.829471 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="978c0240-3a07-400d-b2ee-247dd5b43bc2" containerName="maas-api" Jul 17 06:35:42.829569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.829483 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="978c0240-3a07-400d-b2ee-247dd5b43bc2" containerName="maas-api" Jul 17 06:35:42.829569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.829493 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fb7717d-f22f-4875-9e89-dde686239a75" containerName="revoke-keys" Jul 17 06:35:42.829569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.829499 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fb7717d-f22f-4875-9e89-dde686239a75" containerName="revoke-keys" Jul 17 06:35:42.829569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.829511 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3236cdf-8991-47b2-aac9-4d061adfe26a" containerName="revoke-keys" Jul 17 06:35:42.829569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.829516 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3236cdf-8991-47b2-aac9-4d061adfe26a" containerName="revoke-keys" Jul 17 06:35:42.829569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.829523 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76" containerName="revoke-keys" Jul 17 06:35:42.829569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.829529 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76" containerName="revoke-keys" Jul 17 06:35:42.829569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.829574 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3236cdf-8991-47b2-aac9-4d061adfe26a" containerName="revoke-keys" Jul 17 06:35:42.829839 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.829583 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76" containerName="revoke-keys" Jul 17 06:35:42.829839 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.829591 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="978c0240-3a07-400d-b2ee-247dd5b43bc2" containerName="maas-api" Jul 17 06:35:42.829839 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.829599 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fb7717d-f22f-4875-9e89-dde686239a75" containerName="revoke-keys" Jul 17 06:35:42.833749 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.833724 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" Jul 17 06:35:42.836268 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.836228 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 06:35:42.836447 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.836352 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:35:42.842700 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.842676 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk"] Jul 17 06:35:42.895087 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.895047 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="978c0240-3a07-400d-b2ee-247dd5b43bc2" path="/var/lib/kubelet/pods/978c0240-3a07-400d-b2ee-247dd5b43bc2/volumes" Jul 17 06:35:42.985209 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:42.985173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5259826-a8cb-4fb6-b477-aafbaa41eea0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-f99jk\" (UID: \"b5259826-a8cb-4fb6-b477-aafbaa41eea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" Jul 17 06:35:43.086608 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:43.086505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5259826-a8cb-4fb6-b477-aafbaa41eea0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-f99jk\" (UID: \"b5259826-a8cb-4fb6-b477-aafbaa41eea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" Jul 17 06:35:43.087186 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:43.087162 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5259826-a8cb-4fb6-b477-aafbaa41eea0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-f99jk\" (UID: \"b5259826-a8cb-4fb6-b477-aafbaa41eea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" Jul 17 06:35:43.145510 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:43.145473 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" Jul 17 06:35:43.280706 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:43.280674 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk"] Jul 17 06:35:43.283128 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:35:43.283089 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5259826_a8cb_4fb6_b477_aafbaa41eea0.slice/crio-17d4938dad1b48cc8dd8c05eca1a0c0bb45b46d9d62f46d41d7eb3606b74204e WatchSource:0}: Error finding container 17d4938dad1b48cc8dd8c05eca1a0c0bb45b46d9d62f46d41d7eb3606b74204e: Status 404 returned error can't find the container with id 17d4938dad1b48cc8dd8c05eca1a0c0bb45b46d9d62f46d41d7eb3606b74204e Jul 17 06:35:43.387699 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:43.387667 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" event={"ID":"b5259826-a8cb-4fb6-b477-aafbaa41eea0","Type":"ContainerStarted","Data":"17d4938dad1b48cc8dd8c05eca1a0c0bb45b46d9d62f46d41d7eb3606b74204e"} Jul 17 06:35:44.041603 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:44.041571 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:44.394595 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:44.394561 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" containerID="0a41f52a7e5646ffae45ba272558664aaaf55f592d42923c5aa67e438deee41e" exitCode=6 Jul 17 06:35:44.394800 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:44.394651 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" event={"ID":"b5259826-a8cb-4fb6-b477-aafbaa41eea0","Type":"ContainerDied","Data":"0a41f52a7e5646ffae45ba272558664aaaf55f592d42923c5aa67e438deee41e"} Jul 17 06:35:44.394871 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:44.394858 2580 scope.go:117] "RemoveContainer" containerID="0a41f52a7e5646ffae45ba272558664aaaf55f592d42923c5aa67e438deee41e" Jul 17 06:35:44.416356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:44.416326 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:44.489237 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:44.489188 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:35:45.401540 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:45.401507 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" containerID="2c6b6bb2d0b90753a2154c0dcde68fe7ecd3748e69f55566bf264dc5c44662cc" exitCode=6 Jul 17 06:35:45.401980 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:45.401595 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" event={"ID":"b5259826-a8cb-4fb6-b477-aafbaa41eea0","Type":"ContainerDied","Data":"2c6b6bb2d0b90753a2154c0dcde68fe7ecd3748e69f55566bf264dc5c44662cc"} Jul 17 06:35:45.401980 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:45.401654 2580 scope.go:117] "RemoveContainer" containerID="0a41f52a7e5646ffae45ba272558664aaaf55f592d42923c5aa67e438deee41e" Jul 17 06:35:45.401980 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:45.401773 2580 scope.go:117] "RemoveContainer" containerID="2c6b6bb2d0b90753a2154c0dcde68fe7ecd3748e69f55566bf264dc5c44662cc" Jul 17 06:35:45.402111 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:35:45.402056 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-f99jk_odh-ai-gateway-infra(b5259826-a8cb-4fb6-b477-aafbaa41eea0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" podUID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" Jul 17 06:35:46.409083 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:46.409058 2580 scope.go:117] "RemoveContainer" containerID="2c6b6bb2d0b90753a2154c0dcde68fe7ecd3748e69f55566bf264dc5c44662cc" Jul 17 06:35:46.409501 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:35:46.409256 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-f99jk_odh-ai-gateway-infra(b5259826-a8cb-4fb6-b477-aafbaa41eea0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" podUID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" Jul 17 06:35:50.310766 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:50.310729 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:50.419842 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:50.419806 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:50.619796 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:50.619762 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:50.841245 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:50.841211 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:51.166872 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:51.166831 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:51.533937 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:51.533816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:51.686334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:51.686294 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:54.786825 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:54.786787 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:54.939557 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:54.939519 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:54.945090 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:54.945061 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6"] Jul 17 06:35:54.950027 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:54.950003 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:35:54.953165 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:54.953140 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 06:35:54.953292 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:54.953141 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpsw\"" Jul 17 06:35:54.953449 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:54.953433 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-ff4257-serving-cert\"" Jul 17 06:35:55.027973 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.027939 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6"] Jul 17 06:35:55.069345 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.069256 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:55.103423 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.103358 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a426f129-0f66-4f5d-9ee5-40fd5fee9581-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6\" (UID: \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:35:55.103604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.103442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a426f129-0f66-4f5d-9ee5-40fd5fee9581-maas-api-tls\") pod \"maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6\" (UID: \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:35:55.103604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.103534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhkl5\" (UniqueName: \"kubernetes.io/projected/a426f129-0f66-4f5d-9ee5-40fd5fee9581-kube-api-access-hhkl5\") pod \"maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6\" (UID: \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:35:55.204897 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.204861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhkl5\" (UniqueName: \"kubernetes.io/projected/a426f129-0f66-4f5d-9ee5-40fd5fee9581-kube-api-access-hhkl5\") pod \"maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6\" (UID: \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:35:55.205090 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.204911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a426f129-0f66-4f5d-9ee5-40fd5fee9581-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6\" (UID: \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:35:55.205090 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.204982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a426f129-0f66-4f5d-9ee5-40fd5fee9581-maas-api-tls\") pod \"maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6\" (UID: \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:35:55.205225 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:35:55.205087 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-serving-cert: secret "maas-api-e2e-shared-a-ff4257-serving-cert" not found Jul 17 06:35:55.205225 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:35:55.205158 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a426f129-0f66-4f5d-9ee5-40fd5fee9581-maas-api-tls podName:a426f129-0f66-4f5d-9ee5-40fd5fee9581 nodeName:}" failed. No retries permitted until 2026-07-17 06:35:55.705137136 +0000 UTC m=+3280.449816459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a426f129-0f66-4f5d-9ee5-40fd5fee9581-maas-api-tls") pod "maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" (UID: "a426f129-0f66-4f5d-9ee5-40fd5fee9581") : secret "maas-api-e2e-shared-a-ff4257-serving-cert" not found Jul 17 06:35:55.205681 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.205657 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a426f129-0f66-4f5d-9ee5-40fd5fee9581-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6\" (UID: \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:35:55.217050 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.217021 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhkl5\" (UniqueName: \"kubernetes.io/projected/a426f129-0f66-4f5d-9ee5-40fd5fee9581-kube-api-access-hhkl5\") pod \"maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6\" (UID: \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:35:55.430289 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.430247 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:55.581866 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.581825 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:55.681006 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.680907 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:55.710308 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.710263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a426f129-0f66-4f5d-9ee5-40fd5fee9581-maas-api-tls\") pod \"maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6\" (UID: \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:35:55.713946 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.713916 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a426f129-0f66-4f5d-9ee5-40fd5fee9581-maas-api-tls\") pod \"maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6\" (UID: \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:35:55.860306 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:55.860271 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:35:56.016471 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:56.016439 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6"] Jul 17 06:35:56.019182 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:35:56.019154 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda426f129_0f66_4f5d_9ee5_40fd5fee9581.slice/crio-7c6001f3c302978e0b87e47720bc6ff16913c414d259b1d1aafbedb1412461b9 WatchSource:0}: Error finding container 7c6001f3c302978e0b87e47720bc6ff16913c414d259b1d1aafbedb1412461b9: Status 404 returned error can't find the container with id 7c6001f3c302978e0b87e47720bc6ff16913c414d259b1d1aafbedb1412461b9 Jul 17 06:35:56.456406 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:56.456355 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" event={"ID":"a426f129-0f66-4f5d-9ee5-40fd5fee9581","Type":"ContainerStarted","Data":"7c6001f3c302978e0b87e47720bc6ff16913c414d259b1d1aafbedb1412461b9"} Jul 17 06:35:57.462677 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:57.462640 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" event={"ID":"a426f129-0f66-4f5d-9ee5-40fd5fee9581","Type":"ContainerStarted","Data":"d6352b7167894e24023cd38df55ec213e455bc19ae41a0b0641bce3c05290920"} Jul 17 06:35:57.463098 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:57.462737 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:35:57.489105 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:57.489043 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" podStartSLOduration=3.040784776 podStartE2EDuration="3.489027378s" totalImagesPullingTime="448.242602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:35:57.487226383 +0000 UTC m=+3282.231905712" watchObservedRunningTime="2026-07-17 06:35:57.489027378 +0000 UTC m=+3282.233706705" Jul 17 06:35:59.283679 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:59.283647 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:59.622509 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:59.622474 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:35:59.922944 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:35:59.922842 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:36:00.191262 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:00.191172 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:36:00.429961 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:00.429927 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6"] Jul 17 06:36:00.433186 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:00.433160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" Jul 17 06:36:00.441336 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:00.441260 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6"] Jul 17 06:36:00.452516 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:00.452481 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4399a320-b012-4338-87f2-2a1bb16fcf86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-glfx6\" (UID: \"4399a320-b012-4338-87f2-2a1bb16fcf86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" Jul 17 06:36:00.553332 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:00.553297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4399a320-b012-4338-87f2-2a1bb16fcf86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-glfx6\" (UID: \"4399a320-b012-4338-87f2-2a1bb16fcf86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" Jul 17 06:36:00.553988 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:00.553968 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4399a320-b012-4338-87f2-2a1bb16fcf86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-glfx6\" (UID: \"4399a320-b012-4338-87f2-2a1bb16fcf86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" Jul 17 06:36:00.745232 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:00.745132 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" Jul 17 06:36:00.886694 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:36:00.886658 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4399a320_b012_4338_87f2_2a1bb16fcf86.slice/crio-65dd856f496e0b264088f4de4335eb3afac25a8e8242a8d694fd6a2c68d741fe WatchSource:0}: Error finding container 65dd856f496e0b264088f4de4335eb3afac25a8e8242a8d694fd6a2c68d741fe: Status 404 returned error can't find the container with id 65dd856f496e0b264088f4de4335eb3afac25a8e8242a8d694fd6a2c68d741fe Jul 17 06:36:00.890785 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:00.890761 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6"] Jul 17 06:36:01.484144 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:01.484108 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" event={"ID":"4399a320-b012-4338-87f2-2a1bb16fcf86","Type":"ContainerStarted","Data":"161b7a38025e32bb31b4156682821935f85963e9d9a3033654cc9e227a8554a6"} Jul 17 06:36:01.484674 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:01.484173 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" event={"ID":"4399a320-b012-4338-87f2-2a1bb16fcf86","Type":"ContainerStarted","Data":"65dd856f496e0b264088f4de4335eb3afac25a8e8242a8d694fd6a2c68d741fe"} Jul 17 06:36:01.506483 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:01.506423 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" podStartSLOduration=1.5063929900000002 podStartE2EDuration="1.50639299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:36:01.504754006 +0000 UTC m=+3286.249433336" watchObservedRunningTime="2026-07-17 06:36:01.50639299 +0000 UTC m=+3286.251072309" Jul 17 06:36:01.884872 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:01.884832 2580 scope.go:117] "RemoveContainer" containerID="2c6b6bb2d0b90753a2154c0dcde68fe7ecd3748e69f55566bf264dc5c44662cc" Jul 17 06:36:02.491327 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:02.491231 2580 generic.go:356] "Generic (PLEG): container finished" podID="4399a320-b012-4338-87f2-2a1bb16fcf86" containerID="161b7a38025e32bb31b4156682821935f85963e9d9a3033654cc9e227a8554a6" exitCode=6 Jul 17 06:36:02.491776 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:02.491321 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" event={"ID":"4399a320-b012-4338-87f2-2a1bb16fcf86","Type":"ContainerDied","Data":"161b7a38025e32bb31b4156682821935f85963e9d9a3033654cc9e227a8554a6"} Jul 17 06:36:02.491776 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:02.491519 2580 scope.go:117] "RemoveContainer" containerID="161b7a38025e32bb31b4156682821935f85963e9d9a3033654cc9e227a8554a6" Jul 17 06:36:02.493423 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:02.493379 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" event={"ID":"b5259826-a8cb-4fb6-b477-aafbaa41eea0","Type":"ContainerStarted","Data":"18c1c3bd2fd21ecc57290b181093e5f92f1bca2af4bedd1ffc02196b06d4784c"} Jul 17 06:36:02.526811 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:02.526762 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" podStartSLOduration=20.526746765 podStartE2EDuration="20.526746765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:36:02.525837884 +0000 UTC m=+3287.270517212" watchObservedRunningTime="2026-07-17 06:36:02.526746765 +0000 UTC m=+3287.271426083" Jul 17 06:36:02.685495 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:02.685457 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:36:02.914354 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:02.914316 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:36:03.046326 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.046296 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx"] Jul 17 06:36:03.049805 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.049785 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:36:03.055186 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.055161 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-346c23-serving-cert\"" Jul 17 06:36:03.071530 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.071498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87691a64-febc-4700-aec9-e965782925e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx\" (UID: \"87691a64-febc-4700-aec9-e965782925e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:36:03.071703 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.071556 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvps7\" (UniqueName: \"kubernetes.io/projected/87691a64-febc-4700-aec9-e965782925e9-kube-api-access-fvps7\") pod \"maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx\" (UID: \"87691a64-febc-4700-aec9-e965782925e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:36:03.071703 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.071675 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87691a64-febc-4700-aec9-e965782925e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx\" (UID: \"87691a64-febc-4700-aec9-e965782925e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:36:03.072513 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.072489 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx"] Jul 17 06:36:03.169405 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.169285 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:36:03.172406 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.172369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87691a64-febc-4700-aec9-e965782925e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx\" (UID: \"87691a64-febc-4700-aec9-e965782925e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:36:03.172489 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.172430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvps7\" (UniqueName: \"kubernetes.io/projected/87691a64-febc-4700-aec9-e965782925e9-kube-api-access-fvps7\") pod \"maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx\" (UID: \"87691a64-febc-4700-aec9-e965782925e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:36:03.172489 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.172483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87691a64-febc-4700-aec9-e965782925e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx\" (UID: \"87691a64-febc-4700-aec9-e965782925e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:36:03.173112 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.173088 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87691a64-febc-4700-aec9-e965782925e9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx\" (UID: \"87691a64-febc-4700-aec9-e965782925e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:36:03.175095 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.175066 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87691a64-febc-4700-aec9-e965782925e9-maas-api-tls\") pod \"maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx\" (UID: \"87691a64-febc-4700-aec9-e965782925e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:36:03.190793 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.190756 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvps7\" (UniqueName: \"kubernetes.io/projected/87691a64-febc-4700-aec9-e965782925e9-kube-api-access-fvps7\") pod \"maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx\" (UID: \"87691a64-febc-4700-aec9-e965782925e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:36:03.296105 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.296056 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:36:03.422507 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.422412 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:36:03.475189 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.475156 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:36:03.502070 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.502032 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" containerID="18c1c3bd2fd21ecc57290b181093e5f92f1bca2af4bedd1ffc02196b06d4784c" exitCode=6 Jul 17 06:36:03.502574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.502073 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" event={"ID":"b5259826-a8cb-4fb6-b477-aafbaa41eea0","Type":"ContainerDied","Data":"18c1c3bd2fd21ecc57290b181093e5f92f1bca2af4bedd1ffc02196b06d4784c"} Jul 17 06:36:03.502574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.502139 2580 scope.go:117] "RemoveContainer" containerID="2c6b6bb2d0b90753a2154c0dcde68fe7ecd3748e69f55566bf264dc5c44662cc" Jul 17 06:36:03.502574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.502545 2580 scope.go:117] "RemoveContainer" containerID="18c1c3bd2fd21ecc57290b181093e5f92f1bca2af4bedd1ffc02196b06d4784c" Jul 17 06:36:03.502838 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:03.502807 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-f99jk_odh-ai-gateway-infra(b5259826-a8cb-4fb6-b477-aafbaa41eea0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" podUID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" Jul 17 06:36:03.504342 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.504310 2580 generic.go:356] "Generic (PLEG): container finished" podID="4399a320-b012-4338-87f2-2a1bb16fcf86" containerID="ac7406af0056b47a9538c5a397b5a322a0c54534be118c383b182866021acbdf" exitCode=6 Jul 17 06:36:03.504479 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.504415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" event={"ID":"4399a320-b012-4338-87f2-2a1bb16fcf86","Type":"ContainerDied","Data":"ac7406af0056b47a9538c5a397b5a322a0c54534be118c383b182866021acbdf"} Jul 17 06:36:03.504652 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.504630 2580 scope.go:117] "RemoveContainer" containerID="ac7406af0056b47a9538c5a397b5a322a0c54534be118c383b182866021acbdf" Jul 17 06:36:03.504880 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:03.504857 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-glfx6_odh-ai-gateway-infra(4399a320-b012-4338-87f2-2a1bb16fcf86)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" podUID="4399a320-b012-4338-87f2-2a1bb16fcf86" Jul 17 06:36:03.517421 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.517369 2580 scope.go:117] "RemoveContainer" containerID="161b7a38025e32bb31b4156682821935f85963e9d9a3033654cc9e227a8554a6" Jul 17 06:36:03.538205 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.538172 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk"] Jul 17 06:36:03.593544 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.593467 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx"] Jul 17 06:36:03.595854 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:36:03.595827 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87691a64_febc_4700_aec9_e965782925e9.slice/crio-785feadf2b156108839a2a471efabb2412dea9906afa05040e09ded9bbafe10c WatchSource:0}: Error finding container 785feadf2b156108839a2a471efabb2412dea9906afa05040e09ded9bbafe10c: Status 404 returned error can't find the container with id 785feadf2b156108839a2a471efabb2412dea9906afa05040e09ded9bbafe10c Jul 17 06:36:03.911155 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:03.911104 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:36:04.007521 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:04.007481 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:36:04.514281 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:04.514244 2580 scope.go:117] "RemoveContainer" containerID="ac7406af0056b47a9538c5a397b5a322a0c54534be118c383b182866021acbdf" Jul 17 06:36:04.514780 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:04.514532 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-glfx6_odh-ai-gateway-infra(4399a320-b012-4338-87f2-2a1bb16fcf86)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" podUID="4399a320-b012-4338-87f2-2a1bb16fcf86" Jul 17 06:36:04.517307 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:04.517274 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" event={"ID":"87691a64-febc-4700-aec9-e965782925e9","Type":"ContainerStarted","Data":"fe3bc23836a0eceea3800e660b145525582c1f555069163d3957bb81b4f1d982"} Jul 17 06:36:04.517479 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:04.517340 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" event={"ID":"87691a64-febc-4700-aec9-e965782925e9","Type":"ContainerStarted","Data":"785feadf2b156108839a2a471efabb2412dea9906afa05040e09ded9bbafe10c"} Jul 17 06:36:04.517479 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:04.517448 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:36:04.550893 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:04.550835 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" podStartSLOduration=1.11340762 podStartE2EDuration="1.550812478s" totalImagesPullingTime="437.404858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:36:04.548955559 +0000 UTC m=+3289.293634887" watchObservedRunningTime="2026-07-17 06:36:04.550812478 +0000 UTC m=+3289.295491804" Jul 17 06:36:04.653899 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:04.653873 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" Jul 17 06:36:04.682246 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:04.682213 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5259826-a8cb-4fb6-b477-aafbaa41eea0-maas-api-service-ca\") pod \"b5259826-a8cb-4fb6-b477-aafbaa41eea0\" (UID: \"b5259826-a8cb-4fb6-b477-aafbaa41eea0\") " Jul 17 06:36:04.682616 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:04.682593 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5259826-a8cb-4fb6-b477-aafbaa41eea0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b5259826-a8cb-4fb6-b477-aafbaa41eea0" (UID: "b5259826-a8cb-4fb6-b477-aafbaa41eea0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:36:04.782879 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:04.782781 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5259826-a8cb-4fb6-b477-aafbaa41eea0-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:36:05.523767 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:05.523727 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" Jul 17 06:36:05.524257 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:05.523775 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk" event={"ID":"b5259826-a8cb-4fb6-b477-aafbaa41eea0","Type":"ContainerDied","Data":"17d4938dad1b48cc8dd8c05eca1a0c0bb45b46d9d62f46d41d7eb3606b74204e"} Jul 17 06:36:05.524257 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:05.523841 2580 scope.go:117] "RemoveContainer" containerID="18c1c3bd2fd21ecc57290b181093e5f92f1bca2af4bedd1ffc02196b06d4784c" Jul 17 06:36:05.542979 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:05.542946 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk"] Jul 17 06:36:05.558950 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:05.558912 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f99jk"] Jul 17 06:36:06.891345 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:06.891304 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" path="/var/lib/kubelet/pods/b5259826-a8cb-4fb6-b477-aafbaa41eea0/volumes" Jul 17 06:36:07.372565 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:07.372519 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:36:10.529669 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:10.529639 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:36:15.883899 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:15.883860 2580 scope.go:117] "RemoveContainer" containerID="ac7406af0056b47a9538c5a397b5a322a0c54534be118c383b182866021acbdf" Jul 17 06:36:16.496466 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:16.496429 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:36:16.585161 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:16.585127 2580 generic.go:356] "Generic (PLEG): container finished" podID="4399a320-b012-4338-87f2-2a1bb16fcf86" containerID="064746b9a85678285d4ed2ae7a9754cf9fae9a0f6a735469d1cc4c3b1d56b9f7" exitCode=6 Jul 17 06:36:16.585335 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:16.585195 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" event={"ID":"4399a320-b012-4338-87f2-2a1bb16fcf86","Type":"ContainerDied","Data":"064746b9a85678285d4ed2ae7a9754cf9fae9a0f6a735469d1cc4c3b1d56b9f7"} Jul 17 06:36:16.585335 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:16.585260 2580 scope.go:117] "RemoveContainer" containerID="ac7406af0056b47a9538c5a397b5a322a0c54534be118c383b182866021acbdf" Jul 17 06:36:16.585643 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:16.585624 2580 scope.go:117] "RemoveContainer" containerID="064746b9a85678285d4ed2ae7a9754cf9fae9a0f6a735469d1cc4c3b1d56b9f7" Jul 17 06:36:16.585871 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:16.585847 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-glfx6_odh-ai-gateway-infra(4399a320-b012-4338-87f2-2a1bb16fcf86)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" podUID="4399a320-b012-4338-87f2-2a1bb16fcf86" Jul 17 06:36:16.652927 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:16.652886 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:36:16.760904 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:16.760730 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:36:17.622266 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:17.622219 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6"] Jul 17 06:36:17.767131 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:17.767101 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" Jul 17 06:36:17.789195 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:17.789159 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4399a320-b012-4338-87f2-2a1bb16fcf86-maas-api-service-ca\") pod \"4399a320-b012-4338-87f2-2a1bb16fcf86\" (UID: \"4399a320-b012-4338-87f2-2a1bb16fcf86\") " Jul 17 06:36:17.789602 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:17.789570 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4399a320-b012-4338-87f2-2a1bb16fcf86-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4399a320-b012-4338-87f2-2a1bb16fcf86" (UID: "4399a320-b012-4338-87f2-2a1bb16fcf86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:36:17.890159 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:17.890061 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4399a320-b012-4338-87f2-2a1bb16fcf86-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:36:18.598958 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:18.598921 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" event={"ID":"4399a320-b012-4338-87f2-2a1bb16fcf86","Type":"ContainerDied","Data":"65dd856f496e0b264088f4de4335eb3afac25a8e8242a8d694fd6a2c68d741fe"} Jul 17 06:36:18.598958 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:18.598944 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6" Jul 17 06:36:18.599204 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:18.598991 2580 scope.go:117] "RemoveContainer" containerID="064746b9a85678285d4ed2ae7a9754cf9fae9a0f6a735469d1cc4c3b1d56b9f7" Jul 17 06:36:18.646413 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:18.646363 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6"] Jul 17 06:36:18.660108 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:18.660067 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-glfx6"] Jul 17 06:36:18.684743 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:18.684704 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:36:18.889640 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:18.889606 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4399a320-b012-4338-87f2-2a1bb16fcf86" path="/var/lib/kubelet/pods/4399a320-b012-4338-87f2-2a1bb16fcf86/volumes" Jul 17 06:36:24.454208 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.454167 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k"] Jul 17 06:36:24.454694 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.454593 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" containerName="revoke-keys" Jul 17 06:36:24.454694 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.454607 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" containerName="revoke-keys" Jul 17 06:36:24.454694 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.454622 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4399a320-b012-4338-87f2-2a1bb16fcf86" containerName="revoke-keys" Jul 17 06:36:24.454694 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.454628 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4399a320-b012-4338-87f2-2a1bb16fcf86" containerName="revoke-keys" Jul 17 06:36:24.454694 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.454636 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4399a320-b012-4338-87f2-2a1bb16fcf86" containerName="revoke-keys" Jul 17 06:36:24.454694 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.454641 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4399a320-b012-4338-87f2-2a1bb16fcf86" containerName="revoke-keys" Jul 17 06:36:24.454694 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.454659 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" containerName="revoke-keys" Jul 17 06:36:24.454694 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.454665 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" containerName="revoke-keys" Jul 17 06:36:24.454945 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.454713 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" containerName="revoke-keys" Jul 17 06:36:24.454945 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.454723 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4399a320-b012-4338-87f2-2a1bb16fcf86" containerName="revoke-keys" Jul 17 06:36:24.454945 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.454731 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" containerName="revoke-keys" Jul 17 06:36:24.454945 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.454737 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4399a320-b012-4338-87f2-2a1bb16fcf86" containerName="revoke-keys" Jul 17 06:36:24.459098 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.459080 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" Jul 17 06:36:24.461446 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.461423 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:36:24.466448 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.466424 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k"] Jul 17 06:36:24.553057 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.553012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698358e2-d936-4c42-8d7c-7b779355a133-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k\" (UID: \"698358e2-d936-4c42-8d7c-7b779355a133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" Jul 17 06:36:24.654372 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.654331 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698358e2-d936-4c42-8d7c-7b779355a133-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k\" (UID: \"698358e2-d936-4c42-8d7c-7b779355a133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" Jul 17 06:36:24.655052 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.655029 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698358e2-d936-4c42-8d7c-7b779355a133-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k\" (UID: \"698358e2-d936-4c42-8d7c-7b779355a133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" Jul 17 06:36:24.770633 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.770537 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" Jul 17 06:36:24.912314 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.912277 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k"] Jul 17 06:36:24.915633 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:36:24.915592 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698358e2_d936_4c42_8d7c_7b779355a133.slice/crio-7d555ad21500619f08ca82bccc5f8259b102992a6f50502c3c461d68b592cdef WatchSource:0}: Error finding container 7d555ad21500619f08ca82bccc5f8259b102992a6f50502c3c461d68b592cdef: Status 404 returned error can't find the container with id 7d555ad21500619f08ca82bccc5f8259b102992a6f50502c3c461d68b592cdef Jul 17 06:36:24.957958 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:24.957790 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:36:25.059239 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:25.059142 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:36:25.635019 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:25.634980 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" event={"ID":"698358e2-d936-4c42-8d7c-7b779355a133","Type":"ContainerStarted","Data":"d079b6248edf0d0ce527c62e956fa3ce96fc8006bc534300587f104df368bdce"} Jul 17 06:36:25.635464 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:25.635050 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" event={"ID":"698358e2-d936-4c42-8d7c-7b779355a133","Type":"ContainerStarted","Data":"7d555ad21500619f08ca82bccc5f8259b102992a6f50502c3c461d68b592cdef"} Jul 17 06:36:25.653364 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:25.653318 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" podStartSLOduration=1.653300567 podStartE2EDuration="1.653300567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:36:25.65288284 +0000 UTC m=+3310.397562169" watchObservedRunningTime="2026-07-17 06:36:25.653300567 +0000 UTC m=+3310.397979895" Jul 17 06:36:26.640784 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:26.640747 2580 generic.go:356] "Generic (PLEG): container finished" podID="698358e2-d936-4c42-8d7c-7b779355a133" containerID="d079b6248edf0d0ce527c62e956fa3ce96fc8006bc534300587f104df368bdce" exitCode=6 Jul 17 06:36:26.641192 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:26.640832 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" event={"ID":"698358e2-d936-4c42-8d7c-7b779355a133","Type":"ContainerDied","Data":"d079b6248edf0d0ce527c62e956fa3ce96fc8006bc534300587f104df368bdce"} Jul 17 06:36:26.641192 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:26.641014 2580 scope.go:117] "RemoveContainer" containerID="d079b6248edf0d0ce527c62e956fa3ce96fc8006bc534300587f104df368bdce" Jul 17 06:36:27.647754 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:27.647707 2580 generic.go:356] "Generic (PLEG): container finished" podID="698358e2-d936-4c42-8d7c-7b779355a133" containerID="2e29a9051664a8ced9b34daccbdcedee4499fe568e531d94598aa296f8b4bec6" exitCode=6 Jul 17 06:36:27.648256 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:27.647790 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" event={"ID":"698358e2-d936-4c42-8d7c-7b779355a133","Type":"ContainerDied","Data":"2e29a9051664a8ced9b34daccbdcedee4499fe568e531d94598aa296f8b4bec6"} Jul 17 06:36:27.648256 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:27.647849 2580 scope.go:117] "RemoveContainer" containerID="d079b6248edf0d0ce527c62e956fa3ce96fc8006bc534300587f104df368bdce" Jul 17 06:36:27.648256 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:27.648024 2580 scope.go:117] "RemoveContainer" containerID="2e29a9051664a8ced9b34daccbdcedee4499fe568e531d94598aa296f8b4bec6" Jul 17 06:36:27.648442 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:27.648306 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k_odh-ai-gateway-infra(698358e2-d936-4c42-8d7c-7b779355a133)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" podUID="698358e2-d936-4c42-8d7c-7b779355a133" Jul 17 06:36:28.655161 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:28.655133 2580 scope.go:117] "RemoveContainer" containerID="2e29a9051664a8ced9b34daccbdcedee4499fe568e531d94598aa296f8b4bec6" Jul 17 06:36:28.655588 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:28.655334 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k_odh-ai-gateway-infra(698358e2-d936-4c42-8d7c-7b779355a133)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" podUID="698358e2-d936-4c42-8d7c-7b779355a133" Jul 17 06:36:30.488134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:30.488092 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:36:36.088265 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:36.088200 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:36:36.560959 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:36.560909 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65"] Jul 17 06:36:36.561418 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:36.561373 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" containerName="revoke-keys" Jul 17 06:36:36.561418 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:36.561420 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" containerName="revoke-keys" Jul 17 06:36:36.561595 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:36.561452 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4399a320-b012-4338-87f2-2a1bb16fcf86" containerName="revoke-keys" Jul 17 06:36:36.561595 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:36.561460 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4399a320-b012-4338-87f2-2a1bb16fcf86" containerName="revoke-keys" Jul 17 06:36:36.561595 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:36.561551 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5259826-a8cb-4fb6-b477-aafbaa41eea0" containerName="revoke-keys" Jul 17 06:36:36.561595 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:36.561567 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4399a320-b012-4338-87f2-2a1bb16fcf86" containerName="revoke-keys" Jul 17 06:36:36.564935 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:36.564910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" Jul 17 06:36:36.575572 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:36.575541 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65"] Jul 17 06:36:36.662074 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:36.662030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6185e4de-cd5f-44ec-89f6-0ba67c016eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-2xm65\" (UID: \"6185e4de-cd5f-44ec-89f6-0ba67c016eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" Jul 17 06:36:36.763226 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:36.763182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6185e4de-cd5f-44ec-89f6-0ba67c016eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-2xm65\" (UID: \"6185e4de-cd5f-44ec-89f6-0ba67c016eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" Jul 17 06:36:36.763938 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:36.763918 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6185e4de-cd5f-44ec-89f6-0ba67c016eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-2xm65\" (UID: \"6185e4de-cd5f-44ec-89f6-0ba67c016eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" Jul 17 06:36:36.875996 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:36.875962 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" Jul 17 06:36:37.013105 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:37.013069 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65"] Jul 17 06:36:37.016224 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:36:37.016192 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6185e4de_cd5f_44ec_89f6_0ba67c016eee.slice/crio-554a9b14decd422039b395302e72fa17cb2b1aca6172ab881300d07b9694568b WatchSource:0}: Error finding container 554a9b14decd422039b395302e72fa17cb2b1aca6172ab881300d07b9694568b: Status 404 returned error can't find the container with id 554a9b14decd422039b395302e72fa17cb2b1aca6172ab881300d07b9694568b Jul 17 06:36:37.699451 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:37.699419 2580 generic.go:356] "Generic (PLEG): container finished" podID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" containerID="07f1834ffc552ae74d1fbc35e18792423af1a5edbaea16afaa7195accfa7570f" exitCode=6 Jul 17 06:36:37.699832 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:37.699515 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" event={"ID":"6185e4de-cd5f-44ec-89f6-0ba67c016eee","Type":"ContainerDied","Data":"07f1834ffc552ae74d1fbc35e18792423af1a5edbaea16afaa7195accfa7570f"} Jul 17 06:36:37.699832 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:37.699568 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" event={"ID":"6185e4de-cd5f-44ec-89f6-0ba67c016eee","Type":"ContainerStarted","Data":"554a9b14decd422039b395302e72fa17cb2b1aca6172ab881300d07b9694568b"} Jul 17 06:36:37.699832 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:37.699665 2580 scope.go:117] "RemoveContainer" containerID="07f1834ffc552ae74d1fbc35e18792423af1a5edbaea16afaa7195accfa7570f" Jul 17 06:36:38.705893 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:38.705846 2580 generic.go:356] "Generic (PLEG): container finished" podID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" containerID="08084f314265cee99041f2fe73e63d95bd0dcc7d48a24117e715ba9833b4d63e" exitCode=6 Jul 17 06:36:38.706370 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:38.705936 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" event={"ID":"6185e4de-cd5f-44ec-89f6-0ba67c016eee","Type":"ContainerDied","Data":"08084f314265cee99041f2fe73e63d95bd0dcc7d48a24117e715ba9833b4d63e"} Jul 17 06:36:38.706370 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:38.705994 2580 scope.go:117] "RemoveContainer" containerID="07f1834ffc552ae74d1fbc35e18792423af1a5edbaea16afaa7195accfa7570f" Jul 17 06:36:38.706370 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:38.706244 2580 scope.go:117] "RemoveContainer" containerID="08084f314265cee99041f2fe73e63d95bd0dcc7d48a24117e715ba9833b4d63e" Jul 17 06:36:38.706583 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:38.706486 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-2xm65_odh-ai-gateway-infra(6185e4de-cd5f-44ec-89f6-0ba67c016eee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" podUID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" Jul 17 06:36:39.712810 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:39.712776 2580 scope.go:117] "RemoveContainer" containerID="08084f314265cee99041f2fe73e63d95bd0dcc7d48a24117e715ba9833b4d63e" Jul 17 06:36:39.713207 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:39.713013 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-2xm65_odh-ai-gateway-infra(6185e4de-cd5f-44ec-89f6-0ba67c016eee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" podUID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" Jul 17 06:36:41.884757 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:41.884716 2580 scope.go:117] "RemoveContainer" containerID="2e29a9051664a8ced9b34daccbdcedee4499fe568e531d94598aa296f8b4bec6" Jul 17 06:36:42.087140 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:42.087101 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:36:42.728402 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:42.728296 2580 generic.go:356] "Generic (PLEG): container finished" podID="698358e2-d936-4c42-8d7c-7b779355a133" containerID="275a5c8045e1957d4bbc4b756b9051d4ce63b577daf51b2112601b3d66792136" exitCode=6 Jul 17 06:36:42.728402 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:42.728369 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" event={"ID":"698358e2-d936-4c42-8d7c-7b779355a133","Type":"ContainerDied","Data":"275a5c8045e1957d4bbc4b756b9051d4ce63b577daf51b2112601b3d66792136"} Jul 17 06:36:42.728596 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:42.728448 2580 scope.go:117] "RemoveContainer" containerID="2e29a9051664a8ced9b34daccbdcedee4499fe568e531d94598aa296f8b4bec6" Jul 17 06:36:42.728755 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:42.728738 2580 scope.go:117] "RemoveContainer" containerID="275a5c8045e1957d4bbc4b756b9051d4ce63b577daf51b2112601b3d66792136" Jul 17 06:36:42.728977 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:42.728939 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k_odh-ai-gateway-infra(698358e2-d936-4c42-8d7c-7b779355a133)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" podUID="698358e2-d936-4c42-8d7c-7b779355a133" Jul 17 06:36:43.307539 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:43.307499 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw"] Jul 17 06:36:43.312138 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:43.312118 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" Jul 17 06:36:43.320202 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:43.320176 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw"] Jul 17 06:36:43.424996 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:43.424960 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-r4flw\" (UID: \"6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" Jul 17 06:36:43.526413 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:43.526344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-r4flw\" (UID: \"6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" Jul 17 06:36:43.526988 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:43.526968 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-r4flw\" (UID: \"6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" Jul 17 06:36:43.625484 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:43.625444 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" Jul 17 06:36:43.758058 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:43.758021 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k"] Jul 17 06:36:43.771032 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:43.770987 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw"] Jul 17 06:36:43.777060 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:36:43.774883 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a91d5e2_62f3_4e0f_9778_3ec2a5d30e5a.slice/crio-9ae94805d55b62c1c6e4b973351474bd344faa933ad9aa5e2854b3798a2d7a9d WatchSource:0}: Error finding container 9ae94805d55b62c1c6e4b973351474bd344faa933ad9aa5e2854b3798a2d7a9d: Status 404 returned error can't find the container with id 9ae94805d55b62c1c6e4b973351474bd344faa933ad9aa5e2854b3798a2d7a9d Jul 17 06:36:43.897433 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:43.897396 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" Jul 17 06:36:44.032158 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:44.032123 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698358e2-d936-4c42-8d7c-7b779355a133-maas-api-service-ca\") pod \"698358e2-d936-4c42-8d7c-7b779355a133\" (UID: \"698358e2-d936-4c42-8d7c-7b779355a133\") " Jul 17 06:36:44.032505 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:44.032484 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698358e2-d936-4c42-8d7c-7b779355a133-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "698358e2-d936-4c42-8d7c-7b779355a133" (UID: "698358e2-d936-4c42-8d7c-7b779355a133"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:36:44.133845 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:44.133748 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/698358e2-d936-4c42-8d7c-7b779355a133-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:36:44.743106 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:44.743077 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" Jul 17 06:36:44.743590 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:44.743106 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k" event={"ID":"698358e2-d936-4c42-8d7c-7b779355a133","Type":"ContainerDied","Data":"7d555ad21500619f08ca82bccc5f8259b102992a6f50502c3c461d68b592cdef"} Jul 17 06:36:44.743590 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:44.743174 2580 scope.go:117] "RemoveContainer" containerID="275a5c8045e1957d4bbc4b756b9051d4ce63b577daf51b2112601b3d66792136" Jul 17 06:36:44.744784 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:44.744756 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" containerID="f78479fad72088c08094ddade1f975ea38294182023c26deab5977bfde24458e" exitCode=6 Jul 17 06:36:44.744905 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:44.744836 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" event={"ID":"6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a","Type":"ContainerDied","Data":"f78479fad72088c08094ddade1f975ea38294182023c26deab5977bfde24458e"} Jul 17 06:36:44.744905 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:44.744874 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" event={"ID":"6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a","Type":"ContainerStarted","Data":"9ae94805d55b62c1c6e4b973351474bd344faa933ad9aa5e2854b3798a2d7a9d"} Jul 17 06:36:44.745066 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:44.745049 2580 scope.go:117] "RemoveContainer" containerID="f78479fad72088c08094ddade1f975ea38294182023c26deab5977bfde24458e" Jul 17 06:36:44.774257 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:44.774224 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k"] Jul 17 06:36:44.779940 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:44.779911 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7mc2k"] Jul 17 06:36:44.890110 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:44.890027 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="698358e2-d936-4c42-8d7c-7b779355a133" path="/var/lib/kubelet/pods/698358e2-d936-4c42-8d7c-7b779355a133/volumes" Jul 17 06:36:45.238840 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.238741 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v76zf"] Jul 17 06:36:45.239233 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.239211 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="698358e2-d936-4c42-8d7c-7b779355a133" containerName="revoke-keys" Jul 17 06:36:45.239339 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.239236 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="698358e2-d936-4c42-8d7c-7b779355a133" containerName="revoke-keys" Jul 17 06:36:45.239339 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.239272 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="698358e2-d936-4c42-8d7c-7b779355a133" containerName="revoke-keys" Jul 17 06:36:45.239339 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.239281 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="698358e2-d936-4c42-8d7c-7b779355a133" containerName="revoke-keys" Jul 17 06:36:45.239339 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.239295 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="698358e2-d936-4c42-8d7c-7b779355a133" containerName="revoke-keys" Jul 17 06:36:45.239339 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.239305 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="698358e2-d936-4c42-8d7c-7b779355a133" containerName="revoke-keys" Jul 17 06:36:45.239607 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.239420 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="698358e2-d936-4c42-8d7c-7b779355a133" containerName="revoke-keys" Jul 17 06:36:45.239607 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.239434 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="698358e2-d936-4c42-8d7c-7b779355a133" containerName="revoke-keys" Jul 17 06:36:45.239607 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.239448 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="698358e2-d936-4c42-8d7c-7b779355a133" containerName="revoke-keys" Jul 17 06:36:45.242667 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.242643 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v76zf" Jul 17 06:36:45.244908 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.244883 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-vw86h\"" Jul 17 06:36:45.249831 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.249764 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v76zf"] Jul 17 06:36:45.343981 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.343941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdtk2\" (UniqueName: \"kubernetes.io/projected/39b0320b-96d8-4a59-b140-6936ca10f03b-kube-api-access-qdtk2\") pod \"kuadrant-operator-catalog-v76zf\" (UID: \"39b0320b-96d8-4a59-b140-6936ca10f03b\") " pod="kuadrant-system/kuadrant-operator-catalog-v76zf" Jul 17 06:36:45.445481 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.445449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtk2\" (UniqueName: \"kubernetes.io/projected/39b0320b-96d8-4a59-b140-6936ca10f03b-kube-api-access-qdtk2\") pod \"kuadrant-operator-catalog-v76zf\" (UID: \"39b0320b-96d8-4a59-b140-6936ca10f03b\") " pod="kuadrant-system/kuadrant-operator-catalog-v76zf" Jul 17 06:36:45.453397 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.453364 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdtk2\" (UniqueName: \"kubernetes.io/projected/39b0320b-96d8-4a59-b140-6936ca10f03b-kube-api-access-qdtk2\") pod \"kuadrant-operator-catalog-v76zf\" (UID: \"39b0320b-96d8-4a59-b140-6936ca10f03b\") " pod="kuadrant-system/kuadrant-operator-catalog-v76zf" Jul 17 06:36:45.553283 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.553199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v76zf" Jul 17 06:36:45.689687 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.689662 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v76zf"] Jul 17 06:36:45.691707 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:36:45.691672 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b0320b_96d8_4a59_b140_6936ca10f03b.slice/crio-d1ccdf4854299ef15b311996963e02e8ec9f8e05ebbf4c549513691852854fa9 WatchSource:0}: Error finding container d1ccdf4854299ef15b311996963e02e8ec9f8e05ebbf4c549513691852854fa9: Status 404 returned error can't find the container with id d1ccdf4854299ef15b311996963e02e8ec9f8e05ebbf4c549513691852854fa9 Jul 17 06:36:45.752293 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.752256 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" containerID="ea8e40d0e2a05109eee633783b2f285b773f7685b2f8b0a356c83a4cec34c93f" exitCode=6 Jul 17 06:36:45.752747 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.752340 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" event={"ID":"6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a","Type":"ContainerDied","Data":"ea8e40d0e2a05109eee633783b2f285b773f7685b2f8b0a356c83a4cec34c93f"} Jul 17 06:36:45.752747 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.752427 2580 scope.go:117] "RemoveContainer" containerID="f78479fad72088c08094ddade1f975ea38294182023c26deab5977bfde24458e" Jul 17 06:36:45.752747 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.752670 2580 scope.go:117] "RemoveContainer" containerID="ea8e40d0e2a05109eee633783b2f285b773f7685b2f8b0a356c83a4cec34c93f" Jul 17 06:36:45.752979 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:45.752953 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-r4flw_odh-ai-gateway-infra(6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" podUID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" Jul 17 06:36:45.753715 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:45.753671 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v76zf" event={"ID":"39b0320b-96d8-4a59-b140-6936ca10f03b","Type":"ContainerStarted","Data":"d1ccdf4854299ef15b311996963e02e8ec9f8e05ebbf4c549513691852854fa9"} Jul 17 06:36:46.763074 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:46.763041 2580 scope.go:117] "RemoveContainer" containerID="ea8e40d0e2a05109eee633783b2f285b773f7685b2f8b0a356c83a4cec34c93f" Jul 17 06:36:46.763568 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:46.763299 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-r4flw_odh-ai-gateway-infra(6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" podUID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" Jul 17 06:36:48.774729 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:48.774688 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v76zf" event={"ID":"39b0320b-96d8-4a59-b140-6936ca10f03b","Type":"ContainerStarted","Data":"4f55ac9f5b27603bc12290eb2839018f8dd185e15b04720cd235ca587cbbc185"} Jul 17 06:36:48.792296 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:48.792239 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-v76zf" podStartSLOduration=1.757498765 podStartE2EDuration="3.792222716s" totalImagesPullingTime="2.034723951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:36:48.790828593 +0000 UTC m=+3333.535507932" watchObservedRunningTime="2026-07-17 06:36:48.792222716 +0000 UTC m=+3333.536902044" Jul 17 06:36:50.475760 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:50.475713 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9"] Jul 17 06:36:50.479446 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:50.479424 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" Jul 17 06:36:50.489931 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:50.489901 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9"] Jul 17 06:36:50.593597 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:50.593546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eecf6c3-a776-484b-ae54-b1f9d99b36f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9\" (UID: \"8eecf6c3-a776-484b-ae54-b1f9d99b36f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" Jul 17 06:36:50.694460 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:50.694411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eecf6c3-a776-484b-ae54-b1f9d99b36f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9\" (UID: \"8eecf6c3-a776-484b-ae54-b1f9d99b36f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" Jul 17 06:36:50.695068 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:50.695047 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eecf6c3-a776-484b-ae54-b1f9d99b36f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9\" (UID: \"8eecf6c3-a776-484b-ae54-b1f9d99b36f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" Jul 17 06:36:50.791271 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:50.791179 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" Jul 17 06:36:50.884328 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:50.884150 2580 scope.go:117] "RemoveContainer" containerID="08084f314265cee99041f2fe73e63d95bd0dcc7d48a24117e715ba9833b4d63e" Jul 17 06:36:50.944534 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:50.939202 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9"] Jul 17 06:36:51.791128 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:51.791028 2580 generic.go:356] "Generic (PLEG): container finished" podID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" containerID="12737c339a41eb08c6c3c0aab953be3ff362f2fec08feadd23513dda70cae5b2" exitCode=6 Jul 17 06:36:51.791590 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:51.791118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" event={"ID":"8eecf6c3-a776-484b-ae54-b1f9d99b36f1","Type":"ContainerDied","Data":"12737c339a41eb08c6c3c0aab953be3ff362f2fec08feadd23513dda70cae5b2"} Jul 17 06:36:51.791590 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:51.791179 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" event={"ID":"8eecf6c3-a776-484b-ae54-b1f9d99b36f1","Type":"ContainerStarted","Data":"c7f8878c0347d1bbfee01c8805c66229be67da0f79180df0ed33ad65c211c5be"} Jul 17 06:36:51.791590 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:51.791318 2580 scope.go:117] "RemoveContainer" containerID="12737c339a41eb08c6c3c0aab953be3ff362f2fec08feadd23513dda70cae5b2" Jul 17 06:36:51.793159 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:51.793134 2580 generic.go:356] "Generic (PLEG): container finished" podID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" containerID="1ea4977c771eb27d389f1bdbbd33271a808d0d66272cfa53049894ce2bb86eae" exitCode=6 Jul 17 06:36:51.793281 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:51.793208 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" event={"ID":"6185e4de-cd5f-44ec-89f6-0ba67c016eee","Type":"ContainerDied","Data":"1ea4977c771eb27d389f1bdbbd33271a808d0d66272cfa53049894ce2bb86eae"} Jul 17 06:36:51.793281 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:51.793264 2580 scope.go:117] "RemoveContainer" containerID="08084f314265cee99041f2fe73e63d95bd0dcc7d48a24117e715ba9833b4d63e" Jul 17 06:36:51.793630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:51.793617 2580 scope.go:117] "RemoveContainer" containerID="1ea4977c771eb27d389f1bdbbd33271a808d0d66272cfa53049894ce2bb86eae" Jul 17 06:36:51.793860 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:51.793839 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-2xm65_odh-ai-gateway-infra(6185e4de-cd5f-44ec-89f6-0ba67c016eee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" podUID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" Jul 17 06:36:52.802042 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:52.801947 2580 generic.go:356] "Generic (PLEG): container finished" podID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" containerID="d353c3587efcbd57cd144d9fade2646e40e06182685a729d812d7d43054180b2" exitCode=6 Jul 17 06:36:52.802042 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:52.802000 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" event={"ID":"8eecf6c3-a776-484b-ae54-b1f9d99b36f1","Type":"ContainerDied","Data":"d353c3587efcbd57cd144d9fade2646e40e06182685a729d812d7d43054180b2"} Jul 17 06:36:52.802549 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:52.802058 2580 scope.go:117] "RemoveContainer" containerID="12737c339a41eb08c6c3c0aab953be3ff362f2fec08feadd23513dda70cae5b2" Jul 17 06:36:52.802549 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:52.802247 2580 scope.go:117] "RemoveContainer" containerID="d353c3587efcbd57cd144d9fade2646e40e06182685a729d812d7d43054180b2" Jul 17 06:36:52.802549 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:52.802485 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9_odh-ai-gateway-infra(8eecf6c3-a776-484b-ae54-b1f9d99b36f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" podUID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" Jul 17 06:36:52.833187 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:52.833146 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65"] Jul 17 06:36:52.973707 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:52.973676 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" Jul 17 06:36:53.121187 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:53.121149 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6185e4de-cd5f-44ec-89f6-0ba67c016eee-maas-api-service-ca\") pod \"6185e4de-cd5f-44ec-89f6-0ba67c016eee\" (UID: \"6185e4de-cd5f-44ec-89f6-0ba67c016eee\") " Jul 17 06:36:53.121557 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:53.121533 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6185e4de-cd5f-44ec-89f6-0ba67c016eee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6185e4de-cd5f-44ec-89f6-0ba67c016eee" (UID: "6185e4de-cd5f-44ec-89f6-0ba67c016eee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:36:53.222435 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:53.222375 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6185e4de-cd5f-44ec-89f6-0ba67c016eee-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:36:53.809943 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:53.809905 2580 scope.go:117] "RemoveContainer" containerID="d353c3587efcbd57cd144d9fade2646e40e06182685a729d812d7d43054180b2" Jul 17 06:36:53.810431 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:53.810144 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9_odh-ai-gateway-infra(8eecf6c3-a776-484b-ae54-b1f9d99b36f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" podUID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" Jul 17 06:36:53.811092 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:53.811069 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" event={"ID":"6185e4de-cd5f-44ec-89f6-0ba67c016eee","Type":"ContainerDied","Data":"554a9b14decd422039b395302e72fa17cb2b1aca6172ab881300d07b9694568b"} Jul 17 06:36:53.811184 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:53.811123 2580 scope.go:117] "RemoveContainer" containerID="1ea4977c771eb27d389f1bdbbd33271a808d0d66272cfa53049894ce2bb86eae" Jul 17 06:36:53.811184 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:53.811078 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65" Jul 17 06:36:53.835933 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:53.835898 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65"] Jul 17 06:36:53.842291 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:53.842262 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-2xm65"] Jul 17 06:36:54.889655 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:54.889620 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" path="/var/lib/kubelet/pods/6185e4de-cd5f-44ec-89f6-0ba67c016eee/volumes" Jul 17 06:36:55.553289 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:55.553251 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-v76zf" Jul 17 06:36:55.553644 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:55.553313 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-v76zf" Jul 17 06:36:55.576903 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:55.576871 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-v76zf" Jul 17 06:36:55.849862 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:55.849782 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-v76zf" Jul 17 06:36:55.886928 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:55.886892 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v76zf"] Jul 17 06:36:55.986025 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:55.985977 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:36:57.835650 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:57.835608 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-v76zf" podUID="39b0320b-96d8-4a59-b140-6936ca10f03b" containerName="registry-server" containerID="cri-o://4f55ac9f5b27603bc12290eb2839018f8dd185e15b04720cd235ca587cbbc185" gracePeriod=2 Jul 17 06:36:58.097266 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.097192 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v76zf" Jul 17 06:36:58.166516 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.166477 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdtk2\" (UniqueName: \"kubernetes.io/projected/39b0320b-96d8-4a59-b140-6936ca10f03b-kube-api-access-qdtk2\") pod \"39b0320b-96d8-4a59-b140-6936ca10f03b\" (UID: \"39b0320b-96d8-4a59-b140-6936ca10f03b\") " Jul 17 06:36:58.168658 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.168634 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b0320b-96d8-4a59-b140-6936ca10f03b-kube-api-access-qdtk2" (OuterVolumeSpecName: "kube-api-access-qdtk2") pod "39b0320b-96d8-4a59-b140-6936ca10f03b" (UID: "39b0320b-96d8-4a59-b140-6936ca10f03b"). InnerVolumeSpecName "kube-api-access-qdtk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:36:58.267682 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.267640 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdtk2\" (UniqueName: \"kubernetes.io/projected/39b0320b-96d8-4a59-b140-6936ca10f03b-kube-api-access-qdtk2\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:36:58.841612 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.841572 2580 generic.go:356] "Generic (PLEG): container finished" podID="39b0320b-96d8-4a59-b140-6936ca10f03b" containerID="4f55ac9f5b27603bc12290eb2839018f8dd185e15b04720cd235ca587cbbc185" exitCode=0 Jul 17 06:36:58.842026 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.841621 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v76zf" Jul 17 06:36:58.842026 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.841658 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v76zf" event={"ID":"39b0320b-96d8-4a59-b140-6936ca10f03b","Type":"ContainerDied","Data":"4f55ac9f5b27603bc12290eb2839018f8dd185e15b04720cd235ca587cbbc185"} Jul 17 06:36:58.842026 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.841709 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v76zf" event={"ID":"39b0320b-96d8-4a59-b140-6936ca10f03b","Type":"ContainerDied","Data":"d1ccdf4854299ef15b311996963e02e8ec9f8e05ebbf4c549513691852854fa9"} Jul 17 06:36:58.842026 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.841733 2580 scope.go:117] "RemoveContainer" containerID="4f55ac9f5b27603bc12290eb2839018f8dd185e15b04720cd235ca587cbbc185" Jul 17 06:36:58.851183 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.851163 2580 scope.go:117] "RemoveContainer" containerID="4f55ac9f5b27603bc12290eb2839018f8dd185e15b04720cd235ca587cbbc185" Jul 17 06:36:58.851497 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:58.851477 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f55ac9f5b27603bc12290eb2839018f8dd185e15b04720cd235ca587cbbc185\": container with ID starting with 4f55ac9f5b27603bc12290eb2839018f8dd185e15b04720cd235ca587cbbc185 not found: ID does not exist" containerID="4f55ac9f5b27603bc12290eb2839018f8dd185e15b04720cd235ca587cbbc185" Jul 17 06:36:58.851550 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.851508 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f55ac9f5b27603bc12290eb2839018f8dd185e15b04720cd235ca587cbbc185"} err="failed to get container status \"4f55ac9f5b27603bc12290eb2839018f8dd185e15b04720cd235ca587cbbc185\": rpc error: code = NotFound desc = could not find container \"4f55ac9f5b27603bc12290eb2839018f8dd185e15b04720cd235ca587cbbc185\": container with ID starting with 4f55ac9f5b27603bc12290eb2839018f8dd185e15b04720cd235ca587cbbc185 not found: ID does not exist" Jul 17 06:36:58.865233 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.865194 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v76zf"] Jul 17 06:36:58.873328 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.873295 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v76zf"] Jul 17 06:36:58.884629 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.884604 2580 scope.go:117] "RemoveContainer" containerID="ea8e40d0e2a05109eee633783b2f285b773f7685b2f8b0a356c83a4cec34c93f" Jul 17 06:36:58.890060 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:58.890027 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39b0320b-96d8-4a59-b140-6936ca10f03b" path="/var/lib/kubelet/pods/39b0320b-96d8-4a59-b140-6936ca10f03b/volumes" Jul 17 06:36:59.849501 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:59.849400 2580 generic.go:356] "Generic (PLEG): container finished" podID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" containerID="4724ff9420292f9f80723ed896372fc9b7f76ed2becaeea088db00879d955fb4" exitCode=6 Jul 17 06:36:59.849501 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:59.849425 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" event={"ID":"6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a","Type":"ContainerDied","Data":"4724ff9420292f9f80723ed896372fc9b7f76ed2becaeea088db00879d955fb4"} Jul 17 06:36:59.849501 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:59.849495 2580 scope.go:117] "RemoveContainer" containerID="ea8e40d0e2a05109eee633783b2f285b773f7685b2f8b0a356c83a4cec34c93f" Jul 17 06:36:59.850056 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:36:59.849803 2580 scope.go:117] "RemoveContainer" containerID="4724ff9420292f9f80723ed896372fc9b7f76ed2becaeea088db00879d955fb4" Jul 17 06:36:59.850115 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:36:59.850067 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-r4flw_odh-ai-gateway-infra(6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" podUID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" Jul 17 06:37:00.881225 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:00.881187 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw"] Jul 17 06:37:01.016596 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:01.016570 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" Jul 17 06:37:01.088585 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:01.088552 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a-maas-api-service-ca\") pod \"6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a\" (UID: \"6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a\") " Jul 17 06:37:01.088956 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:01.088931 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" (UID: "6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:37:01.189318 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:01.189206 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:37:01.590559 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:01.590446 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:37:01.868754 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:01.868709 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" event={"ID":"6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a","Type":"ContainerDied","Data":"9ae94805d55b62c1c6e4b973351474bd344faa933ad9aa5e2854b3798a2d7a9d"} Jul 17 06:37:01.868945 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:01.868806 2580 scope.go:117] "RemoveContainer" containerID="4724ff9420292f9f80723ed896372fc9b7f76ed2becaeea088db00879d955fb4" Jul 17 06:37:01.868945 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:01.868723 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw" Jul 17 06:37:01.891659 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:01.891626 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw"] Jul 17 06:37:01.895330 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:01.895305 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-r4flw"] Jul 17 06:37:02.890096 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:02.890062 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" path="/var/lib/kubelet/pods/6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a/volumes" Jul 17 06:37:06.888217 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:06.888187 2580 scope.go:117] "RemoveContainer" containerID="d353c3587efcbd57cd144d9fade2646e40e06182685a729d812d7d43054180b2" Jul 17 06:37:07.902339 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:07.902303 2580 generic.go:356] "Generic (PLEG): container finished" podID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" containerID="fe4fa1e573b9205d5ba994b350c145afbebda2e2a11b34756d9a9c6ba96243e3" exitCode=6 Jul 17 06:37:07.902776 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:07.902352 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" event={"ID":"8eecf6c3-a776-484b-ae54-b1f9d99b36f1","Type":"ContainerDied","Data":"fe4fa1e573b9205d5ba994b350c145afbebda2e2a11b34756d9a9c6ba96243e3"} Jul 17 06:37:07.902776 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:07.902417 2580 scope.go:117] "RemoveContainer" containerID="d353c3587efcbd57cd144d9fade2646e40e06182685a729d812d7d43054180b2" Jul 17 06:37:07.902776 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:07.902715 2580 scope.go:117] "RemoveContainer" containerID="fe4fa1e573b9205d5ba994b350c145afbebda2e2a11b34756d9a9c6ba96243e3" Jul 17 06:37:07.902931 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:37:07.902911 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9_odh-ai-gateway-infra(8eecf6c3-a776-484b-ae54-b1f9d99b36f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" podUID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" Jul 17 06:37:08.931073 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:08.931032 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9"] Jul 17 06:37:09.069100 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:09.069066 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" Jul 17 06:37:09.163657 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:09.163621 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eecf6c3-a776-484b-ae54-b1f9d99b36f1-maas-api-service-ca\") pod \"8eecf6c3-a776-484b-ae54-b1f9d99b36f1\" (UID: \"8eecf6c3-a776-484b-ae54-b1f9d99b36f1\") " Jul 17 06:37:09.164049 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:09.164022 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eecf6c3-a776-484b-ae54-b1f9d99b36f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8eecf6c3-a776-484b-ae54-b1f9d99b36f1" (UID: "8eecf6c3-a776-484b-ae54-b1f9d99b36f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:37:09.264837 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:09.264746 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eecf6c3-a776-484b-ae54-b1f9d99b36f1-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:37:09.914879 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:09.914847 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" Jul 17 06:37:09.915090 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:09.914881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9" event={"ID":"8eecf6c3-a776-484b-ae54-b1f9d99b36f1","Type":"ContainerDied","Data":"c7f8878c0347d1bbfee01c8805c66229be67da0f79180df0ed33ad65c211c5be"} Jul 17 06:37:09.915090 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:09.914944 2580 scope.go:117] "RemoveContainer" containerID="fe4fa1e573b9205d5ba994b350c145afbebda2e2a11b34756d9a9c6ba96243e3" Jul 17 06:37:09.936555 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:09.936519 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9"] Jul 17 06:37:09.940440 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:09.940412 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-hh7w9"] Jul 17 06:37:10.889589 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:10.889554 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" path="/var/lib/kubelet/pods/8eecf6c3-a776-484b-ae54-b1f9d99b36f1/volumes" Jul 17 06:37:15.578561 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:15.578516 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:37:24.498719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.498677 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm"] Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499057 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" containerName="revoke-keys" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499069 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" containerName="revoke-keys" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499078 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b0320b-96d8-4a59-b140-6936ca10f03b" containerName="registry-server" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499084 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b0320b-96d8-4a59-b140-6936ca10f03b" containerName="registry-server" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499094 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" containerName="revoke-keys" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499101 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" containerName="revoke-keys" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499108 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" containerName="revoke-keys" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499114 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" containerName="revoke-keys" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499125 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" containerName="revoke-keys" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499130 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" containerName="revoke-keys" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499137 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" containerName="revoke-keys" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499142 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" containerName="revoke-keys" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499148 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" containerName="revoke-keys" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499153 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" containerName="revoke-keys" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499159 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" containerName="revoke-keys" Jul 17 06:37:24.499197 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499165 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" containerName="revoke-keys" Jul 17 06:37:24.499856 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499236 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" containerName="revoke-keys" Jul 17 06:37:24.499856 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499246 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" containerName="revoke-keys" Jul 17 06:37:24.499856 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499253 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" containerName="revoke-keys" Jul 17 06:37:24.499856 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499258 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b0320b-96d8-4a59-b140-6936ca10f03b" containerName="registry-server" Jul 17 06:37:24.499856 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499264 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" containerName="revoke-keys" Jul 17 06:37:24.499856 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499269 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" containerName="revoke-keys" Jul 17 06:37:24.499856 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499275 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" containerName="revoke-keys" Jul 17 06:37:24.499856 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.499282 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" containerName="revoke-keys" Jul 17 06:37:24.504853 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.504832 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" Jul 17 06:37:24.507330 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.507302 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:37:24.508734 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.508702 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm"] Jul 17 06:37:24.600261 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.600218 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b37db6-38f3-498c-b372-8da065ef709f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm\" (UID: \"60b37db6-38f3-498c-b372-8da065ef709f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" Jul 17 06:37:24.701187 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.701141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b37db6-38f3-498c-b372-8da065ef709f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm\" (UID: \"60b37db6-38f3-498c-b372-8da065ef709f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" Jul 17 06:37:24.701824 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.701802 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b37db6-38f3-498c-b372-8da065ef709f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm\" (UID: \"60b37db6-38f3-498c-b372-8da065ef709f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" Jul 17 06:37:24.817670 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.817566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" Jul 17 06:37:24.954860 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.954832 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm"] Jul 17 06:37:24.957670 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:37:24.957637 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b37db6_38f3_498c_b372_8da065ef709f.slice/crio-8a4e959ea7f94c20198829b9295a771971c664aadb2a0878958770ef1de69833 WatchSource:0}: Error finding container 8a4e959ea7f94c20198829b9295a771971c664aadb2a0878958770ef1de69833: Status 404 returned error can't find the container with id 8a4e959ea7f94c20198829b9295a771971c664aadb2a0878958770ef1de69833 Jul 17 06:37:24.986315 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:24.986286 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" event={"ID":"60b37db6-38f3-498c-b372-8da065ef709f","Type":"ContainerStarted","Data":"8a4e959ea7f94c20198829b9295a771971c664aadb2a0878958770ef1de69833"} Jul 17 06:37:25.993056 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:25.992957 2580 generic.go:356] "Generic (PLEG): container finished" podID="60b37db6-38f3-498c-b372-8da065ef709f" containerID="f838d41397e4ce1a2c87628d01bfdad8499f630d86da13fcd31129fe3cdc81ca" exitCode=6 Jul 17 06:37:25.993056 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:25.993038 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" event={"ID":"60b37db6-38f3-498c-b372-8da065ef709f","Type":"ContainerDied","Data":"f838d41397e4ce1a2c87628d01bfdad8499f630d86da13fcd31129fe3cdc81ca"} Jul 17 06:37:25.993495 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:25.993222 2580 scope.go:117] "RemoveContainer" containerID="f838d41397e4ce1a2c87628d01bfdad8499f630d86da13fcd31129fe3cdc81ca" Jul 17 06:37:26.598809 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.598774 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb"] Jul 17 06:37:26.599143 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.599131 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" containerName="revoke-keys" Jul 17 06:37:26.599185 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.599145 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eecf6c3-a776-484b-ae54-b1f9d99b36f1" containerName="revoke-keys" Jul 17 06:37:26.599185 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.599162 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" containerName="revoke-keys" Jul 17 06:37:26.599185 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.599167 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" containerName="revoke-keys" Jul 17 06:37:26.599279 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.599236 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a91d5e2-62f3-4e0f-9778-3ec2a5d30e5a" containerName="revoke-keys" Jul 17 06:37:26.599279 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.599246 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6185e4de-cd5f-44ec-89f6-0ba67c016eee" containerName="revoke-keys" Jul 17 06:37:26.602201 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.602178 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" Jul 17 06:37:26.609604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.609575 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb"] Jul 17 06:37:26.719417 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.719355 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb\" (UID: \"44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" Jul 17 06:37:26.820129 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.820007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb\" (UID: \"44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" Jul 17 06:37:26.820724 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.820705 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb\" (UID: \"44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" Jul 17 06:37:26.913908 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.913867 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" Jul 17 06:37:26.999886 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.999829 2580 generic.go:356] "Generic (PLEG): container finished" podID="60b37db6-38f3-498c-b372-8da065ef709f" containerID="d9cd080d5e6e88498e4056df02098c6b15f38c32c1e027398436906fa8e538be" exitCode=6 Jul 17 06:37:27.000919 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:26.999966 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" event={"ID":"60b37db6-38f3-498c-b372-8da065ef709f","Type":"ContainerDied","Data":"d9cd080d5e6e88498e4056df02098c6b15f38c32c1e027398436906fa8e538be"} Jul 17 06:37:27.000919 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:27.000034 2580 scope.go:117] "RemoveContainer" containerID="f838d41397e4ce1a2c87628d01bfdad8499f630d86da13fcd31129fe3cdc81ca" Jul 17 06:37:27.000919 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:27.000467 2580 scope.go:117] "RemoveContainer" containerID="d9cd080d5e6e88498e4056df02098c6b15f38c32c1e027398436906fa8e538be" Jul 17 06:37:27.000919 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:37:27.000730 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm_odh-ai-gateway-infra(60b37db6-38f3-498c-b372-8da065ef709f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" podUID="60b37db6-38f3-498c-b372-8da065ef709f" Jul 17 06:37:27.055674 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:27.055616 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb"] Jul 17 06:37:27.063812 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:37:27.063776 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b5b2a0_cfe5_4fa8_ab01_fafcd00d53b4.slice/crio-f92b65bf425bd44433a3e383a57abef4b83f732e6cd2409b2d90323dbace5fc4 WatchSource:0}: Error finding container f92b65bf425bd44433a3e383a57abef4b83f732e6cd2409b2d90323dbace5fc4: Status 404 returned error can't find the container with id f92b65bf425bd44433a3e383a57abef4b83f732e6cd2409b2d90323dbace5fc4 Jul 17 06:37:28.007174 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:28.007075 2580 scope.go:117] "RemoveContainer" containerID="d9cd080d5e6e88498e4056df02098c6b15f38c32c1e027398436906fa8e538be" Jul 17 06:37:28.007661 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:37:28.007346 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm_odh-ai-gateway-infra(60b37db6-38f3-498c-b372-8da065ef709f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" podUID="60b37db6-38f3-498c-b372-8da065ef709f" Jul 17 06:37:28.008356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:28.008333 2580 generic.go:356] "Generic (PLEG): container finished" podID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" containerID="f47f349a20fa642b3ff1d78efc2067c458f94f8b7f60bc5c36f97746b97992e8" exitCode=6 Jul 17 06:37:28.008475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:28.008422 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" event={"ID":"44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4","Type":"ContainerDied","Data":"f47f349a20fa642b3ff1d78efc2067c458f94f8b7f60bc5c36f97746b97992e8"} Jul 17 06:37:28.008475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:28.008469 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" event={"ID":"44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4","Type":"ContainerStarted","Data":"f92b65bf425bd44433a3e383a57abef4b83f732e6cd2409b2d90323dbace5fc4"} Jul 17 06:37:28.008586 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:28.008571 2580 scope.go:117] "RemoveContainer" containerID="f47f349a20fa642b3ff1d78efc2067c458f94f8b7f60bc5c36f97746b97992e8" Jul 17 06:37:29.015332 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:29.015220 2580 generic.go:356] "Generic (PLEG): container finished" podID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" containerID="e5d445d46268a21ba5c01ce6d2ea8546950ab7f0fd8b71c0ee4d276ae490d4ae" exitCode=6 Jul 17 06:37:29.015332 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:29.015310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" event={"ID":"44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4","Type":"ContainerDied","Data":"e5d445d46268a21ba5c01ce6d2ea8546950ab7f0fd8b71c0ee4d276ae490d4ae"} Jul 17 06:37:29.015864 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:29.015395 2580 scope.go:117] "RemoveContainer" containerID="f47f349a20fa642b3ff1d78efc2067c458f94f8b7f60bc5c36f97746b97992e8" Jul 17 06:37:29.015864 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:29.015555 2580 scope.go:117] "RemoveContainer" containerID="e5d445d46268a21ba5c01ce6d2ea8546950ab7f0fd8b71c0ee4d276ae490d4ae" Jul 17 06:37:29.015864 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:37:29.015822 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb_odh-ai-gateway-infra(44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" podUID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" Jul 17 06:37:30.022220 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:30.022191 2580 scope.go:117] "RemoveContainer" containerID="e5d445d46268a21ba5c01ce6d2ea8546950ab7f0fd8b71c0ee4d276ae490d4ae" Jul 17 06:37:30.022653 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:37:30.022410 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb_odh-ai-gateway-infra(44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" podUID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" Jul 17 06:37:33.295328 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:33.295288 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:37:33.351595 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:33.351553 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb"] Jul 17 06:37:33.356229 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:33.356200 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" Jul 17 06:37:33.362297 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:33.362253 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb"] Jul 17 06:37:33.474286 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:33.474247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8b3474-6d4e-49fb-9953-49faf8f4069e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-q5nlb\" (UID: \"3d8b3474-6d4e-49fb-9953-49faf8f4069e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" Jul 17 06:37:33.574865 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:33.574768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8b3474-6d4e-49fb-9953-49faf8f4069e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-q5nlb\" (UID: \"3d8b3474-6d4e-49fb-9953-49faf8f4069e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" Jul 17 06:37:33.575441 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:33.575423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8b3474-6d4e-49fb-9953-49faf8f4069e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-q5nlb\" (UID: \"3d8b3474-6d4e-49fb-9953-49faf8f4069e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" Jul 17 06:37:33.668460 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:33.668375 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" Jul 17 06:37:33.804280 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:33.804253 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb"] Jul 17 06:37:33.806699 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:37:33.806670 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8b3474_6d4e_49fb_9953_49faf8f4069e.slice/crio-717c58267a0c21884ea01ca848236f542cdf4172cbef2d89e62d7f7154e67e35 WatchSource:0}: Error finding container 717c58267a0c21884ea01ca848236f542cdf4172cbef2d89e62d7f7154e67e35: Status 404 returned error can't find the container with id 717c58267a0c21884ea01ca848236f542cdf4172cbef2d89e62d7f7154e67e35 Jul 17 06:37:34.043107 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:34.043070 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" event={"ID":"3d8b3474-6d4e-49fb-9953-49faf8f4069e","Type":"ContainerStarted","Data":"717c58267a0c21884ea01ca848236f542cdf4172cbef2d89e62d7f7154e67e35"} Jul 17 06:37:35.049635 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:35.049594 2580 generic.go:356] "Generic (PLEG): container finished" podID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" containerID="ff7f7822f7d1cd48da31e3e81cb18b9fb6915a50418b9854039c3e20b8334e50" exitCode=6 Jul 17 06:37:35.050024 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:35.049641 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" event={"ID":"3d8b3474-6d4e-49fb-9953-49faf8f4069e","Type":"ContainerDied","Data":"ff7f7822f7d1cd48da31e3e81cb18b9fb6915a50418b9854039c3e20b8334e50"} Jul 17 06:37:35.050024 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:35.049908 2580 scope.go:117] "RemoveContainer" containerID="ff7f7822f7d1cd48da31e3e81cb18b9fb6915a50418b9854039c3e20b8334e50" Jul 17 06:37:35.676038 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:37:35.675997 2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8b3474_6d4e_49fb_9953_49faf8f4069e.slice/crio-conmon-7b63c2d0f15eb1ef9b3d1859970e487b523e9ded3f0140185da7a6f4f3592747.scope\": RecentStats: unable to find data in memory cache]" Jul 17 06:37:36.056278 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:36.056178 2580 generic.go:356] "Generic (PLEG): container finished" podID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" containerID="7b63c2d0f15eb1ef9b3d1859970e487b523e9ded3f0140185da7a6f4f3592747" exitCode=6 Jul 17 06:37:36.056717 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:36.056276 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" event={"ID":"3d8b3474-6d4e-49fb-9953-49faf8f4069e","Type":"ContainerDied","Data":"7b63c2d0f15eb1ef9b3d1859970e487b523e9ded3f0140185da7a6f4f3592747"} Jul 17 06:37:36.056717 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:36.056345 2580 scope.go:117] "RemoveContainer" containerID="ff7f7822f7d1cd48da31e3e81cb18b9fb6915a50418b9854039c3e20b8334e50" Jul 17 06:37:36.056717 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:36.056550 2580 scope.go:117] "RemoveContainer" containerID="7b63c2d0f15eb1ef9b3d1859970e487b523e9ded3f0140185da7a6f4f3592747" Jul 17 06:37:36.056844 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:37:36.056792 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-q5nlb_odh-ai-gateway-infra(3d8b3474-6d4e-49fb-9953-49faf8f4069e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" podUID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" Jul 17 06:37:37.062837 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:37.062808 2580 scope.go:117] "RemoveContainer" containerID="7b63c2d0f15eb1ef9b3d1859970e487b523e9ded3f0140185da7a6f4f3592747" Jul 17 06:37:37.063231 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:37:37.063022 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-q5nlb_odh-ai-gateway-infra(3d8b3474-6d4e-49fb-9953-49faf8f4069e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" podUID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" Jul 17 06:37:40.522314 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:40.522275 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs"] Jul 17 06:37:40.527185 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:40.527160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" Jul 17 06:37:40.532461 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:40.532429 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs"] Jul 17 06:37:40.641495 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:40.641457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b688fc3-9540-4168-b8e5-ae5b4fe00355-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs\" (UID: \"0b688fc3-9540-4168-b8e5-ae5b4fe00355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" Jul 17 06:37:40.742841 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:40.742802 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b688fc3-9540-4168-b8e5-ae5b4fe00355-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs\" (UID: \"0b688fc3-9540-4168-b8e5-ae5b4fe00355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" Jul 17 06:37:40.743463 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:40.743444 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b688fc3-9540-4168-b8e5-ae5b4fe00355-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs\" (UID: \"0b688fc3-9540-4168-b8e5-ae5b4fe00355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" Jul 17 06:37:40.840440 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:40.840326 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" Jul 17 06:37:40.982013 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:40.981954 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs"] Jul 17 06:37:40.984366 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:37:40.984335 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b688fc3_9540_4168_b8e5_ae5b4fe00355.slice/crio-fbbe3367e7b90bff54b1c74bfffe2c6ad4c8bb9ca569f297b84601205ce77c11 WatchSource:0}: Error finding container fbbe3367e7b90bff54b1c74bfffe2c6ad4c8bb9ca569f297b84601205ce77c11: Status 404 returned error can't find the container with id fbbe3367e7b90bff54b1c74bfffe2c6ad4c8bb9ca569f297b84601205ce77c11 Jul 17 06:37:41.083718 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:41.083679 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" event={"ID":"0b688fc3-9540-4168-b8e5-ae5b4fe00355","Type":"ContainerStarted","Data":"fbbe3367e7b90bff54b1c74bfffe2c6ad4c8bb9ca569f297b84601205ce77c11"} Jul 17 06:37:41.884584 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:41.884551 2580 scope.go:117] "RemoveContainer" containerID="e5d445d46268a21ba5c01ce6d2ea8546950ab7f0fd8b71c0ee4d276ae490d4ae" Jul 17 06:37:42.090464 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:42.090425 2580 generic.go:356] "Generic (PLEG): container finished" podID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" containerID="075652f674098f0cdbea262b700d0f0ea33b1089328e03f02f8ca06a24aeeb88" exitCode=6 Jul 17 06:37:42.090649 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:42.090480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" event={"ID":"0b688fc3-9540-4168-b8e5-ae5b4fe00355","Type":"ContainerDied","Data":"075652f674098f0cdbea262b700d0f0ea33b1089328e03f02f8ca06a24aeeb88"} Jul 17 06:37:42.090789 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:42.090774 2580 scope.go:117] "RemoveContainer" containerID="075652f674098f0cdbea262b700d0f0ea33b1089328e03f02f8ca06a24aeeb88" Jul 17 06:37:42.884272 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:42.884239 2580 scope.go:117] "RemoveContainer" containerID="d9cd080d5e6e88498e4056df02098c6b15f38c32c1e027398436906fa8e538be" Jul 17 06:37:43.097760 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:43.097726 2580 generic.go:356] "Generic (PLEG): container finished" podID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" containerID="39f530a0b67860b4396f48e553e1c0b6bdc74858b426ce65c22c9b2a6cdf8f90" exitCode=6 Jul 17 06:37:43.098319 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:43.097810 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" event={"ID":"0b688fc3-9540-4168-b8e5-ae5b4fe00355","Type":"ContainerDied","Data":"39f530a0b67860b4396f48e553e1c0b6bdc74858b426ce65c22c9b2a6cdf8f90"} Jul 17 06:37:43.098319 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:43.097882 2580 scope.go:117] "RemoveContainer" containerID="075652f674098f0cdbea262b700d0f0ea33b1089328e03f02f8ca06a24aeeb88" Jul 17 06:37:43.098319 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:43.098035 2580 scope.go:117] "RemoveContainer" containerID="39f530a0b67860b4396f48e553e1c0b6bdc74858b426ce65c22c9b2a6cdf8f90" Jul 17 06:37:43.098319 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:37:43.098276 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs_odh-ai-gateway-infra(0b688fc3-9540-4168-b8e5-ae5b4fe00355)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" podUID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" Jul 17 06:37:43.099853 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:43.099826 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" event={"ID":"60b37db6-38f3-498c-b372-8da065ef709f","Type":"ContainerStarted","Data":"48105a50af50300376208652de7f3cdd7fc184e406db70f5d8e5872f5667314a"} Jul 17 06:37:43.101609 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:43.101590 2580 generic.go:356] "Generic (PLEG): container finished" podID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" containerID="84efa452d3ec32a02751fbb860781cdaac1396111c1300bdad80e438b14ca6e5" exitCode=6 Jul 17 06:37:43.101713 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:43.101674 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" event={"ID":"44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4","Type":"ContainerDied","Data":"84efa452d3ec32a02751fbb860781cdaac1396111c1300bdad80e438b14ca6e5"} Jul 17 06:37:43.102022 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:43.102002 2580 scope.go:117] "RemoveContainer" containerID="84efa452d3ec32a02751fbb860781cdaac1396111c1300bdad80e438b14ca6e5" Jul 17 06:37:43.102254 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:37:43.102233 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb_odh-ai-gateway-infra(44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" podUID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" Jul 17 06:37:43.109270 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:43.109253 2580 scope.go:117] "RemoveContainer" containerID="e5d445d46268a21ba5c01ce6d2ea8546950ab7f0fd8b71c0ee4d276ae490d4ae" Jul 17 06:37:43.130372 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:43.130322 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" podStartSLOduration=19.130306003 podStartE2EDuration="19.130306003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:37:43.129243907 +0000 UTC m=+3387.873923241" watchObservedRunningTime="2026-07-17 06:37:43.130306003 +0000 UTC m=+3387.874985368" Jul 17 06:37:44.108266 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:44.108232 2580 scope.go:117] "RemoveContainer" containerID="39f530a0b67860b4396f48e553e1c0b6bdc74858b426ce65c22c9b2a6cdf8f90" Jul 17 06:37:44.108720 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:37:44.108488 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs_odh-ai-gateway-infra(0b688fc3-9540-4168-b8e5-ae5b4fe00355)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" podUID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" Jul 17 06:37:44.109771 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:44.109745 2580 generic.go:356] "Generic (PLEG): container finished" podID="60b37db6-38f3-498c-b372-8da065ef709f" containerID="48105a50af50300376208652de7f3cdd7fc184e406db70f5d8e5872f5667314a" exitCode=6 Jul 17 06:37:44.109890 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:44.109828 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" event={"ID":"60b37db6-38f3-498c-b372-8da065ef709f","Type":"ContainerDied","Data":"48105a50af50300376208652de7f3cdd7fc184e406db70f5d8e5872f5667314a"} Jul 17 06:37:44.109949 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:44.109936 2580 scope.go:117] "RemoveContainer" containerID="d9cd080d5e6e88498e4056df02098c6b15f38c32c1e027398436906fa8e538be" Jul 17 06:37:44.110224 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:44.110205 2580 scope.go:117] "RemoveContainer" containerID="48105a50af50300376208652de7f3cdd7fc184e406db70f5d8e5872f5667314a" Jul 17 06:37:44.110463 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:37:44.110440 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm_odh-ai-gateway-infra(60b37db6-38f3-498c-b372-8da065ef709f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" podUID="60b37db6-38f3-498c-b372-8da065ef709f" Jul 17 06:37:44.142148 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:44.142110 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm"] Jul 17 06:37:44.160611 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:44.160573 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb"] Jul 17 06:37:44.294959 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:44.294928 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" Jul 17 06:37:44.380087 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:44.379990 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4-maas-api-service-ca\") pod \"44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4\" (UID: \"44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4\") " Jul 17 06:37:44.380328 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:44.380304 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" (UID: "44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:37:44.481651 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:44.481589 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:37:45.119250 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:45.119226 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" Jul 17 06:37:45.119695 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:45.119226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb" event={"ID":"44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4","Type":"ContainerDied","Data":"f92b65bf425bd44433a3e383a57abef4b83f732e6cd2409b2d90323dbace5fc4"} Jul 17 06:37:45.119695 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:45.119363 2580 scope.go:117] "RemoveContainer" containerID="84efa452d3ec32a02751fbb860781cdaac1396111c1300bdad80e438b14ca6e5" Jul 17 06:37:45.149455 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:45.149352 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb"] Jul 17 06:37:45.153443 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:45.153409 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-l8nvb"] Jul 17 06:37:45.255828 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:45.255797 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" Jul 17 06:37:45.389531 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:45.389487 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b37db6-38f3-498c-b372-8da065ef709f-maas-api-service-ca\") pod \"60b37db6-38f3-498c-b372-8da065ef709f\" (UID: \"60b37db6-38f3-498c-b372-8da065ef709f\") " Jul 17 06:37:45.389871 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:45.389842 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b37db6-38f3-498c-b372-8da065ef709f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60b37db6-38f3-498c-b372-8da065ef709f" (UID: "60b37db6-38f3-498c-b372-8da065ef709f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:37:45.491008 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:45.490906 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b37db6-38f3-498c-b372-8da065ef709f-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:37:46.125372 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:46.125335 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" Jul 17 06:37:46.125895 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:46.125334 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm" event={"ID":"60b37db6-38f3-498c-b372-8da065ef709f","Type":"ContainerDied","Data":"8a4e959ea7f94c20198829b9295a771971c664aadb2a0878958770ef1de69833"} Jul 17 06:37:46.125895 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:46.125453 2580 scope.go:117] "RemoveContainer" containerID="48105a50af50300376208652de7f3cdd7fc184e406db70f5d8e5872f5667314a" Jul 17 06:37:46.148038 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:46.148003 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm"] Jul 17 06:37:46.149780 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:46.149759 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-s6cdm"] Jul 17 06:37:46.889347 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:46.889315 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" path="/var/lib/kubelet/pods/44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4/volumes" Jul 17 06:37:46.889754 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:46.889738 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60b37db6-38f3-498c-b372-8da065ef709f" path="/var/lib/kubelet/pods/60b37db6-38f3-498c-b372-8da065ef709f/volumes" Jul 17 06:37:47.885654 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:47.885615 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:37:48.884188 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:48.884151 2580 scope.go:117] "RemoveContainer" containerID="7b63c2d0f15eb1ef9b3d1859970e487b523e9ded3f0140185da7a6f4f3592747" Jul 17 06:37:49.144508 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:49.144418 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" event={"ID":"3d8b3474-6d4e-49fb-9953-49faf8f4069e","Type":"ContainerStarted","Data":"f3be87d2bfaa343781f56e921773a3ef897c9a3cbf09a1fead2bc0cbd4ad87f2"} Jul 17 06:37:50.151313 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:50.151278 2580 generic.go:356] "Generic (PLEG): container finished" podID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" containerID="f3be87d2bfaa343781f56e921773a3ef897c9a3cbf09a1fead2bc0cbd4ad87f2" exitCode=6 Jul 17 06:37:50.151757 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:50.151350 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" event={"ID":"3d8b3474-6d4e-49fb-9953-49faf8f4069e","Type":"ContainerDied","Data":"f3be87d2bfaa343781f56e921773a3ef897c9a3cbf09a1fead2bc0cbd4ad87f2"} Jul 17 06:37:50.151757 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:50.151446 2580 scope.go:117] "RemoveContainer" containerID="7b63c2d0f15eb1ef9b3d1859970e487b523e9ded3f0140185da7a6f4f3592747" Jul 17 06:37:50.151757 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:50.151744 2580 scope.go:117] "RemoveContainer" containerID="f3be87d2bfaa343781f56e921773a3ef897c9a3cbf09a1fead2bc0cbd4ad87f2" Jul 17 06:37:50.151966 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:37:50.151945 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-q5nlb_odh-ai-gateway-infra(3d8b3474-6d4e-49fb-9953-49faf8f4069e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" podUID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" Jul 17 06:37:50.186071 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:50.186028 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb"] Jul 17 06:37:51.299610 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:51.299583 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" Jul 17 06:37:51.342609 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:51.342572 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8b3474-6d4e-49fb-9953-49faf8f4069e-maas-api-service-ca\") pod \"3d8b3474-6d4e-49fb-9953-49faf8f4069e\" (UID: \"3d8b3474-6d4e-49fb-9953-49faf8f4069e\") " Jul 17 06:37:51.342924 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:51.342900 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8b3474-6d4e-49fb-9953-49faf8f4069e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d8b3474-6d4e-49fb-9953-49faf8f4069e" (UID: "3d8b3474-6d4e-49fb-9953-49faf8f4069e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:37:51.443904 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:51.443811 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8b3474-6d4e-49fb-9953-49faf8f4069e-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:37:52.164407 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:52.164355 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" Jul 17 06:37:52.164598 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:52.164405 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb" event={"ID":"3d8b3474-6d4e-49fb-9953-49faf8f4069e","Type":"ContainerDied","Data":"717c58267a0c21884ea01ca848236f542cdf4172cbef2d89e62d7f7154e67e35"} Jul 17 06:37:52.164598 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:52.164463 2580 scope.go:117] "RemoveContainer" containerID="f3be87d2bfaa343781f56e921773a3ef897c9a3cbf09a1fead2bc0cbd4ad87f2" Jul 17 06:37:52.186429 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:52.186367 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb"] Jul 17 06:37:52.194058 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:52.194027 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-q5nlb"] Jul 17 06:37:52.890249 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:52.890217 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" path="/var/lib/kubelet/pods/3d8b3474-6d4e-49fb-9953-49faf8f4069e/volumes" Jul 17 06:37:59.884552 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:37:59.884520 2580 scope.go:117] "RemoveContainer" containerID="39f530a0b67860b4396f48e553e1c0b6bdc74858b426ce65c22c9b2a6cdf8f90" Jul 17 06:38:00.207271 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:00.207185 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" event={"ID":"0b688fc3-9540-4168-b8e5-ae5b4fe00355","Type":"ContainerStarted","Data":"081a8a8a81ba5036919c805a26b6ed4641f73aac48a6a7fbe8416ea2d582f639"} Jul 17 06:38:00.225467 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:00.225405 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" podStartSLOduration=20.225373861 podStartE2EDuration="20.225373861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:38:00.224133473 +0000 UTC m=+3404.968812822" watchObservedRunningTime="2026-07-17 06:38:00.225373861 +0000 UTC m=+3404.970053189" Jul 17 06:38:01.213924 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:01.213886 2580 generic.go:356] "Generic (PLEG): container finished" podID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" containerID="081a8a8a81ba5036919c805a26b6ed4641f73aac48a6a7fbe8416ea2d582f639" exitCode=6 Jul 17 06:38:01.214369 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:01.213954 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" event={"ID":"0b688fc3-9540-4168-b8e5-ae5b4fe00355","Type":"ContainerDied","Data":"081a8a8a81ba5036919c805a26b6ed4641f73aac48a6a7fbe8416ea2d582f639"} Jul 17 06:38:01.214369 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:01.214044 2580 scope.go:117] "RemoveContainer" containerID="39f530a0b67860b4396f48e553e1c0b6bdc74858b426ce65c22c9b2a6cdf8f90" Jul 17 06:38:01.214369 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:01.214284 2580 scope.go:117] "RemoveContainer" containerID="081a8a8a81ba5036919c805a26b6ed4641f73aac48a6a7fbe8416ea2d582f639" Jul 17 06:38:01.214579 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:38:01.214556 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs_odh-ai-gateway-infra(0b688fc3-9540-4168-b8e5-ae5b4fe00355)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" podUID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" Jul 17 06:38:01.240673 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:01.240635 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs"] Jul 17 06:38:02.355878 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:02.355848 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" Jul 17 06:38:02.442468 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:02.442426 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b688fc3-9540-4168-b8e5-ae5b4fe00355-maas-api-service-ca\") pod \"0b688fc3-9540-4168-b8e5-ae5b4fe00355\" (UID: \"0b688fc3-9540-4168-b8e5-ae5b4fe00355\") " Jul 17 06:38:02.442840 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:02.442811 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b688fc3-9540-4168-b8e5-ae5b4fe00355-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b688fc3-9540-4168-b8e5-ae5b4fe00355" (UID: "0b688fc3-9540-4168-b8e5-ae5b4fe00355"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:38:02.543958 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:02.543858 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b688fc3-9540-4168-b8e5-ae5b4fe00355-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:38:03.227053 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:03.226957 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" event={"ID":"0b688fc3-9540-4168-b8e5-ae5b4fe00355","Type":"ContainerDied","Data":"fbbe3367e7b90bff54b1c74bfffe2c6ad4c8bb9ca569f297b84601205ce77c11"} Jul 17 06:38:03.227053 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:03.227027 2580 scope.go:117] "RemoveContainer" containerID="081a8a8a81ba5036919c805a26b6ed4641f73aac48a6a7fbe8416ea2d582f639" Jul 17 06:38:03.227279 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:03.226981 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs" Jul 17 06:38:03.244077 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:03.244040 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs"] Jul 17 06:38:03.251027 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:03.250996 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-pbmvs"] Jul 17 06:38:04.889759 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:04.889725 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" path="/var/lib/kubelet/pods/0b688fc3-9540-4168-b8e5-ae5b4fe00355/volumes" Jul 17 06:38:07.386336 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:07.386246 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:38:16.647587 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.647548 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9"] Jul 17 06:38:16.647990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.647918 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" containerName="revoke-keys" Jul 17 06:38:16.647990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.647930 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" containerName="revoke-keys" Jul 17 06:38:16.647990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.647940 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" containerName="revoke-keys" Jul 17 06:38:16.647990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.647947 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" containerName="revoke-keys" Jul 17 06:38:16.647990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.647954 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" containerName="revoke-keys" Jul 17 06:38:16.647990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.647960 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" containerName="revoke-keys" Jul 17 06:38:16.647990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.647967 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b37db6-38f3-498c-b372-8da065ef709f" containerName="revoke-keys" Jul 17 06:38:16.647990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.647972 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b37db6-38f3-498c-b372-8da065ef709f" containerName="revoke-keys" Jul 17 06:38:16.647990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.647979 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" containerName="revoke-keys" Jul 17 06:38:16.647990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.647984 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" containerName="revoke-keys" Jul 17 06:38:16.647990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.647993 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.647998 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648004 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648009 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648018 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b37db6-38f3-498c-b372-8da065ef709f" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648023 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b37db6-38f3-498c-b372-8da065ef709f" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648032 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b37db6-38f3-498c-b372-8da065ef709f" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648038 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b37db6-38f3-498c-b372-8da065ef709f" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648058 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648063 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648122 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648130 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b37db6-38f3-498c-b372-8da065ef709f" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648137 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b37db6-38f3-498c-b372-8da065ef709f" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648142 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648148 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648153 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648158 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648163 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648168 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648172 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b688fc3-9540-4168-b8e5-ae5b4fe00355" containerName="revoke-keys" Jul 17 06:38:16.648334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.648177 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" containerName="revoke-keys" Jul 17 06:38:16.652638 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.652615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" Jul 17 06:38:16.655003 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.654972 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:38:16.659813 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.659782 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9"] Jul 17 06:38:16.768824 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.768777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea75135e-c561-43c5-8bda-8fc2bb760b77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9\" (UID: \"ea75135e-c561-43c5-8bda-8fc2bb760b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" Jul 17 06:38:16.869951 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.869908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea75135e-c561-43c5-8bda-8fc2bb760b77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9\" (UID: \"ea75135e-c561-43c5-8bda-8fc2bb760b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" Jul 17 06:38:16.870602 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.870580 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea75135e-c561-43c5-8bda-8fc2bb760b77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9\" (UID: \"ea75135e-c561-43c5-8bda-8fc2bb760b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" Jul 17 06:38:16.964346 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:16.964233 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" Jul 17 06:38:17.106401 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:17.106357 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9"] Jul 17 06:38:17.109279 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:38:17.109251 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea75135e_c561_43c5_8bda_8fc2bb760b77.slice/crio-9ac7e0c655dde6025b4328932b06ea0f2c10dfc61b25272f614d8c568652ec30 WatchSource:0}: Error finding container 9ac7e0c655dde6025b4328932b06ea0f2c10dfc61b25272f614d8c568652ec30: Status 404 returned error can't find the container with id 9ac7e0c655dde6025b4328932b06ea0f2c10dfc61b25272f614d8c568652ec30 Jul 17 06:38:17.302523 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:17.302432 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" event={"ID":"ea75135e-c561-43c5-8bda-8fc2bb760b77","Type":"ContainerStarted","Data":"9ac7e0c655dde6025b4328932b06ea0f2c10dfc61b25272f614d8c568652ec30"} Jul 17 06:38:18.309806 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:18.309765 2580 generic.go:356] "Generic (PLEG): container finished" podID="ea75135e-c561-43c5-8bda-8fc2bb760b77" containerID="e1d143aba1561911cc9e83029ea0fad61759e5df29225609abaa8999b6da5d3a" exitCode=6 Jul 17 06:38:18.310245 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:18.309847 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" event={"ID":"ea75135e-c561-43c5-8bda-8fc2bb760b77","Type":"ContainerDied","Data":"e1d143aba1561911cc9e83029ea0fad61759e5df29225609abaa8999b6da5d3a"} Jul 17 06:38:18.310245 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:18.310033 2580 scope.go:117] "RemoveContainer" containerID="e1d143aba1561911cc9e83029ea0fad61759e5df29225609abaa8999b6da5d3a" Jul 17 06:38:19.316358 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:19.316265 2580 generic.go:356] "Generic (PLEG): container finished" podID="ea75135e-c561-43c5-8bda-8fc2bb760b77" containerID="9516f95c770dced3142169a0728a92cdb21a91a9bac26b04c3ccabf53f8ddd64" exitCode=6 Jul 17 06:38:19.316358 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:19.316333 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" event={"ID":"ea75135e-c561-43c5-8bda-8fc2bb760b77","Type":"ContainerDied","Data":"9516f95c770dced3142169a0728a92cdb21a91a9bac26b04c3ccabf53f8ddd64"} Jul 17 06:38:19.316860 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:19.316423 2580 scope.go:117] "RemoveContainer" containerID="e1d143aba1561911cc9e83029ea0fad61759e5df29225609abaa8999b6da5d3a" Jul 17 06:38:19.316860 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:19.316691 2580 scope.go:117] "RemoveContainer" containerID="9516f95c770dced3142169a0728a92cdb21a91a9bac26b04c3ccabf53f8ddd64" Jul 17 06:38:19.316941 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:38:19.316909 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9_odh-ai-gateway-infra(ea75135e-c561-43c5-8bda-8fc2bb760b77)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" podUID="ea75135e-c561-43c5-8bda-8fc2bb760b77" Jul 17 06:38:20.323288 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:20.323260 2580 scope.go:117] "RemoveContainer" containerID="9516f95c770dced3142169a0728a92cdb21a91a9bac26b04c3ccabf53f8ddd64" Jul 17 06:38:20.323728 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:38:20.323491 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9_odh-ai-gateway-infra(ea75135e-c561-43c5-8bda-8fc2bb760b77)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" podUID="ea75135e-c561-43c5-8bda-8fc2bb760b77" Jul 17 06:38:23.399192 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:23.399153 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69"] Jul 17 06:38:23.399620 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:23.399530 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" containerName="revoke-keys" Jul 17 06:38:23.399620 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:23.399542 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8b3474-6d4e-49fb-9953-49faf8f4069e" containerName="revoke-keys" Jul 17 06:38:23.399620 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:23.399549 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" containerName="revoke-keys" Jul 17 06:38:23.399620 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:23.399555 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b5b2a0-cfe5-4fa8-ab01-fafcd00d53b4" containerName="revoke-keys" Jul 17 06:38:23.399620 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:23.399615 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b37db6-38f3-498c-b372-8da065ef709f" containerName="revoke-keys" Jul 17 06:38:23.403709 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:23.403690 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" Jul 17 06:38:23.409689 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:23.409661 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69"] Jul 17 06:38:23.532181 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:23.532142 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e859cf-8268-4769-b0f5-72c770970acb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-hhb69\" (UID: \"f7e859cf-8268-4769-b0f5-72c770970acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" Jul 17 06:38:23.632716 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:23.632677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e859cf-8268-4769-b0f5-72c770970acb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-hhb69\" (UID: \"f7e859cf-8268-4769-b0f5-72c770970acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" Jul 17 06:38:23.633325 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:23.633303 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e859cf-8268-4769-b0f5-72c770970acb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-hhb69\" (UID: \"f7e859cf-8268-4769-b0f5-72c770970acb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" Jul 17 06:38:23.715140 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:23.715054 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" Jul 17 06:38:23.845593 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:23.845566 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69"] Jul 17 06:38:23.847794 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:38:23.847759 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e859cf_8268_4769_b0f5_72c770970acb.slice/crio-9ed4a65011cd1293a2ce64fd820ef9c2c1fa1aedcd372c6c12ae745ec0d5aea9 WatchSource:0}: Error finding container 9ed4a65011cd1293a2ce64fd820ef9c2c1fa1aedcd372c6c12ae745ec0d5aea9: Status 404 returned error can't find the container with id 9ed4a65011cd1293a2ce64fd820ef9c2c1fa1aedcd372c6c12ae745ec0d5aea9 Jul 17 06:38:24.343916 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:24.343827 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" event={"ID":"f7e859cf-8268-4769-b0f5-72c770970acb","Type":"ContainerStarted","Data":"6b3401e73b551f11448a71bb59666040d40cdabf9837a77a2fec50fc3ccb4a51"} Jul 17 06:38:24.343916 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:24.343881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" event={"ID":"f7e859cf-8268-4769-b0f5-72c770970acb","Type":"ContainerStarted","Data":"9ed4a65011cd1293a2ce64fd820ef9c2c1fa1aedcd372c6c12ae745ec0d5aea9"} Jul 17 06:38:24.358600 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:24.358374 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" podStartSLOduration=1.35835447 podStartE2EDuration="1.35835447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:38:24.358054401 +0000 UTC m=+3429.102733729" watchObservedRunningTime="2026-07-17 06:38:24.35835447 +0000 UTC m=+3429.103033795" Jul 17 06:38:24.537162 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:24.537130 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s"] Jul 17 06:38:24.540586 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:24.540562 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" Jul 17 06:38:24.547311 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:24.547275 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s"] Jul 17 06:38:24.641150 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:24.641101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf9113a-56d3-4688-b5b9-148329d04c60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-fch9s\" (UID: \"3cf9113a-56d3-4688-b5b9-148329d04c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" Jul 17 06:38:24.742273 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:24.742236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf9113a-56d3-4688-b5b9-148329d04c60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-fch9s\" (UID: \"3cf9113a-56d3-4688-b5b9-148329d04c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" Jul 17 06:38:24.742900 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:24.742877 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf9113a-56d3-4688-b5b9-148329d04c60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-fch9s\" (UID: \"3cf9113a-56d3-4688-b5b9-148329d04c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" Jul 17 06:38:24.908440 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:24.908310 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" Jul 17 06:38:25.051545 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:25.051478 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s"] Jul 17 06:38:25.053768 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:38:25.053742 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf9113a_56d3_4688_b5b9_148329d04c60.slice/crio-6fad8956876cbd82913a7abf8e812eb4a858dbe9fce576479c106049bb2d2981 WatchSource:0}: Error finding container 6fad8956876cbd82913a7abf8e812eb4a858dbe9fce576479c106049bb2d2981: Status 404 returned error can't find the container with id 6fad8956876cbd82913a7abf8e812eb4a858dbe9fce576479c106049bb2d2981 Jul 17 06:38:25.350734 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:25.350704 2580 generic.go:356] "Generic (PLEG): container finished" podID="f7e859cf-8268-4769-b0f5-72c770970acb" containerID="6b3401e73b551f11448a71bb59666040d40cdabf9837a77a2fec50fc3ccb4a51" exitCode=6 Jul 17 06:38:25.350874 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:25.350778 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" event={"ID":"f7e859cf-8268-4769-b0f5-72c770970acb","Type":"ContainerDied","Data":"6b3401e73b551f11448a71bb59666040d40cdabf9837a77a2fec50fc3ccb4a51"} Jul 17 06:38:25.351015 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:25.351002 2580 scope.go:117] "RemoveContainer" containerID="6b3401e73b551f11448a71bb59666040d40cdabf9837a77a2fec50fc3ccb4a51" Jul 17 06:38:25.358865 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:25.358783 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" event={"ID":"3cf9113a-56d3-4688-b5b9-148329d04c60","Type":"ContainerStarted","Data":"6fad8956876cbd82913a7abf8e812eb4a858dbe9fce576479c106049bb2d2981"} Jul 17 06:38:25.380160 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:25.380107 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" podStartSLOduration=1.380087221 podStartE2EDuration="1.380087221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:38:25.379742753 +0000 UTC m=+3430.124422104" watchObservedRunningTime="2026-07-17 06:38:25.380087221 +0000 UTC m=+3430.124766549" Jul 17 06:38:26.365104 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:26.365066 2580 generic.go:356] "Generic (PLEG): container finished" podID="f7e859cf-8268-4769-b0f5-72c770970acb" containerID="1455dc8fb9d72a31bcb460a178979cf49fc14a774e7ccc93bb410a26479ed2a4" exitCode=6 Jul 17 06:38:26.365627 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:26.365159 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" event={"ID":"f7e859cf-8268-4769-b0f5-72c770970acb","Type":"ContainerDied","Data":"1455dc8fb9d72a31bcb460a178979cf49fc14a774e7ccc93bb410a26479ed2a4"} Jul 17 06:38:26.365627 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:26.365230 2580 scope.go:117] "RemoveContainer" containerID="6b3401e73b551f11448a71bb59666040d40cdabf9837a77a2fec50fc3ccb4a51" Jul 17 06:38:26.365627 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:26.365339 2580 scope.go:117] "RemoveContainer" containerID="1455dc8fb9d72a31bcb460a178979cf49fc14a774e7ccc93bb410a26479ed2a4" Jul 17 06:38:26.365627 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:38:26.365611 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-hhb69_odh-ai-gateway-infra(f7e859cf-8268-4769-b0f5-72c770970acb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" podUID="f7e859cf-8268-4769-b0f5-72c770970acb" Jul 17 06:38:26.366811 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:26.366788 2580 generic.go:356] "Generic (PLEG): container finished" podID="3cf9113a-56d3-4688-b5b9-148329d04c60" containerID="c5d38b7cf44136ae52195762b45be4c9b2bd1ed97ce236ebafd5667194aef8d3" exitCode=6 Jul 17 06:38:26.366906 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:26.366872 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" event={"ID":"3cf9113a-56d3-4688-b5b9-148329d04c60","Type":"ContainerDied","Data":"c5d38b7cf44136ae52195762b45be4c9b2bd1ed97ce236ebafd5667194aef8d3"} Jul 17 06:38:26.367028 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:26.367013 2580 scope.go:117] "RemoveContainer" containerID="c5d38b7cf44136ae52195762b45be4c9b2bd1ed97ce236ebafd5667194aef8d3" Jul 17 06:38:27.275125 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:27.275030 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:38:27.373257 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:27.373228 2580 scope.go:117] "RemoveContainer" containerID="1455dc8fb9d72a31bcb460a178979cf49fc14a774e7ccc93bb410a26479ed2a4" Jul 17 06:38:27.373766 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:38:27.373508 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-hhb69_odh-ai-gateway-infra(f7e859cf-8268-4769-b0f5-72c770970acb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" podUID="f7e859cf-8268-4769-b0f5-72c770970acb" Jul 17 06:38:27.374874 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:27.374849 2580 generic.go:356] "Generic (PLEG): container finished" podID="3cf9113a-56d3-4688-b5b9-148329d04c60" containerID="1bc8b426a330342c54a49965b80d18765b057e164aa751a413a24df033cd6325" exitCode=6 Jul 17 06:38:27.374989 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:27.374931 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" event={"ID":"3cf9113a-56d3-4688-b5b9-148329d04c60","Type":"ContainerDied","Data":"1bc8b426a330342c54a49965b80d18765b057e164aa751a413a24df033cd6325"} Jul 17 06:38:27.374989 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:27.374987 2580 scope.go:117] "RemoveContainer" containerID="c5d38b7cf44136ae52195762b45be4c9b2bd1ed97ce236ebafd5667194aef8d3" Jul 17 06:38:27.375128 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:27.375110 2580 scope.go:117] "RemoveContainer" containerID="1bc8b426a330342c54a49965b80d18765b057e164aa751a413a24df033cd6325" Jul 17 06:38:27.375339 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:38:27.375320 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-fch9s_odh-ai-gateway-infra(3cf9113a-56d3-4688-b5b9-148329d04c60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" podUID="3cf9113a-56d3-4688-b5b9-148329d04c60" Jul 17 06:38:28.382045 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:28.382018 2580 scope.go:117] "RemoveContainer" containerID="1bc8b426a330342c54a49965b80d18765b057e164aa751a413a24df033cd6325" Jul 17 06:38:28.382506 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:38:28.382205 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-fch9s_odh-ai-gateway-infra(3cf9113a-56d3-4688-b5b9-148329d04c60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" podUID="3cf9113a-56d3-4688-b5b9-148329d04c60" Jul 17 06:38:35.883958 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:35.883921 2580 scope.go:117] "RemoveContainer" containerID="9516f95c770dced3142169a0728a92cdb21a91a9bac26b04c3ccabf53f8ddd64" Jul 17 06:38:36.422913 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:36.422875 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" event={"ID":"ea75135e-c561-43c5-8bda-8fc2bb760b77","Type":"ContainerStarted","Data":"8c1354fa4f10f881d8bf49cb5e9f392a86b9bfb1576c3affc699f7c906336e7f"} Jul 17 06:38:36.440802 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:36.440750 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" podStartSLOduration=20.44073139 podStartE2EDuration="20.44073139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:38:36.43885966 +0000 UTC m=+3441.183539001" watchObservedRunningTime="2026-07-17 06:38:36.44073139 +0000 UTC m=+3441.185410718" Jul 17 06:38:37.429869 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:37.429831 2580 generic.go:356] "Generic (PLEG): container finished" podID="ea75135e-c561-43c5-8bda-8fc2bb760b77" containerID="8c1354fa4f10f881d8bf49cb5e9f392a86b9bfb1576c3affc699f7c906336e7f" exitCode=6 Jul 17 06:38:37.430307 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:37.429903 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" event={"ID":"ea75135e-c561-43c5-8bda-8fc2bb760b77","Type":"ContainerDied","Data":"8c1354fa4f10f881d8bf49cb5e9f392a86b9bfb1576c3affc699f7c906336e7f"} Jul 17 06:38:37.430307 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:37.429967 2580 scope.go:117] "RemoveContainer" containerID="9516f95c770dced3142169a0728a92cdb21a91a9bac26b04c3ccabf53f8ddd64" Jul 17 06:38:37.430307 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:37.430283 2580 scope.go:117] "RemoveContainer" containerID="8c1354fa4f10f881d8bf49cb5e9f392a86b9bfb1576c3affc699f7c906336e7f" Jul 17 06:38:37.430599 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:38:37.430578 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9_odh-ai-gateway-infra(ea75135e-c561-43c5-8bda-8fc2bb760b77)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" podUID="ea75135e-c561-43c5-8bda-8fc2bb760b77" Jul 17 06:38:37.457451 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:37.457413 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9"] Jul 17 06:38:38.564856 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:38.564831 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" Jul 17 06:38:38.666466 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:38.666432 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea75135e-c561-43c5-8bda-8fc2bb760b77-maas-api-service-ca\") pod \"ea75135e-c561-43c5-8bda-8fc2bb760b77\" (UID: \"ea75135e-c561-43c5-8bda-8fc2bb760b77\") " Jul 17 06:38:38.666787 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:38.666760 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea75135e-c561-43c5-8bda-8fc2bb760b77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea75135e-c561-43c5-8bda-8fc2bb760b77" (UID: "ea75135e-c561-43c5-8bda-8fc2bb760b77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:38:38.767574 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:38.767471 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea75135e-c561-43c5-8bda-8fc2bb760b77-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:38:38.884715 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:38.884683 2580 scope.go:117] "RemoveContainer" containerID="1455dc8fb9d72a31bcb460a178979cf49fc14a774e7ccc93bb410a26479ed2a4" Jul 17 06:38:39.444015 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:39.443989 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" Jul 17 06:38:39.444182 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:39.444010 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9" event={"ID":"ea75135e-c561-43c5-8bda-8fc2bb760b77","Type":"ContainerDied","Data":"9ac7e0c655dde6025b4328932b06ea0f2c10dfc61b25272f614d8c568652ec30"} Jul 17 06:38:39.444182 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:39.444079 2580 scope.go:117] "RemoveContainer" containerID="8c1354fa4f10f881d8bf49cb5e9f392a86b9bfb1576c3affc699f7c906336e7f" Jul 17 06:38:39.446074 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:39.446052 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" event={"ID":"f7e859cf-8268-4769-b0f5-72c770970acb","Type":"ContainerStarted","Data":"61a0884adbfd08ebe72f3d8b43eea46a1b0622097de5218d6e8c7359103b9584"} Jul 17 06:38:39.500709 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:39.500679 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9"] Jul 17 06:38:39.504604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:39.504575 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-qq9z9"] Jul 17 06:38:39.883884 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:39.883849 2580 scope.go:117] "RemoveContainer" containerID="1bc8b426a330342c54a49965b80d18765b057e164aa751a413a24df033cd6325" Jul 17 06:38:40.452559 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.452526 2580 generic.go:356] "Generic (PLEG): container finished" podID="f7e859cf-8268-4769-b0f5-72c770970acb" containerID="61a0884adbfd08ebe72f3d8b43eea46a1b0622097de5218d6e8c7359103b9584" exitCode=6 Jul 17 06:38:40.452689 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.452596 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" event={"ID":"f7e859cf-8268-4769-b0f5-72c770970acb","Type":"ContainerDied","Data":"61a0884adbfd08ebe72f3d8b43eea46a1b0622097de5218d6e8c7359103b9584"} Jul 17 06:38:40.452737 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.452688 2580 scope.go:117] "RemoveContainer" containerID="1455dc8fb9d72a31bcb460a178979cf49fc14a774e7ccc93bb410a26479ed2a4" Jul 17 06:38:40.452968 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.452948 2580 scope.go:117] "RemoveContainer" containerID="61a0884adbfd08ebe72f3d8b43eea46a1b0622097de5218d6e8c7359103b9584" Jul 17 06:38:40.453212 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:38:40.453189 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-hhb69_odh-ai-gateway-infra(f7e859cf-8268-4769-b0f5-72c770970acb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" podUID="f7e859cf-8268-4769-b0f5-72c770970acb" Jul 17 06:38:40.454475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.454456 2580 generic.go:356] "Generic (PLEG): container finished" podID="3cf9113a-56d3-4688-b5b9-148329d04c60" containerID="66faaa6b166606ce719511900ea6b6d499ca66b6ce4c23c63dce0caa3a662b95" exitCode=6 Jul 17 06:38:40.454551 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.454525 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" event={"ID":"3cf9113a-56d3-4688-b5b9-148329d04c60","Type":"ContainerDied","Data":"66faaa6b166606ce719511900ea6b6d499ca66b6ce4c23c63dce0caa3a662b95"} Jul 17 06:38:40.454823 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.454807 2580 scope.go:117] "RemoveContainer" containerID="66faaa6b166606ce719511900ea6b6d499ca66b6ce4c23c63dce0caa3a662b95" Jul 17 06:38:40.455004 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:38:40.454985 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-fch9s_odh-ai-gateway-infra(3cf9113a-56d3-4688-b5b9-148329d04c60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" podUID="3cf9113a-56d3-4688-b5b9-148329d04c60" Jul 17 06:38:40.477181 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.477147 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69"] Jul 17 06:38:40.480704 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.480684 2580 scope.go:117] "RemoveContainer" containerID="1bc8b426a330342c54a49965b80d18765b057e164aa751a413a24df033cd6325" Jul 17 06:38:40.568902 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.568863 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk"] Jul 17 06:38:40.569220 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.569207 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea75135e-c561-43c5-8bda-8fc2bb760b77" containerName="revoke-keys" Jul 17 06:38:40.569266 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.569222 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea75135e-c561-43c5-8bda-8fc2bb760b77" containerName="revoke-keys" Jul 17 06:38:40.569266 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.569247 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea75135e-c561-43c5-8bda-8fc2bb760b77" containerName="revoke-keys" Jul 17 06:38:40.569266 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.569252 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea75135e-c561-43c5-8bda-8fc2bb760b77" containerName="revoke-keys" Jul 17 06:38:40.569368 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.569304 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea75135e-c561-43c5-8bda-8fc2bb760b77" containerName="revoke-keys" Jul 17 06:38:40.569368 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.569316 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea75135e-c561-43c5-8bda-8fc2bb760b77" containerName="revoke-keys" Jul 17 06:38:40.573548 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.573528 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" Jul 17 06:38:40.577817 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.577790 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk"] Jul 17 06:38:40.684626 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.684576 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c42d0-1a86-40da-869c-272cb161d239-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-rtttk\" (UID: \"f72c42d0-1a86-40da-869c-272cb161d239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" Jul 17 06:38:40.785848 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.785759 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c42d0-1a86-40da-869c-272cb161d239-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-rtttk\" (UID: \"f72c42d0-1a86-40da-869c-272cb161d239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" Jul 17 06:38:40.786372 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.786352 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c42d0-1a86-40da-869c-272cb161d239-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-rtttk\" (UID: \"f72c42d0-1a86-40da-869c-272cb161d239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" Jul 17 06:38:40.884172 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.884137 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" Jul 17 06:38:40.890284 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:40.890257 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea75135e-c561-43c5-8bda-8fc2bb760b77" path="/var/lib/kubelet/pods/ea75135e-c561-43c5-8bda-8fc2bb760b77/volumes" Jul 17 06:38:41.016170 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:41.016138 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk"] Jul 17 06:38:41.018194 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:38:41.018167 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72c42d0_1a86_40da_869c_272cb161d239.slice/crio-22356dafd4cb0174ca37fdb900b13be7201ebe28cfae02da581186932e05df07 WatchSource:0}: Error finding container 22356dafd4cb0174ca37fdb900b13be7201ebe28cfae02da581186932e05df07: Status 404 returned error can't find the container with id 22356dafd4cb0174ca37fdb900b13be7201ebe28cfae02da581186932e05df07 Jul 17 06:38:41.461616 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:41.461576 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" event={"ID":"f72c42d0-1a86-40da-869c-272cb161d239","Type":"ContainerStarted","Data":"7d5bdf3c0a6e8f39f9828999128d2122a3de3d803cc563eaa2ff665526a9fc93"} Jul 17 06:38:41.461813 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:41.461639 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" event={"ID":"f72c42d0-1a86-40da-869c-272cb161d239","Type":"ContainerStarted","Data":"22356dafd4cb0174ca37fdb900b13be7201ebe28cfae02da581186932e05df07"} Jul 17 06:38:41.479719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:41.479648 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" podStartSLOduration=1.479627485 podStartE2EDuration="1.479627485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:38:41.476652188 +0000 UTC m=+3446.221331516" watchObservedRunningTime="2026-07-17 06:38:41.479627485 +0000 UTC m=+3446.224306816" Jul 17 06:38:41.498035 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:41.498001 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s"] Jul 17 06:38:41.603084 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:41.603059 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" Jul 17 06:38:41.639856 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:41.639828 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" Jul 17 06:38:41.794675 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:41.794638 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf9113a-56d3-4688-b5b9-148329d04c60-maas-api-service-ca\") pod \"3cf9113a-56d3-4688-b5b9-148329d04c60\" (UID: \"3cf9113a-56d3-4688-b5b9-148329d04c60\") " Jul 17 06:38:41.794852 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:41.794747 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e859cf-8268-4769-b0f5-72c770970acb-maas-api-service-ca\") pod \"f7e859cf-8268-4769-b0f5-72c770970acb\" (UID: \"f7e859cf-8268-4769-b0f5-72c770970acb\") " Jul 17 06:38:41.795124 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:41.795100 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf9113a-56d3-4688-b5b9-148329d04c60-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3cf9113a-56d3-4688-b5b9-148329d04c60" (UID: "3cf9113a-56d3-4688-b5b9-148329d04c60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:38:41.795167 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:41.795096 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e859cf-8268-4769-b0f5-72c770970acb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7e859cf-8268-4769-b0f5-72c770970acb" (UID: "f7e859cf-8268-4769-b0f5-72c770970acb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:38:41.896036 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:41.895946 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf9113a-56d3-4688-b5b9-148329d04c60-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:38:41.896036 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:41.895977 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e859cf-8268-4769-b0f5-72c770970acb-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:38:42.065857 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.065813 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:38:42.471006 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.470973 2580 generic.go:356] "Generic (PLEG): container finished" podID="f72c42d0-1a86-40da-869c-272cb161d239" containerID="7d5bdf3c0a6e8f39f9828999128d2122a3de3d803cc563eaa2ff665526a9fc93" exitCode=6 Jul 17 06:38:42.471189 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.471069 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" event={"ID":"f72c42d0-1a86-40da-869c-272cb161d239","Type":"ContainerDied","Data":"7d5bdf3c0a6e8f39f9828999128d2122a3de3d803cc563eaa2ff665526a9fc93"} Jul 17 06:38:42.471275 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.471219 2580 scope.go:117] "RemoveContainer" containerID="7d5bdf3c0a6e8f39f9828999128d2122a3de3d803cc563eaa2ff665526a9fc93" Jul 17 06:38:42.472432 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.472411 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" Jul 17 06:38:42.472550 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.472415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69" event={"ID":"f7e859cf-8268-4769-b0f5-72c770970acb","Type":"ContainerDied","Data":"9ed4a65011cd1293a2ce64fd820ef9c2c1fa1aedcd372c6c12ae745ec0d5aea9"} Jul 17 06:38:42.472616 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.472555 2580 scope.go:117] "RemoveContainer" containerID="61a0884adbfd08ebe72f3d8b43eea46a1b0622097de5218d6e8c7359103b9584" Jul 17 06:38:42.473750 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.473725 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" event={"ID":"3cf9113a-56d3-4688-b5b9-148329d04c60","Type":"ContainerDied","Data":"6fad8956876cbd82913a7abf8e812eb4a858dbe9fce576479c106049bb2d2981"} Jul 17 06:38:42.473831 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.473816 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s" Jul 17 06:38:42.485716 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.485226 2580 scope.go:117] "RemoveContainer" containerID="66faaa6b166606ce719511900ea6b6d499ca66b6ce4c23c63dce0caa3a662b95" Jul 17 06:38:42.499285 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.499252 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69"] Jul 17 06:38:42.504973 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.504947 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-hhb69"] Jul 17 06:38:42.518341 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.518316 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s"] Jul 17 06:38:42.524779 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.524750 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-fch9s"] Jul 17 06:38:42.891089 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.891050 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cf9113a-56d3-4688-b5b9-148329d04c60" path="/var/lib/kubelet/pods/3cf9113a-56d3-4688-b5b9-148329d04c60/volumes" Jul 17 06:38:42.891629 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.891603 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e859cf-8268-4769-b0f5-72c770970acb" path="/var/lib/kubelet/pods/f7e859cf-8268-4769-b0f5-72c770970acb/volumes" Jul 17 06:38:42.934623 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:42.934592 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:38:43.481231 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:43.481146 2580 generic.go:356] "Generic (PLEG): container finished" podID="f72c42d0-1a86-40da-869c-272cb161d239" containerID="a6cab79583941220c62a21f1484e8466be8b0b7b875e8016ee83db432d1d7613" exitCode=6 Jul 17 06:38:43.481231 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:43.481221 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" event={"ID":"f72c42d0-1a86-40da-869c-272cb161d239","Type":"ContainerDied","Data":"a6cab79583941220c62a21f1484e8466be8b0b7b875e8016ee83db432d1d7613"} Jul 17 06:38:43.481485 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:43.481278 2580 scope.go:117] "RemoveContainer" containerID="7d5bdf3c0a6e8f39f9828999128d2122a3de3d803cc563eaa2ff665526a9fc93" Jul 17 06:38:43.481485 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:43.481476 2580 scope.go:117] "RemoveContainer" containerID="a6cab79583941220c62a21f1484e8466be8b0b7b875e8016ee83db432d1d7613" Jul 17 06:38:43.481749 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:38:43.481720 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-rtttk_odh-ai-gateway-infra(f72c42d0-1a86-40da-869c-272cb161d239)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" podUID="f72c42d0-1a86-40da-869c-272cb161d239" Jul 17 06:38:44.488683 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:44.488653 2580 scope.go:117] "RemoveContainer" containerID="a6cab79583941220c62a21f1484e8466be8b0b7b875e8016ee83db432d1d7613" Jul 17 06:38:44.489083 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:38:44.488832 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-rtttk_odh-ai-gateway-infra(f72c42d0-1a86-40da-869c-272cb161d239)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" podUID="f72c42d0-1a86-40da-869c-272cb161d239" Jul 17 06:38:45.880403 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:45.880347 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:38:56.886589 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:56.886559 2580 scope.go:117] "RemoveContainer" containerID="a6cab79583941220c62a21f1484e8466be8b0b7b875e8016ee83db432d1d7613" Jul 17 06:38:57.550679 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:57.550641 2580 generic.go:356] "Generic (PLEG): container finished" podID="f72c42d0-1a86-40da-869c-272cb161d239" containerID="385c076c5ddd8024672ab80315899cb59f8a3a906d5e50dab7e32683343fbbe0" exitCode=6 Jul 17 06:38:57.550859 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:57.550711 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" event={"ID":"f72c42d0-1a86-40da-869c-272cb161d239","Type":"ContainerDied","Data":"385c076c5ddd8024672ab80315899cb59f8a3a906d5e50dab7e32683343fbbe0"} Jul 17 06:38:57.550859 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:57.550804 2580 scope.go:117] "RemoveContainer" containerID="a6cab79583941220c62a21f1484e8466be8b0b7b875e8016ee83db432d1d7613" Jul 17 06:38:57.551090 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:57.551072 2580 scope.go:117] "RemoveContainer" containerID="385c076c5ddd8024672ab80315899cb59f8a3a906d5e50dab7e32683343fbbe0" Jul 17 06:38:57.551302 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:38:57.551281 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-rtttk_odh-ai-gateway-infra(f72c42d0-1a86-40da-869c-272cb161d239)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" podUID="f72c42d0-1a86-40da-869c-272cb161d239" Jul 17 06:38:58.513799 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:58.513766 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:38:58.575984 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:58.575952 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk"] Jul 17 06:38:58.708549 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:58.708522 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" Jul 17 06:38:58.726744 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:58.726696 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c42d0-1a86-40da-869c-272cb161d239-maas-api-service-ca\") pod \"f72c42d0-1a86-40da-869c-272cb161d239\" (UID: \"f72c42d0-1a86-40da-869c-272cb161d239\") " Jul 17 06:38:58.727133 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:58.727103 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72c42d0-1a86-40da-869c-272cb161d239-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f72c42d0-1a86-40da-869c-272cb161d239" (UID: "f72c42d0-1a86-40da-869c-272cb161d239"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:38:58.828069 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:58.827974 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72c42d0-1a86-40da-869c-272cb161d239-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:38:59.563108 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:59.563073 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" event={"ID":"f72c42d0-1a86-40da-869c-272cb161d239","Type":"ContainerDied","Data":"22356dafd4cb0174ca37fdb900b13be7201ebe28cfae02da581186932e05df07"} Jul 17 06:38:59.563108 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:59.563099 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk" Jul 17 06:38:59.563623 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:59.563133 2580 scope.go:117] "RemoveContainer" containerID="385c076c5ddd8024672ab80315899cb59f8a3a906d5e50dab7e32683343fbbe0" Jul 17 06:38:59.581315 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:59.581288 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk"] Jul 17 06:38:59.586906 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:38:59.586882 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-rtttk"] Jul 17 06:39:00.889627 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:00.889595 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f72c42d0-1a86-40da-869c-272cb161d239" path="/var/lib/kubelet/pods/f72c42d0-1a86-40da-869c-272cb161d239/volumes" Jul 17 06:39:03.077775 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:03.077735 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:39:05.563670 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:05.563635 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:39:13.448134 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448101 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n"] Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448478 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e859cf-8268-4769-b0f5-72c770970acb" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448491 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e859cf-8268-4769-b0f5-72c770970acb" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448504 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea75135e-c561-43c5-8bda-8fc2bb760b77" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448513 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea75135e-c561-43c5-8bda-8fc2bb760b77" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448522 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf9113a-56d3-4688-b5b9-148329d04c60" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448529 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf9113a-56d3-4688-b5b9-148329d04c60" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448536 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72c42d0-1a86-40da-869c-272cb161d239" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448542 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72c42d0-1a86-40da-869c-272cb161d239" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448548 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72c42d0-1a86-40da-869c-272cb161d239" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448553 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72c42d0-1a86-40da-869c-272cb161d239" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448561 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72c42d0-1a86-40da-869c-272cb161d239" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448566 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72c42d0-1a86-40da-869c-272cb161d239" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448572 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf9113a-56d3-4688-b5b9-148329d04c60" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448577 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf9113a-56d3-4688-b5b9-148329d04c60" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448584 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf9113a-56d3-4688-b5b9-148329d04c60" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448592 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf9113a-56d3-4688-b5b9-148329d04c60" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448612 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e859cf-8268-4769-b0f5-72c770970acb" containerName="revoke-keys" Jul 17 06:39:13.448631 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448617 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e859cf-8268-4769-b0f5-72c770970acb" containerName="revoke-keys" Jul 17 06:39:13.449174 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448666 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cf9113a-56d3-4688-b5b9-148329d04c60" containerName="revoke-keys" Jul 17 06:39:13.449174 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448677 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cf9113a-56d3-4688-b5b9-148329d04c60" containerName="revoke-keys" Jul 17 06:39:13.449174 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448684 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e859cf-8268-4769-b0f5-72c770970acb" containerName="revoke-keys" Jul 17 06:39:13.449174 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448690 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e859cf-8268-4769-b0f5-72c770970acb" containerName="revoke-keys" Jul 17 06:39:13.449174 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448696 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea75135e-c561-43c5-8bda-8fc2bb760b77" containerName="revoke-keys" Jul 17 06:39:13.449174 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448701 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72c42d0-1a86-40da-869c-272cb161d239" containerName="revoke-keys" Jul 17 06:39:13.449174 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448706 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72c42d0-1a86-40da-869c-272cb161d239" containerName="revoke-keys" Jul 17 06:39:13.449174 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.448712 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cf9113a-56d3-4688-b5b9-148329d04c60" containerName="revoke-keys" Jul 17 06:39:13.453010 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.452986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" Jul 17 06:39:13.455837 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.455814 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:39:13.460562 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.460466 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n"] Jul 17 06:39:13.557132 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.557086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44040a-8943-4bb3-8700-39abd26a4250-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-4zq7n\" (UID: \"1b44040a-8943-4bb3-8700-39abd26a4250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" Jul 17 06:39:13.658131 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.658084 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44040a-8943-4bb3-8700-39abd26a4250-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-4zq7n\" (UID: \"1b44040a-8943-4bb3-8700-39abd26a4250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" Jul 17 06:39:13.658756 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.658735 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44040a-8943-4bb3-8700-39abd26a4250-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-4zq7n\" (UID: \"1b44040a-8943-4bb3-8700-39abd26a4250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" Jul 17 06:39:13.764064 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.763978 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" Jul 17 06:39:13.871670 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.871635 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:39:13.893818 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.893743 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n"] Jul 17 06:39:13.896217 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:39:13.896191 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b44040a_8943_4bb3_8700_39abd26a4250.slice/crio-b99c23bc93b1a5c32e216a669e742c7521a5fbacc8a07be8334f373809be4a68 WatchSource:0}: Error finding container b99c23bc93b1a5c32e216a669e742c7521a5fbacc8a07be8334f373809be4a68: Status 404 returned error can't find the container with id b99c23bc93b1a5c32e216a669e742c7521a5fbacc8a07be8334f373809be4a68 Jul 17 06:39:13.972200 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:13.972160 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:39:14.572966 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.572934 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45"] Jul 17 06:39:14.573335 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.573276 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e859cf-8268-4769-b0f5-72c770970acb" containerName="revoke-keys" Jul 17 06:39:14.573335 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.573287 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e859cf-8268-4769-b0f5-72c770970acb" containerName="revoke-keys" Jul 17 06:39:14.573434 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.573351 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e859cf-8268-4769-b0f5-72c770970acb" containerName="revoke-keys" Jul 17 06:39:14.573434 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.573359 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72c42d0-1a86-40da-869c-272cb161d239" containerName="revoke-keys" Jul 17 06:39:14.576439 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.576421 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" Jul 17 06:39:14.583239 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.583208 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45"] Jul 17 06:39:14.635356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.635321 2580 generic.go:356] "Generic (PLEG): container finished" podID="1b44040a-8943-4bb3-8700-39abd26a4250" containerID="ba0910cd2b2e3df90401751828d527ae890aa0e518923dd35662fb638d95a694" exitCode=6 Jul 17 06:39:14.635356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.635360 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" event={"ID":"1b44040a-8943-4bb3-8700-39abd26a4250","Type":"ContainerDied","Data":"ba0910cd2b2e3df90401751828d527ae890aa0e518923dd35662fb638d95a694"} Jul 17 06:39:14.635589 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.635412 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" event={"ID":"1b44040a-8943-4bb3-8700-39abd26a4250","Type":"ContainerStarted","Data":"b99c23bc93b1a5c32e216a669e742c7521a5fbacc8a07be8334f373809be4a68"} Jul 17 06:39:14.635589 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.635563 2580 scope.go:117] "RemoveContainer" containerID="ba0910cd2b2e3df90401751828d527ae890aa0e518923dd35662fb638d95a694" Jul 17 06:39:14.668454 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.668428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e82a2bc-a447-4e5e-8dbc-7f88512328d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-lmh45\" (UID: \"6e82a2bc-a447-4e5e-8dbc-7f88512328d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" Jul 17 06:39:14.769066 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.769033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e82a2bc-a447-4e5e-8dbc-7f88512328d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-lmh45\" (UID: \"6e82a2bc-a447-4e5e-8dbc-7f88512328d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" Jul 17 06:39:14.769725 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.769702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e82a2bc-a447-4e5e-8dbc-7f88512328d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-lmh45\" (UID: \"6e82a2bc-a447-4e5e-8dbc-7f88512328d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" Jul 17 06:39:14.903707 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:14.903669 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" Jul 17 06:39:15.041745 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:15.041720 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45"] Jul 17 06:39:15.043794 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:39:15.043769 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e82a2bc_a447_4e5e_8dbc_7f88512328d3.slice/crio-36a07aba8a33e02910e5ceb007231a45e85fea03a2d07978bb02ce010c5b21ff WatchSource:0}: Error finding container 36a07aba8a33e02910e5ceb007231a45e85fea03a2d07978bb02ce010c5b21ff: Status 404 returned error can't find the container with id 36a07aba8a33e02910e5ceb007231a45e85fea03a2d07978bb02ce010c5b21ff Jul 17 06:39:15.643265 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:15.643234 2580 generic.go:356] "Generic (PLEG): container finished" podID="1b44040a-8943-4bb3-8700-39abd26a4250" containerID="1c0fc46dc8b3007cf380dec2a224d7f375fcb698dbe48e45fa5cde3022d8e4b1" exitCode=6 Jul 17 06:39:15.643669 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:15.643359 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" event={"ID":"1b44040a-8943-4bb3-8700-39abd26a4250","Type":"ContainerDied","Data":"1c0fc46dc8b3007cf380dec2a224d7f375fcb698dbe48e45fa5cde3022d8e4b1"} Jul 17 06:39:15.643669 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:15.643446 2580 scope.go:117] "RemoveContainer" containerID="ba0910cd2b2e3df90401751828d527ae890aa0e518923dd35662fb638d95a694" Jul 17 06:39:15.643669 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:15.643592 2580 scope.go:117] "RemoveContainer" containerID="1c0fc46dc8b3007cf380dec2a224d7f375fcb698dbe48e45fa5cde3022d8e4b1" Jul 17 06:39:15.643840 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:39:15.643813 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-4zq7n_odh-ai-gateway-infra(1b44040a-8943-4bb3-8700-39abd26a4250)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" podUID="1b44040a-8943-4bb3-8700-39abd26a4250" Jul 17 06:39:15.645326 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:15.645301 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" event={"ID":"6e82a2bc-a447-4e5e-8dbc-7f88512328d3","Type":"ContainerStarted","Data":"c7ac4ba576263e8e72ac9efc9ed67aaa640037d3bfea7d218b0e1baf0b1b0d02"} Jul 17 06:39:15.645457 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:15.645361 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" event={"ID":"6e82a2bc-a447-4e5e-8dbc-7f88512328d3","Type":"ContainerStarted","Data":"36a07aba8a33e02910e5ceb007231a45e85fea03a2d07978bb02ce010c5b21ff"} Jul 17 06:39:15.672106 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:15.672011 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" podStartSLOduration=1.671995158 podStartE2EDuration="1.671995158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:39:15.671575031 +0000 UTC m=+3480.416254360" watchObservedRunningTime="2026-07-17 06:39:15.671995158 +0000 UTC m=+3480.416674540" Jul 17 06:39:16.651959 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:16.651920 2580 generic.go:356] "Generic (PLEG): container finished" podID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" containerID="c7ac4ba576263e8e72ac9efc9ed67aaa640037d3bfea7d218b0e1baf0b1b0d02" exitCode=6 Jul 17 06:39:16.652450 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:16.652010 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" event={"ID":"6e82a2bc-a447-4e5e-8dbc-7f88512328d3","Type":"ContainerDied","Data":"c7ac4ba576263e8e72ac9efc9ed67aaa640037d3bfea7d218b0e1baf0b1b0d02"} Jul 17 06:39:16.652450 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:16.652172 2580 scope.go:117] "RemoveContainer" containerID="c7ac4ba576263e8e72ac9efc9ed67aaa640037d3bfea7d218b0e1baf0b1b0d02" Jul 17 06:39:16.653779 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:16.653764 2580 scope.go:117] "RemoveContainer" containerID="1c0fc46dc8b3007cf380dec2a224d7f375fcb698dbe48e45fa5cde3022d8e4b1" Jul 17 06:39:16.653929 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:39:16.653912 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-4zq7n_odh-ai-gateway-infra(1b44040a-8943-4bb3-8700-39abd26a4250)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" podUID="1b44040a-8943-4bb3-8700-39abd26a4250" Jul 17 06:39:16.699400 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:16.699349 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb"] Jul 17 06:39:16.704402 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:16.704360 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" Jul 17 06:39:16.710746 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:16.710685 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb"] Jul 17 06:39:16.785992 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:16.785962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da561351-46ad-4764-8261-15c12e02a1fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-twckb\" (UID: \"da561351-46ad-4764-8261-15c12e02a1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" Jul 17 06:39:16.886520 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:16.886493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da561351-46ad-4764-8261-15c12e02a1fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-twckb\" (UID: \"da561351-46ad-4764-8261-15c12e02a1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" Jul 17 06:39:16.887234 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:16.887199 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da561351-46ad-4764-8261-15c12e02a1fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-twckb\" (UID: \"da561351-46ad-4764-8261-15c12e02a1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" Jul 17 06:39:17.017114 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:17.017077 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" Jul 17 06:39:17.147759 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:17.147731 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb"] Jul 17 06:39:17.149602 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:39:17.149576 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda561351_46ad_4764_8261_15c12e02a1fb.slice/crio-0cff7b493d40919e6b6537bfd147f5d19820b11d498abb6b36661d8a5399f3e0 WatchSource:0}: Error finding container 0cff7b493d40919e6b6537bfd147f5d19820b11d498abb6b36661d8a5399f3e0: Status 404 returned error can't find the container with id 0cff7b493d40919e6b6537bfd147f5d19820b11d498abb6b36661d8a5399f3e0 Jul 17 06:39:17.662982 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:17.662939 2580 generic.go:356] "Generic (PLEG): container finished" podID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" containerID="541b95e43896b7dc9397770c5683a7f46ec14ab178a029c9cb194a702e4d75b8" exitCode=6 Jul 17 06:39:17.663462 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:17.663019 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" event={"ID":"6e82a2bc-a447-4e5e-8dbc-7f88512328d3","Type":"ContainerDied","Data":"541b95e43896b7dc9397770c5683a7f46ec14ab178a029c9cb194a702e4d75b8"} Jul 17 06:39:17.663462 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:17.663081 2580 scope.go:117] "RemoveContainer" containerID="c7ac4ba576263e8e72ac9efc9ed67aaa640037d3bfea7d218b0e1baf0b1b0d02" Jul 17 06:39:17.663462 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:17.663276 2580 scope.go:117] "RemoveContainer" containerID="541b95e43896b7dc9397770c5683a7f46ec14ab178a029c9cb194a702e4d75b8" Jul 17 06:39:17.663651 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:39:17.663538 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-lmh45_odh-ai-gateway-infra(6e82a2bc-a447-4e5e-8dbc-7f88512328d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" podUID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" Jul 17 06:39:17.664676 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:17.664654 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" event={"ID":"da561351-46ad-4764-8261-15c12e02a1fb","Type":"ContainerStarted","Data":"df22f83b171c16bd2f5df3e6b7fcd32a8be281a5010498339e2633e577ac6753"} Jul 17 06:39:17.664766 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:17.664698 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" event={"ID":"da561351-46ad-4764-8261-15c12e02a1fb","Type":"ContainerStarted","Data":"0cff7b493d40919e6b6537bfd147f5d19820b11d498abb6b36661d8a5399f3e0"} Jul 17 06:39:17.696551 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:17.696502 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" podStartSLOduration=1.696487254 podStartE2EDuration="1.696487254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:39:17.695003577 +0000 UTC m=+3482.439682904" watchObservedRunningTime="2026-07-17 06:39:17.696487254 +0000 UTC m=+3482.441166640" Jul 17 06:39:18.672117 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:18.672083 2580 scope.go:117] "RemoveContainer" containerID="541b95e43896b7dc9397770c5683a7f46ec14ab178a029c9cb194a702e4d75b8" Jul 17 06:39:18.672574 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:39:18.672292 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-lmh45_odh-ai-gateway-infra(6e82a2bc-a447-4e5e-8dbc-7f88512328d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" podUID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" Jul 17 06:39:18.673237 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:18.673211 2580 generic.go:356] "Generic (PLEG): container finished" podID="da561351-46ad-4764-8261-15c12e02a1fb" containerID="df22f83b171c16bd2f5df3e6b7fcd32a8be281a5010498339e2633e577ac6753" exitCode=6 Jul 17 06:39:18.673360 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:18.673241 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" event={"ID":"da561351-46ad-4764-8261-15c12e02a1fb","Type":"ContainerDied","Data":"df22f83b171c16bd2f5df3e6b7fcd32a8be281a5010498339e2633e577ac6753"} Jul 17 06:39:18.673443 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:18.673428 2580 scope.go:117] "RemoveContainer" containerID="df22f83b171c16bd2f5df3e6b7fcd32a8be281a5010498339e2633e577ac6753" Jul 17 06:39:19.679879 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:19.679844 2580 generic.go:356] "Generic (PLEG): container finished" podID="da561351-46ad-4764-8261-15c12e02a1fb" containerID="afb19ea2950ce5146d981b8f624cd4285d3a01af8e6eb4779a76234c41f8a4bc" exitCode=6 Jul 17 06:39:19.680300 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:19.679897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" event={"ID":"da561351-46ad-4764-8261-15c12e02a1fb","Type":"ContainerDied","Data":"afb19ea2950ce5146d981b8f624cd4285d3a01af8e6eb4779a76234c41f8a4bc"} Jul 17 06:39:19.680300 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:19.679955 2580 scope.go:117] "RemoveContainer" containerID="df22f83b171c16bd2f5df3e6b7fcd32a8be281a5010498339e2633e577ac6753" Jul 17 06:39:19.680300 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:19.680165 2580 scope.go:117] "RemoveContainer" containerID="afb19ea2950ce5146d981b8f624cd4285d3a01af8e6eb4779a76234c41f8a4bc" Jul 17 06:39:19.680480 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:39:19.680353 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-twckb_odh-ai-gateway-infra(da561351-46ad-4764-8261-15c12e02a1fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" podUID="da561351-46ad-4764-8261-15c12e02a1fb" Jul 17 06:39:20.686488 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:20.686455 2580 scope.go:117] "RemoveContainer" containerID="afb19ea2950ce5146d981b8f624cd4285d3a01af8e6eb4779a76234c41f8a4bc" Jul 17 06:39:20.686894 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:39:20.686674 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-twckb_odh-ai-gateway-infra(da561351-46ad-4764-8261-15c12e02a1fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" podUID="da561351-46ad-4764-8261-15c12e02a1fb" Jul 17 06:39:24.710741 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:24.710708 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:39:28.677457 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:28.677412 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:39:29.884348 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:29.884316 2580 scope.go:117] "RemoveContainer" containerID="1c0fc46dc8b3007cf380dec2a224d7f375fcb698dbe48e45fa5cde3022d8e4b1" Jul 17 06:39:30.608523 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:30.608488 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc"] Jul 17 06:39:30.611941 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:30.611922 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" Jul 17 06:39:30.619161 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:30.618986 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc"] Jul 17 06:39:30.708937 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:30.708899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b18e65-acf8-46e4-b630-c1ea06ce8df5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc\" (UID: \"18b18e65-acf8-46e4-b630-c1ea06ce8df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" Jul 17 06:39:30.737303 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:30.737269 2580 generic.go:356] "Generic (PLEG): container finished" podID="1b44040a-8943-4bb3-8700-39abd26a4250" containerID="a1d541ada1cc587dfd06a98c9ad8def364235a5da776322397c91453db99b3ca" exitCode=6 Jul 17 06:39:30.737488 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:30.737358 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" event={"ID":"1b44040a-8943-4bb3-8700-39abd26a4250","Type":"ContainerDied","Data":"a1d541ada1cc587dfd06a98c9ad8def364235a5da776322397c91453db99b3ca"} Jul 17 06:39:30.737488 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:30.737452 2580 scope.go:117] "RemoveContainer" containerID="1c0fc46dc8b3007cf380dec2a224d7f375fcb698dbe48e45fa5cde3022d8e4b1" Jul 17 06:39:30.737790 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:30.737756 2580 scope.go:117] "RemoveContainer" containerID="a1d541ada1cc587dfd06a98c9ad8def364235a5da776322397c91453db99b3ca" Jul 17 06:39:30.738037 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:39:30.738013 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-4zq7n_odh-ai-gateway-infra(1b44040a-8943-4bb3-8700-39abd26a4250)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" podUID="1b44040a-8943-4bb3-8700-39abd26a4250" Jul 17 06:39:30.810143 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:30.810060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b18e65-acf8-46e4-b630-c1ea06ce8df5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc\" (UID: \"18b18e65-acf8-46e4-b630-c1ea06ce8df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" Jul 17 06:39:30.810735 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:30.810715 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b18e65-acf8-46e4-b630-c1ea06ce8df5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc\" (UID: \"18b18e65-acf8-46e4-b630-c1ea06ce8df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" Jul 17 06:39:30.923495 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:30.923461 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" Jul 17 06:39:31.055616 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:31.055594 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc"] Jul 17 06:39:31.057336 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:39:31.057305 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b18e65_acf8_46e4_b630_c1ea06ce8df5.slice/crio-d39a92edda0b2157ba7778a49f5468a8d5f684883b6e992c7ce9878a3183eb17 WatchSource:0}: Error finding container d39a92edda0b2157ba7778a49f5468a8d5f684883b6e992c7ce9878a3183eb17: Status 404 returned error can't find the container with id d39a92edda0b2157ba7778a49f5468a8d5f684883b6e992c7ce9878a3183eb17 Jul 17 06:39:31.744759 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:31.744723 2580 generic.go:356] "Generic (PLEG): container finished" podID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" containerID="e9c0f9b5cd2b7bee57f190889b354c4bccfe099f4cb583f07278b1c36659b978" exitCode=6 Jul 17 06:39:31.744887 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:31.744769 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" event={"ID":"18b18e65-acf8-46e4-b630-c1ea06ce8df5","Type":"ContainerDied","Data":"e9c0f9b5cd2b7bee57f190889b354c4bccfe099f4cb583f07278b1c36659b978"} Jul 17 06:39:31.744887 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:31.744832 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" event={"ID":"18b18e65-acf8-46e4-b630-c1ea06ce8df5","Type":"ContainerStarted","Data":"d39a92edda0b2157ba7778a49f5468a8d5f684883b6e992c7ce9878a3183eb17"} Jul 17 06:39:31.745007 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:31.744995 2580 scope.go:117] "RemoveContainer" containerID="e9c0f9b5cd2b7bee57f190889b354c4bccfe099f4cb583f07278b1c36659b978" Jul 17 06:39:31.761726 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:31.761699 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n"] Jul 17 06:39:31.879126 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:31.879102 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" Jul 17 06:39:31.883831 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:31.883811 2580 scope.go:117] "RemoveContainer" containerID="afb19ea2950ce5146d981b8f624cd4285d3a01af8e6eb4779a76234c41f8a4bc" Jul 17 06:39:32.021886 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.021856 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44040a-8943-4bb3-8700-39abd26a4250-maas-api-service-ca\") pod \"1b44040a-8943-4bb3-8700-39abd26a4250\" (UID: \"1b44040a-8943-4bb3-8700-39abd26a4250\") " Jul 17 06:39:32.022244 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.022172 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b44040a-8943-4bb3-8700-39abd26a4250-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b44040a-8943-4bb3-8700-39abd26a4250" (UID: "1b44040a-8943-4bb3-8700-39abd26a4250"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:39:32.122819 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.122765 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b44040a-8943-4bb3-8700-39abd26a4250-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:39:32.751045 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.750958 2580 generic.go:356] "Generic (PLEG): container finished" podID="da561351-46ad-4764-8261-15c12e02a1fb" containerID="804705bed8313533fe2283f0be8fd2aa5bdb14343e8e66a80331d3773539820d" exitCode=6 Jul 17 06:39:32.751045 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.751029 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" event={"ID":"da561351-46ad-4764-8261-15c12e02a1fb","Type":"ContainerDied","Data":"804705bed8313533fe2283f0be8fd2aa5bdb14343e8e66a80331d3773539820d"} Jul 17 06:39:32.751253 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.751107 2580 scope.go:117] "RemoveContainer" containerID="afb19ea2950ce5146d981b8f624cd4285d3a01af8e6eb4779a76234c41f8a4bc" Jul 17 06:39:32.751380 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.751363 2580 scope.go:117] "RemoveContainer" containerID="804705bed8313533fe2283f0be8fd2aa5bdb14343e8e66a80331d3773539820d" Jul 17 06:39:32.751644 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:39:32.751615 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-twckb_odh-ai-gateway-infra(da561351-46ad-4764-8261-15c12e02a1fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" podUID="da561351-46ad-4764-8261-15c12e02a1fb" Jul 17 06:39:32.752948 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.752922 2580 generic.go:356] "Generic (PLEG): container finished" podID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" containerID="026af2694c7bb93fdabe3492b280c61d401e87085d5680c1544078e11103e3f7" exitCode=6 Jul 17 06:39:32.753033 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.752987 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" event={"ID":"18b18e65-acf8-46e4-b630-c1ea06ce8df5","Type":"ContainerDied","Data":"026af2694c7bb93fdabe3492b280c61d401e87085d5680c1544078e11103e3f7"} Jul 17 06:39:32.753177 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.753164 2580 scope.go:117] "RemoveContainer" containerID="026af2694c7bb93fdabe3492b280c61d401e87085d5680c1544078e11103e3f7" Jul 17 06:39:32.753380 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:39:32.753361 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc_odh-ai-gateway-infra(18b18e65-acf8-46e4-b630-c1ea06ce8df5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" podUID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" Jul 17 06:39:32.754216 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.754196 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" event={"ID":"1b44040a-8943-4bb3-8700-39abd26a4250","Type":"ContainerDied","Data":"b99c23bc93b1a5c32e216a669e742c7521a5fbacc8a07be8334f373809be4a68"} Jul 17 06:39:32.754292 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.754203 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n" Jul 17 06:39:32.763192 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.763176 2580 scope.go:117] "RemoveContainer" containerID="e9c0f9b5cd2b7bee57f190889b354c4bccfe099f4cb583f07278b1c36659b978" Jul 17 06:39:32.774350 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.774331 2580 scope.go:117] "RemoveContainer" containerID="a1d541ada1cc587dfd06a98c9ad8def364235a5da776322397c91453db99b3ca" Jul 17 06:39:32.793703 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.793681 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n"] Jul 17 06:39:32.797221 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.797198 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4zq7n"] Jul 17 06:39:32.883802 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.883772 2580 scope.go:117] "RemoveContainer" containerID="541b95e43896b7dc9397770c5683a7f46ec14ab178a029c9cb194a702e4d75b8" Jul 17 06:39:32.889429 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:32.889407 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b44040a-8943-4bb3-8700-39abd26a4250" path="/var/lib/kubelet/pods/1b44040a-8943-4bb3-8700-39abd26a4250/volumes" Jul 17 06:39:33.761329 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:33.761249 2580 generic.go:356] "Generic (PLEG): container finished" podID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" containerID="76ab9f2a2ecd8e7e1859f6a96039180017033516a9d73802c46160d2b84a4a22" exitCode=6 Jul 17 06:39:33.761812 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:33.761329 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" event={"ID":"6e82a2bc-a447-4e5e-8dbc-7f88512328d3","Type":"ContainerDied","Data":"76ab9f2a2ecd8e7e1859f6a96039180017033516a9d73802c46160d2b84a4a22"} Jul 17 06:39:33.761812 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:33.761412 2580 scope.go:117] "RemoveContainer" containerID="541b95e43896b7dc9397770c5683a7f46ec14ab178a029c9cb194a702e4d75b8" Jul 17 06:39:33.761812 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:33.761708 2580 scope.go:117] "RemoveContainer" containerID="76ab9f2a2ecd8e7e1859f6a96039180017033516a9d73802c46160d2b84a4a22" Jul 17 06:39:33.761995 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:39:33.761899 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-lmh45_odh-ai-gateway-infra(6e82a2bc-a447-4e5e-8dbc-7f88512328d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" podUID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" Jul 17 06:39:33.764857 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:33.764828 2580 scope.go:117] "RemoveContainer" containerID="026af2694c7bb93fdabe3492b280c61d401e87085d5680c1544078e11103e3f7" Jul 17 06:39:33.765032 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:39:33.765005 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc_odh-ai-gateway-infra(18b18e65-acf8-46e4-b630-c1ea06ce8df5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" podUID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" Jul 17 06:39:33.779541 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:33.779520 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb"] Jul 17 06:39:33.912563 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:33.912541 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" Jul 17 06:39:34.041534 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:34.041460 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da561351-46ad-4764-8261-15c12e02a1fb-maas-api-service-ca\") pod \"da561351-46ad-4764-8261-15c12e02a1fb\" (UID: \"da561351-46ad-4764-8261-15c12e02a1fb\") " Jul 17 06:39:34.041757 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:34.041737 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da561351-46ad-4764-8261-15c12e02a1fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da561351-46ad-4764-8261-15c12e02a1fb" (UID: "da561351-46ad-4764-8261-15c12e02a1fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:39:34.143133 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:34.143103 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da561351-46ad-4764-8261-15c12e02a1fb-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:39:34.773267 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:34.773237 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" Jul 17 06:39:34.773686 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:34.773235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb" event={"ID":"da561351-46ad-4764-8261-15c12e02a1fb","Type":"ContainerDied","Data":"0cff7b493d40919e6b6537bfd147f5d19820b11d498abb6b36661d8a5399f3e0"} Jul 17 06:39:34.773686 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:34.773395 2580 scope.go:117] "RemoveContainer" containerID="804705bed8313533fe2283f0be8fd2aa5bdb14343e8e66a80331d3773539820d" Jul 17 06:39:34.788189 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:34.788166 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45"] Jul 17 06:39:34.794527 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:34.794505 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb"] Jul 17 06:39:34.801738 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:34.801719 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-twckb"] Jul 17 06:39:34.893117 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:34.892732 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da561351-46ad-4764-8261-15c12e02a1fb" path="/var/lib/kubelet/pods/da561351-46ad-4764-8261-15c12e02a1fb/volumes" Jul 17 06:39:34.929930 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:34.929903 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" Jul 17 06:39:35.052007 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:35.051936 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e82a2bc-a447-4e5e-8dbc-7f88512328d3-maas-api-service-ca\") pod \"6e82a2bc-a447-4e5e-8dbc-7f88512328d3\" (UID: \"6e82a2bc-a447-4e5e-8dbc-7f88512328d3\") " Jul 17 06:39:35.052274 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:35.052254 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e82a2bc-a447-4e5e-8dbc-7f88512328d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e82a2bc-a447-4e5e-8dbc-7f88512328d3" (UID: "6e82a2bc-a447-4e5e-8dbc-7f88512328d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:39:35.152905 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:35.152860 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e82a2bc-a447-4e5e-8dbc-7f88512328d3-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:39:35.779496 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:35.779458 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" Jul 17 06:39:35.779956 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:35.779489 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45" event={"ID":"6e82a2bc-a447-4e5e-8dbc-7f88512328d3","Type":"ContainerDied","Data":"36a07aba8a33e02910e5ceb007231a45e85fea03a2d07978bb02ce010c5b21ff"} Jul 17 06:39:35.779956 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:35.779588 2580 scope.go:117] "RemoveContainer" containerID="76ab9f2a2ecd8e7e1859f6a96039180017033516a9d73802c46160d2b84a4a22" Jul 17 06:39:35.804328 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:35.804297 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45"] Jul 17 06:39:35.806415 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:35.806392 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-lmh45"] Jul 17 06:39:36.889013 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:36.888980 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" path="/var/lib/kubelet/pods/6e82a2bc-a447-4e5e-8dbc-7f88512328d3/volumes" Jul 17 06:39:39.578853 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:39.578816 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:39:46.891831 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:46.889060 2580 scope.go:117] "RemoveContainer" containerID="026af2694c7bb93fdabe3492b280c61d401e87085d5680c1544078e11103e3f7" Jul 17 06:39:47.842163 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:47.842079 2580 generic.go:356] "Generic (PLEG): container finished" podID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" containerID="467e8ae6b3efde668753cfa6f9253110be550dc7ec0d6ad7418aabd16ebd8f83" exitCode=6 Jul 17 06:39:47.842163 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:47.842131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" event={"ID":"18b18e65-acf8-46e4-b630-c1ea06ce8df5","Type":"ContainerDied","Data":"467e8ae6b3efde668753cfa6f9253110be550dc7ec0d6ad7418aabd16ebd8f83"} Jul 17 06:39:47.842341 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:47.842177 2580 scope.go:117] "RemoveContainer" containerID="026af2694c7bb93fdabe3492b280c61d401e87085d5680c1544078e11103e3f7" Jul 17 06:39:47.842502 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:47.842476 2580 scope.go:117] "RemoveContainer" containerID="467e8ae6b3efde668753cfa6f9253110be550dc7ec0d6ad7418aabd16ebd8f83" Jul 17 06:39:47.842710 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:39:47.842691 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc_odh-ai-gateway-infra(18b18e65-acf8-46e4-b630-c1ea06ce8df5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" podUID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" Jul 17 06:39:48.867555 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:48.867525 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc"] Jul 17 06:39:48.997348 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:48.997326 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" Jul 17 06:39:49.068373 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:49.068336 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b18e65-acf8-46e4-b630-c1ea06ce8df5-maas-api-service-ca\") pod \"18b18e65-acf8-46e4-b630-c1ea06ce8df5\" (UID: \"18b18e65-acf8-46e4-b630-c1ea06ce8df5\") " Jul 17 06:39:49.068658 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:49.068634 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b18e65-acf8-46e4-b630-c1ea06ce8df5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18b18e65-acf8-46e4-b630-c1ea06ce8df5" (UID: "18b18e65-acf8-46e4-b630-c1ea06ce8df5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:39:49.169927 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:49.169848 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b18e65-acf8-46e4-b630-c1ea06ce8df5-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:39:49.854521 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:49.854493 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" Jul 17 06:39:49.854677 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:49.854532 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc" event={"ID":"18b18e65-acf8-46e4-b630-c1ea06ce8df5","Type":"ContainerDied","Data":"d39a92edda0b2157ba7778a49f5468a8d5f684883b6e992c7ce9878a3183eb17"} Jul 17 06:39:49.854677 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:49.854589 2580 scope.go:117] "RemoveContainer" containerID="467e8ae6b3efde668753cfa6f9253110be550dc7ec0d6ad7418aabd16ebd8f83" Jul 17 06:39:49.875304 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:49.875280 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc"] Jul 17 06:39:49.880180 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:49.880160 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-5jxcc"] Jul 17 06:39:50.888437 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:50.888404 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" path="/var/lib/kubelet/pods/18b18e65-acf8-46e4-b630-c1ea06ce8df5/volumes" Jul 17 06:39:55.994829 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:55.994797 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:39:56.910497 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:39:56.910466 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:40:00.880439 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:00.880401 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:40:03.489138 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489103 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6"] Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489447 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b44040a-8943-4bb3-8700-39abd26a4250" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489459 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b44040a-8943-4bb3-8700-39abd26a4250" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489469 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489474 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489480 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da561351-46ad-4764-8261-15c12e02a1fb" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489485 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da561351-46ad-4764-8261-15c12e02a1fb" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489494 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da561351-46ad-4764-8261-15c12e02a1fb" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489499 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da561351-46ad-4764-8261-15c12e02a1fb" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489505 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da561351-46ad-4764-8261-15c12e02a1fb" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489510 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da561351-46ad-4764-8261-15c12e02a1fb" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489516 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489521 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489526 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489532 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489539 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b44040a-8943-4bb3-8700-39abd26a4250" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489544 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b44040a-8943-4bb3-8700-39abd26a4250" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489552 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489557 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489609 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da561351-46ad-4764-8261-15c12e02a1fb" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489616 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b44040a-8943-4bb3-8700-39abd26a4250" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489623 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489628 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489634 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" containerName="revoke-keys" Jul 17 06:40:03.489630 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489639 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" containerName="revoke-keys" Jul 17 06:40:03.490350 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489644 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" containerName="revoke-keys" Jul 17 06:40:03.490350 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489650 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da561351-46ad-4764-8261-15c12e02a1fb" containerName="revoke-keys" Jul 17 06:40:03.490350 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.489656 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b44040a-8943-4bb3-8700-39abd26a4250" containerName="revoke-keys" Jul 17 06:40:03.492741 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.492724 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" Jul 17 06:40:03.495326 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.495305 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:40:03.500529 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.500508 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6"] Jul 17 06:40:03.582579 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.582550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b52f47-5eb2-4a52-82fd-a8f5a84573f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-f8bk6\" (UID: \"b9b52f47-5eb2-4a52-82fd-a8f5a84573f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" Jul 17 06:40:03.683005 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.682973 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b52f47-5eb2-4a52-82fd-a8f5a84573f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-f8bk6\" (UID: \"b9b52f47-5eb2-4a52-82fd-a8f5a84573f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" Jul 17 06:40:03.683609 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.683588 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b52f47-5eb2-4a52-82fd-a8f5a84573f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-f8bk6\" (UID: \"b9b52f47-5eb2-4a52-82fd-a8f5a84573f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" Jul 17 06:40:03.804275 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.804185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" Jul 17 06:40:03.928096 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:03.928068 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6"] Jul 17 06:40:03.930023 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:40:03.929992 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b52f47_5eb2_4a52_82fd_a8f5a84573f9.slice/crio-219063f3078eedd7fc460b6ea8123bbdd5002529ac2405dd5079ab57ee207fba WatchSource:0}: Error finding container 219063f3078eedd7fc460b6ea8123bbdd5002529ac2405dd5079ab57ee207fba: Status 404 returned error can't find the container with id 219063f3078eedd7fc460b6ea8123bbdd5002529ac2405dd5079ab57ee207fba Jul 17 06:40:04.930341 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:04.930305 2580 generic.go:356] "Generic (PLEG): container finished" podID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" containerID="401f80986ba16c7d63a2322c2a98fd2e172fea00562cf1e3b107dd6c3fcfb531" exitCode=6 Jul 17 06:40:04.930781 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:04.930402 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" event={"ID":"b9b52f47-5eb2-4a52-82fd-a8f5a84573f9","Type":"ContainerDied","Data":"401f80986ba16c7d63a2322c2a98fd2e172fea00562cf1e3b107dd6c3fcfb531"} Jul 17 06:40:04.930781 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:04.930458 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" event={"ID":"b9b52f47-5eb2-4a52-82fd-a8f5a84573f9","Type":"ContainerStarted","Data":"219063f3078eedd7fc460b6ea8123bbdd5002529ac2405dd5079ab57ee207fba"} Jul 17 06:40:04.930781 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:04.930535 2580 scope.go:117] "RemoveContainer" containerID="401f80986ba16c7d63a2322c2a98fd2e172fea00562cf1e3b107dd6c3fcfb531" Jul 17 06:40:05.936784 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:05.936694 2580 generic.go:356] "Generic (PLEG): container finished" podID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" containerID="a05e3a636ab94ae23d4d3c6c3d799408c4a9108f99efcfae10165d6354f13225" exitCode=6 Jul 17 06:40:05.937182 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:05.936778 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" event={"ID":"b9b52f47-5eb2-4a52-82fd-a8f5a84573f9","Type":"ContainerDied","Data":"a05e3a636ab94ae23d4d3c6c3d799408c4a9108f99efcfae10165d6354f13225"} Jul 17 06:40:05.937182 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:05.936841 2580 scope.go:117] "RemoveContainer" containerID="401f80986ba16c7d63a2322c2a98fd2e172fea00562cf1e3b107dd6c3fcfb531" Jul 17 06:40:05.937182 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:05.937069 2580 scope.go:117] "RemoveContainer" containerID="a05e3a636ab94ae23d4d3c6c3d799408c4a9108f99efcfae10165d6354f13225" Jul 17 06:40:05.937315 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:40:05.937299 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-f8bk6_odh-ai-gateway-infra(b9b52f47-5eb2-4a52-82fd-a8f5a84573f9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" podUID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" Jul 17 06:40:06.746679 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.746640 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4"] Jul 17 06:40:06.746989 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.746977 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b44040a-8943-4bb3-8700-39abd26a4250" containerName="revoke-keys" Jul 17 06:40:06.747039 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.746991 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b44040a-8943-4bb3-8700-39abd26a4250" containerName="revoke-keys" Jul 17 06:40:06.747039 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.747019 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" containerName="revoke-keys" Jul 17 06:40:06.747039 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.747024 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" containerName="revoke-keys" Jul 17 06:40:06.747039 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.747032 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" containerName="revoke-keys" Jul 17 06:40:06.747039 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.747038 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e82a2bc-a447-4e5e-8dbc-7f88512328d3" containerName="revoke-keys" Jul 17 06:40:06.747193 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.747088 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18b18e65-acf8-46e4-b630-c1ea06ce8df5" containerName="revoke-keys" Jul 17 06:40:06.747193 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.747100 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b44040a-8943-4bb3-8700-39abd26a4250" containerName="revoke-keys" Jul 17 06:40:06.747193 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.747107 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da561351-46ad-4764-8261-15c12e02a1fb" containerName="revoke-keys" Jul 17 06:40:06.751351 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.751333 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" Jul 17 06:40:06.757654 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.757627 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4"] Jul 17 06:40:06.806363 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.806322 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a029d9b0-2113-4f3a-af11-7839a7a2b30a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4\" (UID: \"a029d9b0-2113-4f3a-af11-7839a7a2b30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" Jul 17 06:40:06.907563 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.907519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a029d9b0-2113-4f3a-af11-7839a7a2b30a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4\" (UID: \"a029d9b0-2113-4f3a-af11-7839a7a2b30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" Jul 17 06:40:06.908158 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.908136 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a029d9b0-2113-4f3a-af11-7839a7a2b30a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4\" (UID: \"a029d9b0-2113-4f3a-af11-7839a7a2b30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" Jul 17 06:40:06.943838 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:06.943806 2580 scope.go:117] "RemoveContainer" containerID="a05e3a636ab94ae23d4d3c6c3d799408c4a9108f99efcfae10165d6354f13225" Jul 17 06:40:06.944222 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:40:06.944012 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-f8bk6_odh-ai-gateway-infra(b9b52f47-5eb2-4a52-82fd-a8f5a84573f9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" podUID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" Jul 17 06:40:07.062160 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:07.062062 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" Jul 17 06:40:07.199879 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:07.199850 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4"] Jul 17 06:40:07.202277 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:40:07.202247 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda029d9b0_2113_4f3a_af11_7839a7a2b30a.slice/crio-f35d1c42483483c1768a10171bb20b579fe3375f37c325a5b8b229ee8cf9065c WatchSource:0}: Error finding container f35d1c42483483c1768a10171bb20b579fe3375f37c325a5b8b229ee8cf9065c: Status 404 returned error can't find the container with id f35d1c42483483c1768a10171bb20b579fe3375f37c325a5b8b229ee8cf9065c Jul 17 06:40:07.951837 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:07.951808 2580 generic.go:356] "Generic (PLEG): container finished" podID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" containerID="a253006bb28ba813670a7ff1e86ccf4aca52487bb0a3ea66414c0f62ee03ab8d" exitCode=6 Jul 17 06:40:07.952182 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:07.951883 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" event={"ID":"a029d9b0-2113-4f3a-af11-7839a7a2b30a","Type":"ContainerDied","Data":"a253006bb28ba813670a7ff1e86ccf4aca52487bb0a3ea66414c0f62ee03ab8d"} Jul 17 06:40:07.952182 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:07.951941 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" event={"ID":"a029d9b0-2113-4f3a-af11-7839a7a2b30a","Type":"ContainerStarted","Data":"f35d1c42483483c1768a10171bb20b579fe3375f37c325a5b8b229ee8cf9065c"} Jul 17 06:40:07.952182 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:07.952095 2580 scope.go:117] "RemoveContainer" containerID="a253006bb28ba813670a7ff1e86ccf4aca52487bb0a3ea66414c0f62ee03ab8d" Jul 17 06:40:08.958284 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:08.958241 2580 generic.go:356] "Generic (PLEG): container finished" podID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" containerID="8fc7586c9ed267f5e5f8d96d423b25b48965e88b64b72b6c2f0bf5f00974f8fd" exitCode=6 Jul 17 06:40:08.958749 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:08.958333 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" event={"ID":"a029d9b0-2113-4f3a-af11-7839a7a2b30a","Type":"ContainerDied","Data":"8fc7586c9ed267f5e5f8d96d423b25b48965e88b64b72b6c2f0bf5f00974f8fd"} Jul 17 06:40:08.958749 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:08.958439 2580 scope.go:117] "RemoveContainer" containerID="a253006bb28ba813670a7ff1e86ccf4aca52487bb0a3ea66414c0f62ee03ab8d" Jul 17 06:40:08.958749 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:08.958683 2580 scope.go:117] "RemoveContainer" containerID="8fc7586c9ed267f5e5f8d96d423b25b48965e88b64b72b6c2f0bf5f00974f8fd" Jul 17 06:40:08.958893 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:40:08.958871 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4_odh-ai-gateway-infra(a029d9b0-2113-4f3a-af11-7839a7a2b30a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" podUID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" Jul 17 06:40:09.905947 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:09.905909 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:40:09.964885 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:09.964851 2580 scope.go:117] "RemoveContainer" containerID="8fc7586c9ed267f5e5f8d96d423b25b48965e88b64b72b6c2f0bf5f00974f8fd" Jul 17 06:40:09.965372 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:40:09.965121 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4_odh-ai-gateway-infra(a029d9b0-2113-4f3a-af11-7839a7a2b30a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" podUID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" Jul 17 06:40:14.621980 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:14.621944 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f"] Jul 17 06:40:14.626613 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:14.626588 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" Jul 17 06:40:14.632584 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:14.632556 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f"] Jul 17 06:40:14.684987 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:14.684950 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:40:14.780572 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:14.780514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496b3c65-1c78-4e6c-9ded-1ddcf87f7030-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f\" (UID: \"496b3c65-1c78-4e6c-9ded-1ddcf87f7030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" Jul 17 06:40:14.881946 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:14.881842 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496b3c65-1c78-4e6c-9ded-1ddcf87f7030-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f\" (UID: \"496b3c65-1c78-4e6c-9ded-1ddcf87f7030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" Jul 17 06:40:14.882562 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:14.882542 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496b3c65-1c78-4e6c-9ded-1ddcf87f7030-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f\" (UID: \"496b3c65-1c78-4e6c-9ded-1ddcf87f7030\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" Jul 17 06:40:14.938026 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:14.937986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" Jul 17 06:40:15.073289 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:15.073257 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f"] Jul 17 06:40:15.075989 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:40:15.075959 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496b3c65_1c78_4e6c_9ded_1ddcf87f7030.slice/crio-1024554c32f4d3835a742001e5854c3cf83d317b5fd9830f7612c24430d1dcb6 WatchSource:0}: Error finding container 1024554c32f4d3835a742001e5854c3cf83d317b5fd9830f7612c24430d1dcb6: Status 404 returned error can't find the container with id 1024554c32f4d3835a742001e5854c3cf83d317b5fd9830f7612c24430d1dcb6 Jul 17 06:40:16.000083 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:15.999975 2580 generic.go:356] "Generic (PLEG): container finished" podID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" containerID="3c7a9582e6f90670bf6cb10fcbb9c4ce6d2745d72c88cf2c131eb47e212ce452" exitCode=6 Jul 17 06:40:16.000083 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:16.000041 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" event={"ID":"496b3c65-1c78-4e6c-9ded-1ddcf87f7030","Type":"ContainerDied","Data":"3c7a9582e6f90670bf6cb10fcbb9c4ce6d2745d72c88cf2c131eb47e212ce452"} Jul 17 06:40:16.000541 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:16.000094 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" event={"ID":"496b3c65-1c78-4e6c-9ded-1ddcf87f7030","Type":"ContainerStarted","Data":"1024554c32f4d3835a742001e5854c3cf83d317b5fd9830f7612c24430d1dcb6"} Jul 17 06:40:16.000541 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:16.000337 2580 scope.go:117] "RemoveContainer" containerID="3c7a9582e6f90670bf6cb10fcbb9c4ce6d2745d72c88cf2c131eb47e212ce452" Jul 17 06:40:17.008266 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:17.008173 2580 generic.go:356] "Generic (PLEG): container finished" podID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" containerID="118642f915f15c2f54787b427a394d6c34673b73f23758cddfd54dfbd35834c5" exitCode=6 Jul 17 06:40:17.008266 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:17.008235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" event={"ID":"496b3c65-1c78-4e6c-9ded-1ddcf87f7030","Type":"ContainerDied","Data":"118642f915f15c2f54787b427a394d6c34673b73f23758cddfd54dfbd35834c5"} Jul 17 06:40:17.008756 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:17.008283 2580 scope.go:117] "RemoveContainer" containerID="3c7a9582e6f90670bf6cb10fcbb9c4ce6d2745d72c88cf2c131eb47e212ce452" Jul 17 06:40:17.008756 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:17.008462 2580 scope.go:117] "RemoveContainer" containerID="118642f915f15c2f54787b427a394d6c34673b73f23758cddfd54dfbd35834c5" Jul 17 06:40:17.008756 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:40:17.008700 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f_odh-ai-gateway-infra(496b3c65-1c78-4e6c-9ded-1ddcf87f7030)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" podUID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" Jul 17 06:40:18.015200 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:18.015168 2580 scope.go:117] "RemoveContainer" containerID="118642f915f15c2f54787b427a394d6c34673b73f23758cddfd54dfbd35834c5" Jul 17 06:40:18.015646 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:40:18.015404 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f_odh-ai-gateway-infra(496b3c65-1c78-4e6c-9ded-1ddcf87f7030)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" podUID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" Jul 17 06:40:18.884063 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:18.884030 2580 scope.go:117] "RemoveContainer" containerID="a05e3a636ab94ae23d4d3c6c3d799408c4a9108f99efcfae10165d6354f13225" Jul 17 06:40:20.025218 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:20.025182 2580 generic.go:356] "Generic (PLEG): container finished" podID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" containerID="816898d78db58d94fc64533684bb53954c8c59c726569d6c290c42d77f53b428" exitCode=6 Jul 17 06:40:20.025684 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:20.025259 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" event={"ID":"b9b52f47-5eb2-4a52-82fd-a8f5a84573f9","Type":"ContainerDied","Data":"816898d78db58d94fc64533684bb53954c8c59c726569d6c290c42d77f53b428"} Jul 17 06:40:20.025684 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:20.025324 2580 scope.go:117] "RemoveContainer" containerID="a05e3a636ab94ae23d4d3c6c3d799408c4a9108f99efcfae10165d6354f13225" Jul 17 06:40:20.025684 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:20.025625 2580 scope.go:117] "RemoveContainer" containerID="816898d78db58d94fc64533684bb53954c8c59c726569d6c290c42d77f53b428" Jul 17 06:40:20.025887 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:40:20.025862 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-f8bk6_odh-ai-gateway-infra(b9b52f47-5eb2-4a52-82fd-a8f5a84573f9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" podUID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" Jul 17 06:40:21.056111 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:21.051986 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6"] Jul 17 06:40:21.195580 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:21.195553 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" Jul 17 06:40:21.339814 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:21.339706 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b52f47-5eb2-4a52-82fd-a8f5a84573f9-maas-api-service-ca\") pod \"b9b52f47-5eb2-4a52-82fd-a8f5a84573f9\" (UID: \"b9b52f47-5eb2-4a52-82fd-a8f5a84573f9\") " Jul 17 06:40:21.340076 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:21.340051 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b52f47-5eb2-4a52-82fd-a8f5a84573f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" (UID: "b9b52f47-5eb2-4a52-82fd-a8f5a84573f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:40:21.441408 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:21.441347 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b52f47-5eb2-4a52-82fd-a8f5a84573f9-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:40:22.037786 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:22.037756 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" Jul 17 06:40:22.037977 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:22.037751 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6" event={"ID":"b9b52f47-5eb2-4a52-82fd-a8f5a84573f9","Type":"ContainerDied","Data":"219063f3078eedd7fc460b6ea8123bbdd5002529ac2405dd5079ab57ee207fba"} Jul 17 06:40:22.037977 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:22.037900 2580 scope.go:117] "RemoveContainer" containerID="816898d78db58d94fc64533684bb53954c8c59c726569d6c290c42d77f53b428" Jul 17 06:40:22.061598 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:22.061566 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6"] Jul 17 06:40:22.067056 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:22.067028 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-f8bk6"] Jul 17 06:40:22.889073 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:22.889040 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" path="/var/lib/kubelet/pods/b9b52f47-5eb2-4a52-82fd-a8f5a84573f9/volumes" Jul 17 06:40:24.884282 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:24.884252 2580 scope.go:117] "RemoveContainer" containerID="8fc7586c9ed267f5e5f8d96d423b25b48965e88b64b72b6c2f0bf5f00974f8fd" Jul 17 06:40:26.059969 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:26.059924 2580 generic.go:356] "Generic (PLEG): container finished" podID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" containerID="c599cd1648c046a5844ff59668854e5830aa1d0e9d5294d786c3c6bc6f189c84" exitCode=6 Jul 17 06:40:26.060457 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:26.059986 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" event={"ID":"a029d9b0-2113-4f3a-af11-7839a7a2b30a","Type":"ContainerDied","Data":"c599cd1648c046a5844ff59668854e5830aa1d0e9d5294d786c3c6bc6f189c84"} Jul 17 06:40:26.060457 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:26.060053 2580 scope.go:117] "RemoveContainer" containerID="8fc7586c9ed267f5e5f8d96d423b25b48965e88b64b72b6c2f0bf5f00974f8fd" Jul 17 06:40:26.060457 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:26.060360 2580 scope.go:117] "RemoveContainer" containerID="c599cd1648c046a5844ff59668854e5830aa1d0e9d5294d786c3c6bc6f189c84" Jul 17 06:40:26.060702 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:40:26.060676 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4_odh-ai-gateway-infra(a029d9b0-2113-4f3a-af11-7839a7a2b30a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" podUID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" Jul 17 06:40:27.084922 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:27.084883 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4"] Jul 17 06:40:27.220494 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:27.220466 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" Jul 17 06:40:27.399152 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:27.399120 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a029d9b0-2113-4f3a-af11-7839a7a2b30a-maas-api-service-ca\") pod \"a029d9b0-2113-4f3a-af11-7839a7a2b30a\" (UID: \"a029d9b0-2113-4f3a-af11-7839a7a2b30a\") " Jul 17 06:40:27.399519 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:27.399497 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a029d9b0-2113-4f3a-af11-7839a7a2b30a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a029d9b0-2113-4f3a-af11-7839a7a2b30a" (UID: "a029d9b0-2113-4f3a-af11-7839a7a2b30a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:40:27.500290 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:27.500248 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a029d9b0-2113-4f3a-af11-7839a7a2b30a-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:40:28.074823 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:28.074780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" event={"ID":"a029d9b0-2113-4f3a-af11-7839a7a2b30a","Type":"ContainerDied","Data":"f35d1c42483483c1768a10171bb20b579fe3375f37c325a5b8b229ee8cf9065c"} Jul 17 06:40:28.074823 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:28.074813 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4" Jul 17 06:40:28.075098 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:28.074859 2580 scope.go:117] "RemoveContainer" containerID="c599cd1648c046a5844ff59668854e5830aa1d0e9d5294d786c3c6bc6f189c84" Jul 17 06:40:28.098581 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:28.098545 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4"] Jul 17 06:40:28.106152 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:28.106114 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-q6tb4"] Jul 17 06:40:28.889958 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:28.889909 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" path="/var/lib/kubelet/pods/a029d9b0-2113-4f3a-af11-7839a7a2b30a/volumes" Jul 17 06:40:32.884215 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:32.884179 2580 scope.go:117] "RemoveContainer" containerID="118642f915f15c2f54787b427a394d6c34673b73f23758cddfd54dfbd35834c5" Jul 17 06:40:34.107033 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:34.106993 2580 generic.go:356] "Generic (PLEG): container finished" podID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" containerID="00b71c67b6434cb53dedb340f0d837c804ac0a8037457751f2df20513ed43bf9" exitCode=6 Jul 17 06:40:34.107519 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:34.107065 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" event={"ID":"496b3c65-1c78-4e6c-9ded-1ddcf87f7030","Type":"ContainerDied","Data":"00b71c67b6434cb53dedb340f0d837c804ac0a8037457751f2df20513ed43bf9"} Jul 17 06:40:34.107519 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:34.107129 2580 scope.go:117] "RemoveContainer" containerID="118642f915f15c2f54787b427a394d6c34673b73f23758cddfd54dfbd35834c5" Jul 17 06:40:34.107519 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:34.107403 2580 scope.go:117] "RemoveContainer" containerID="00b71c67b6434cb53dedb340f0d837c804ac0a8037457751f2df20513ed43bf9" Jul 17 06:40:34.107652 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:40:34.107633 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f_odh-ai-gateway-infra(496b3c65-1c78-4e6c-9ded-1ddcf87f7030)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" podUID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" Jul 17 06:40:35.137202 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:35.137163 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f"] Jul 17 06:40:35.275002 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:35.274972 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" Jul 17 06:40:35.368132 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:35.368099 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496b3c65-1c78-4e6c-9ded-1ddcf87f7030-maas-api-service-ca\") pod \"496b3c65-1c78-4e6c-9ded-1ddcf87f7030\" (UID: \"496b3c65-1c78-4e6c-9ded-1ddcf87f7030\") " Jul 17 06:40:35.368501 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:35.368476 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496b3c65-1c78-4e6c-9ded-1ddcf87f7030-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "496b3c65-1c78-4e6c-9ded-1ddcf87f7030" (UID: "496b3c65-1c78-4e6c-9ded-1ddcf87f7030"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:40:35.469002 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:35.468904 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/496b3c65-1c78-4e6c-9ded-1ddcf87f7030-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:40:36.118995 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:36.118963 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" Jul 17 06:40:36.119205 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:36.118988 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f" event={"ID":"496b3c65-1c78-4e6c-9ded-1ddcf87f7030","Type":"ContainerDied","Data":"1024554c32f4d3835a742001e5854c3cf83d317b5fd9830f7612c24430d1dcb6"} Jul 17 06:40:36.119205 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:36.119069 2580 scope.go:117] "RemoveContainer" containerID="00b71c67b6434cb53dedb340f0d837c804ac0a8037457751f2df20513ed43bf9" Jul 17 06:40:36.140646 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:36.140608 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f"] Jul 17 06:40:36.147261 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:36.147233 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-qvf4f"] Jul 17 06:40:36.889160 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:36.889125 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" path="/var/lib/kubelet/pods/496b3c65-1c78-4e6c-9ded-1ddcf87f7030/volumes" Jul 17 06:40:37.690428 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:37.690377 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:40:41.091791 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:41.091750 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:40:41.191927 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:41.191890 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:40:41.479334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:41.479290 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.538894 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz"] Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539722 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539742 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539756 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539765 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539816 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539826 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539847 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539856 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539865 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539874 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539893 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539902 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539916 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539924 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539939 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.539947 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.540089 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.540101 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.540112 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.540127 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.540136 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.540152 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" containerName="revoke-keys" Jul 17 06:40:53.541998 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.540162 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a029d9b0-2113-4f3a-af11-7839a7a2b30a" containerName="revoke-keys" Jul 17 06:40:53.547329 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.547302 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" Jul 17 06:40:53.549625 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.549593 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz"] Jul 17 06:40:53.549893 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.549873 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:40:53.617950 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.617910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163952e1-6ecf-4860-a346-3075e7b00c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-bzpjz\" (UID: \"163952e1-6ecf-4860-a346-3075e7b00c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" Jul 17 06:40:53.718727 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.718681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163952e1-6ecf-4860-a346-3075e7b00c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-bzpjz\" (UID: \"163952e1-6ecf-4860-a346-3075e7b00c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" Jul 17 06:40:53.719401 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.719361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163952e1-6ecf-4860-a346-3075e7b00c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-bzpjz\" (UID: \"163952e1-6ecf-4860-a346-3075e7b00c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" Jul 17 06:40:53.858526 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.858487 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" Jul 17 06:40:53.998903 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:53.998799 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz"] Jul 17 06:40:54.001666 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:40:54.001621 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163952e1_6ecf_4860_a346_3075e7b00c87.slice/crio-dae5f3090ac22fa1e6221ef0b213d09013eb3239be350551bca1bf66641742fa WatchSource:0}: Error finding container dae5f3090ac22fa1e6221ef0b213d09013eb3239be350551bca1bf66641742fa: Status 404 returned error can't find the container with id dae5f3090ac22fa1e6221ef0b213d09013eb3239be350551bca1bf66641742fa Jul 17 06:40:54.205750 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:54.205655 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" event={"ID":"163952e1-6ecf-4860-a346-3075e7b00c87","Type":"ContainerStarted","Data":"dae5f3090ac22fa1e6221ef0b213d09013eb3239be350551bca1bf66641742fa"} Jul 17 06:40:55.212334 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:55.212294 2580 generic.go:356] "Generic (PLEG): container finished" podID="163952e1-6ecf-4860-a346-3075e7b00c87" containerID="21b3d80fc3dd6c5f583cb57f44ccf6f8a8e52a000088f6ab80594cf92f845421" exitCode=6 Jul 17 06:40:55.212741 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:55.212375 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" event={"ID":"163952e1-6ecf-4860-a346-3075e7b00c87","Type":"ContainerDied","Data":"21b3d80fc3dd6c5f583cb57f44ccf6f8a8e52a000088f6ab80594cf92f845421"} Jul 17 06:40:55.212741 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:55.212546 2580 scope.go:117] "RemoveContainer" containerID="21b3d80fc3dd6c5f583cb57f44ccf6f8a8e52a000088f6ab80594cf92f845421" Jul 17 06:40:56.218928 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:56.218840 2580 generic.go:356] "Generic (PLEG): container finished" podID="163952e1-6ecf-4860-a346-3075e7b00c87" containerID="a717b9223c389bf5cb3241e8f7d4c118d07bb87c73e3f4d02601a25a4d86ab6e" exitCode=6 Jul 17 06:40:56.218928 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:56.218912 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" event={"ID":"163952e1-6ecf-4860-a346-3075e7b00c87","Type":"ContainerDied","Data":"a717b9223c389bf5cb3241e8f7d4c118d07bb87c73e3f4d02601a25a4d86ab6e"} Jul 17 06:40:56.219418 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:56.218953 2580 scope.go:117] "RemoveContainer" containerID="21b3d80fc3dd6c5f583cb57f44ccf6f8a8e52a000088f6ab80594cf92f845421" Jul 17 06:40:56.219418 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:56.219188 2580 scope.go:117] "RemoveContainer" containerID="a717b9223c389bf5cb3241e8f7d4c118d07bb87c73e3f4d02601a25a4d86ab6e" Jul 17 06:40:56.219532 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:40:56.219413 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-bzpjz_odh-ai-gateway-infra(163952e1-6ecf-4860-a346-3075e7b00c87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" podUID="163952e1-6ecf-4860-a346-3075e7b00c87" Jul 17 06:40:57.225542 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:40:57.225509 2580 scope.go:117] "RemoveContainer" containerID="a717b9223c389bf5cb3241e8f7d4c118d07bb87c73e3f4d02601a25a4d86ab6e" Jul 17 06:40:57.225967 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:40:57.225716 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-bzpjz_odh-ai-gateway-infra(163952e1-6ecf-4860-a346-3075e7b00c87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" podUID="163952e1-6ecf-4860-a346-3075e7b00c87" Jul 17 06:41:02.684340 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:02.684294 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:41:06.791453 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:06.791415 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd"] Jul 17 06:41:06.791884 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:06.791832 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" containerName="revoke-keys" Jul 17 06:41:06.791884 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:06.791846 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" containerName="revoke-keys" Jul 17 06:41:06.791957 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:06.791926 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b52f47-5eb2-4a52-82fd-a8f5a84573f9" containerName="revoke-keys" Jul 17 06:41:06.791957 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:06.791935 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="496b3c65-1c78-4e6c-9ded-1ddcf87f7030" containerName="revoke-keys" Jul 17 06:41:06.795245 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:06.795227 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" Jul 17 06:41:06.801201 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:06.801166 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd"] Jul 17 06:41:06.943420 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:06.942279 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/373596fe-7b81-4e20-8e87-e1043ea2dfd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd\" (UID: \"373596fe-7b81-4e20-8e87-e1043ea2dfd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" Jul 17 06:41:07.043831 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:07.043722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/373596fe-7b81-4e20-8e87-e1043ea2dfd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd\" (UID: \"373596fe-7b81-4e20-8e87-e1043ea2dfd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" Jul 17 06:41:07.044495 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:07.044470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/373596fe-7b81-4e20-8e87-e1043ea2dfd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd\" (UID: \"373596fe-7b81-4e20-8e87-e1043ea2dfd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" Jul 17 06:41:07.107400 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:07.107354 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" Jul 17 06:41:07.241691 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:07.241661 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd"] Jul 17 06:41:07.243852 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:41:07.243820 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373596fe_7b81_4e20_8e87_e1043ea2dfd6.slice/crio-b07f93448ed002172350d3a2829327c3ca508a665a72b314316bd57e2f62b671 WatchSource:0}: Error finding container b07f93448ed002172350d3a2829327c3ca508a665a72b314316bd57e2f62b671: Status 404 returned error can't find the container with id b07f93448ed002172350d3a2829327c3ca508a665a72b314316bd57e2f62b671 Jul 17 06:41:07.272223 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:07.272182 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" event={"ID":"373596fe-7b81-4e20-8e87-e1043ea2dfd6","Type":"ContainerStarted","Data":"b07f93448ed002172350d3a2829327c3ca508a665a72b314316bd57e2f62b671"} Jul 17 06:41:08.278156 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:08.278120 2580 generic.go:356] "Generic (PLEG): container finished" podID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" containerID="5e79f0c59442a9f1803347ddb6cdeeb22ba460e1c4ffb094252d114b75619cac" exitCode=6 Jul 17 06:41:08.278601 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:08.278212 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" event={"ID":"373596fe-7b81-4e20-8e87-e1043ea2dfd6","Type":"ContainerDied","Data":"5e79f0c59442a9f1803347ddb6cdeeb22ba460e1c4ffb094252d114b75619cac"} Jul 17 06:41:08.278601 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:08.278433 2580 scope.go:117] "RemoveContainer" containerID="5e79f0c59442a9f1803347ddb6cdeeb22ba460e1c4ffb094252d114b75619cac" Jul 17 06:41:09.285210 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:09.285172 2580 generic.go:356] "Generic (PLEG): container finished" podID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" containerID="0e883358942294b9103e0a6a7427c2516b3e9440c211831ff9378443a758485a" exitCode=6 Jul 17 06:41:09.285711 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:09.285218 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" event={"ID":"373596fe-7b81-4e20-8e87-e1043ea2dfd6","Type":"ContainerDied","Data":"0e883358942294b9103e0a6a7427c2516b3e9440c211831ff9378443a758485a"} Jul 17 06:41:09.285711 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:09.285269 2580 scope.go:117] "RemoveContainer" containerID="5e79f0c59442a9f1803347ddb6cdeeb22ba460e1c4ffb094252d114b75619cac" Jul 17 06:41:09.285711 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:09.285478 2580 scope.go:117] "RemoveContainer" containerID="0e883358942294b9103e0a6a7427c2516b3e9440c211831ff9378443a758485a" Jul 17 06:41:09.285852 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:41:09.285712 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd_odh-ai-gateway-infra(373596fe-7b81-4e20-8e87-e1043ea2dfd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" podUID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" Jul 17 06:41:09.883952 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:09.883908 2580 scope.go:117] "RemoveContainer" containerID="a717b9223c389bf5cb3241e8f7d4c118d07bb87c73e3f4d02601a25a4d86ab6e" Jul 17 06:41:10.292037 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:10.291988 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" event={"ID":"163952e1-6ecf-4860-a346-3075e7b00c87","Type":"ContainerStarted","Data":"1891720e1f31a0db3072a37b422f27ede6162ff179744e491fcdc3185270662c"} Jul 17 06:41:10.293859 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:10.293840 2580 scope.go:117] "RemoveContainer" containerID="0e883358942294b9103e0a6a7427c2516b3e9440c211831ff9378443a758485a" Jul 17 06:41:10.294024 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:41:10.294006 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd_odh-ai-gateway-infra(373596fe-7b81-4e20-8e87-e1043ea2dfd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" podUID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" Jul 17 06:41:10.309229 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:10.309171 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" podStartSLOduration=17.309154637 podStartE2EDuration="17.309154637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:41:10.308261859 +0000 UTC m=+3595.052941188" watchObservedRunningTime="2026-07-17 06:41:10.309154637 +0000 UTC m=+3595.053833965" Jul 17 06:41:11.299830 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:11.299798 2580 generic.go:356] "Generic (PLEG): container finished" podID="163952e1-6ecf-4860-a346-3075e7b00c87" containerID="1891720e1f31a0db3072a37b422f27ede6162ff179744e491fcdc3185270662c" exitCode=6 Jul 17 06:41:11.300273 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:11.299869 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" event={"ID":"163952e1-6ecf-4860-a346-3075e7b00c87","Type":"ContainerDied","Data":"1891720e1f31a0db3072a37b422f27ede6162ff179744e491fcdc3185270662c"} Jul 17 06:41:11.300273 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:11.299929 2580 scope.go:117] "RemoveContainer" containerID="a717b9223c389bf5cb3241e8f7d4c118d07bb87c73e3f4d02601a25a4d86ab6e" Jul 17 06:41:11.300273 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:11.300185 2580 scope.go:117] "RemoveContainer" containerID="1891720e1f31a0db3072a37b422f27ede6162ff179744e491fcdc3185270662c" Jul 17 06:41:11.300462 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:41:11.300372 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-bzpjz_odh-ai-gateway-infra(163952e1-6ecf-4860-a346-3075e7b00c87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" podUID="163952e1-6ecf-4860-a346-3075e7b00c87" Jul 17 06:41:11.316901 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:11.316870 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz"] Jul 17 06:41:12.440585 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:12.440554 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" Jul 17 06:41:12.589979 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:12.589877 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163952e1-6ecf-4860-a346-3075e7b00c87-maas-api-service-ca\") pod \"163952e1-6ecf-4860-a346-3075e7b00c87\" (UID: \"163952e1-6ecf-4860-a346-3075e7b00c87\") " Jul 17 06:41:12.590309 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:12.590279 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163952e1-6ecf-4860-a346-3075e7b00c87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "163952e1-6ecf-4860-a346-3075e7b00c87" (UID: "163952e1-6ecf-4860-a346-3075e7b00c87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:41:12.690997 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:12.690941 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/163952e1-6ecf-4860-a346-3075e7b00c87-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:41:13.313580 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:13.313488 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" Jul 17 06:41:13.313580 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:13.313488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz" event={"ID":"163952e1-6ecf-4860-a346-3075e7b00c87","Type":"ContainerDied","Data":"dae5f3090ac22fa1e6221ef0b213d09013eb3239be350551bca1bf66641742fa"} Jul 17 06:41:13.313840 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:13.313625 2580 scope.go:117] "RemoveContainer" containerID="1891720e1f31a0db3072a37b422f27ede6162ff179744e491fcdc3185270662c" Jul 17 06:41:13.331909 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:13.331881 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz"] Jul 17 06:41:13.338449 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:13.338419 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-bzpjz"] Jul 17 06:41:14.673094 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:14.673060 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm"] Jul 17 06:41:14.673509 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:14.673419 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163952e1-6ecf-4860-a346-3075e7b00c87" containerName="revoke-keys" Jul 17 06:41:14.673509 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:14.673433 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="163952e1-6ecf-4860-a346-3075e7b00c87" containerName="revoke-keys" Jul 17 06:41:14.673509 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:14.673458 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163952e1-6ecf-4860-a346-3075e7b00c87" containerName="revoke-keys" Jul 17 06:41:14.673509 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:14.673463 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="163952e1-6ecf-4860-a346-3075e7b00c87" containerName="revoke-keys" Jul 17 06:41:14.673648 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:14.673514 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="163952e1-6ecf-4860-a346-3075e7b00c87" containerName="revoke-keys" Jul 17 06:41:14.673648 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:14.673528 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="163952e1-6ecf-4860-a346-3075e7b00c87" containerName="revoke-keys" Jul 17 06:41:14.676566 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:14.676539 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" Jul 17 06:41:14.683077 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:14.683042 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm"] Jul 17 06:41:14.810033 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:14.809992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d7a935-4201-4101-84fc-ba8bfd1eb8c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-67spm\" (UID: \"c5d7a935-4201-4101-84fc-ba8bfd1eb8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" Jul 17 06:41:14.890100 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:14.890061 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="163952e1-6ecf-4860-a346-3075e7b00c87" path="/var/lib/kubelet/pods/163952e1-6ecf-4860-a346-3075e7b00c87/volumes" Jul 17 06:41:14.910946 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:14.910909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d7a935-4201-4101-84fc-ba8bfd1eb8c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-67spm\" (UID: \"c5d7a935-4201-4101-84fc-ba8bfd1eb8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" Jul 17 06:41:14.911606 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:14.911581 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d7a935-4201-4101-84fc-ba8bfd1eb8c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-67spm\" (UID: \"c5d7a935-4201-4101-84fc-ba8bfd1eb8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" Jul 17 06:41:14.987961 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:14.987871 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" Jul 17 06:41:15.151119 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:15.151012 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm"] Jul 17 06:41:15.153594 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:41:15.153560 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d7a935_4201_4101_84fc_ba8bfd1eb8c9.slice/crio-06ad25349b982d7e67126704fced6e82e0ef2c3786a5b2435e7a385b183b7fc9 WatchSource:0}: Error finding container 06ad25349b982d7e67126704fced6e82e0ef2c3786a5b2435e7a385b183b7fc9: Status 404 returned error can't find the container with id 06ad25349b982d7e67126704fced6e82e0ef2c3786a5b2435e7a385b183b7fc9 Jul 17 06:41:15.325457 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:15.325334 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" event={"ID":"c5d7a935-4201-4101-84fc-ba8bfd1eb8c9","Type":"ContainerStarted","Data":"06ad25349b982d7e67126704fced6e82e0ef2c3786a5b2435e7a385b183b7fc9"} Jul 17 06:41:16.332000 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:16.331964 2580 generic.go:356] "Generic (PLEG): container finished" podID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" containerID="1a7f56dbb14674df86219bf4cb0c04a075f957ea6bbd946c335454ef6057619b" exitCode=6 Jul 17 06:41:16.332424 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:16.332011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" event={"ID":"c5d7a935-4201-4101-84fc-ba8bfd1eb8c9","Type":"ContainerDied","Data":"1a7f56dbb14674df86219bf4cb0c04a075f957ea6bbd946c335454ef6057619b"} Jul 17 06:41:16.332424 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:16.332306 2580 scope.go:117] "RemoveContainer" containerID="1a7f56dbb14674df86219bf4cb0c04a075f957ea6bbd946c335454ef6057619b" Jul 17 06:41:17.339602 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:17.339514 2580 generic.go:356] "Generic (PLEG): container finished" podID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" containerID="2a9bf7877be23ae98fe6ec362f9098749c4c445aa95e2b94a0b3bb0f48c5e320" exitCode=6 Jul 17 06:41:17.339602 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:17.339552 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" event={"ID":"c5d7a935-4201-4101-84fc-ba8bfd1eb8c9","Type":"ContainerDied","Data":"2a9bf7877be23ae98fe6ec362f9098749c4c445aa95e2b94a0b3bb0f48c5e320"} Jul 17 06:41:17.339602 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:17.339598 2580 scope.go:117] "RemoveContainer" containerID="1a7f56dbb14674df86219bf4cb0c04a075f957ea6bbd946c335454ef6057619b" Jul 17 06:41:17.340102 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:17.339826 2580 scope.go:117] "RemoveContainer" containerID="2a9bf7877be23ae98fe6ec362f9098749c4c445aa95e2b94a0b3bb0f48c5e320" Jul 17 06:41:17.340102 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:41:17.340056 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-67spm_odh-ai-gateway-infra(c5d7a935-4201-4101-84fc-ba8bfd1eb8c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" podUID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" Jul 17 06:41:17.521536 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:17.521501 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:41:18.278708 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:18.278673 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:41:18.346049 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:18.346019 2580 scope.go:117] "RemoveContainer" containerID="2a9bf7877be23ae98fe6ec362f9098749c4c445aa95e2b94a0b3bb0f48c5e320" Jul 17 06:41:18.346461 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:41:18.346211 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-67spm_odh-ai-gateway-infra(c5d7a935-4201-4101-84fc-ba8bfd1eb8c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" podUID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" Jul 17 06:41:18.809681 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:18.809643 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:41:20.701900 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:20.701858 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2"] Jul 17 06:41:20.702435 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:20.702362 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163952e1-6ecf-4860-a346-3075e7b00c87" containerName="revoke-keys" Jul 17 06:41:20.702435 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:20.702400 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="163952e1-6ecf-4860-a346-3075e7b00c87" containerName="revoke-keys" Jul 17 06:41:20.702569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:20.702533 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="163952e1-6ecf-4860-a346-3075e7b00c87" containerName="revoke-keys" Jul 17 06:41:20.708175 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:20.708135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" Jul 17 06:41:20.713468 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:20.713438 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2"] Jul 17 06:41:20.868001 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:20.867965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b50f59-e745-4e5f-9464-baa5fa0716a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2\" (UID: \"e9b50f59-e745-4e5f-9464-baa5fa0716a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" Jul 17 06:41:20.968738 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:20.968635 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b50f59-e745-4e5f-9464-baa5fa0716a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2\" (UID: \"e9b50f59-e745-4e5f-9464-baa5fa0716a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" Jul 17 06:41:20.969256 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:20.969235 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b50f59-e745-4e5f-9464-baa5fa0716a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2\" (UID: \"e9b50f59-e745-4e5f-9464-baa5fa0716a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" Jul 17 06:41:21.022656 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:21.022609 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" Jul 17 06:41:21.158757 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:21.158728 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2"] Jul 17 06:41:21.160749 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:41:21.160718 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b50f59_e745_4e5f_9464_baa5fa0716a1.slice/crio-07b7aa5a20278ec0607bc04ba8ace166639b6eeef83575f6dbb256b875521b0c WatchSource:0}: Error finding container 07b7aa5a20278ec0607bc04ba8ace166639b6eeef83575f6dbb256b875521b0c: Status 404 returned error can't find the container with id 07b7aa5a20278ec0607bc04ba8ace166639b6eeef83575f6dbb256b875521b0c Jul 17 06:41:21.361404 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:21.361069 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" event={"ID":"e9b50f59-e745-4e5f-9464-baa5fa0716a1","Type":"ContainerStarted","Data":"07b7aa5a20278ec0607bc04ba8ace166639b6eeef83575f6dbb256b875521b0c"} Jul 17 06:41:22.367990 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:22.367947 2580 generic.go:356] "Generic (PLEG): container finished" podID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" containerID="9a6990d354be99d49d77e4814283e08c9defbb57e382ed5efe7d563f21415aa5" exitCode=6 Jul 17 06:41:22.368456 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:22.368022 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" event={"ID":"e9b50f59-e745-4e5f-9464-baa5fa0716a1","Type":"ContainerDied","Data":"9a6990d354be99d49d77e4814283e08c9defbb57e382ed5efe7d563f21415aa5"} Jul 17 06:41:22.368456 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:22.368308 2580 scope.go:117] "RemoveContainer" containerID="9a6990d354be99d49d77e4814283e08c9defbb57e382ed5efe7d563f21415aa5" Jul 17 06:41:23.374489 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:23.374452 2580 generic.go:356] "Generic (PLEG): container finished" podID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" containerID="70075de9d118c000a60ba09e3e2d09c79cc5458fde5406880291d01d84da8dc5" exitCode=6 Jul 17 06:41:23.374966 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:23.374543 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" event={"ID":"e9b50f59-e745-4e5f-9464-baa5fa0716a1","Type":"ContainerDied","Data":"70075de9d118c000a60ba09e3e2d09c79cc5458fde5406880291d01d84da8dc5"} Jul 17 06:41:23.374966 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:23.374603 2580 scope.go:117] "RemoveContainer" containerID="9a6990d354be99d49d77e4814283e08c9defbb57e382ed5efe7d563f21415aa5" Jul 17 06:41:23.374966 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:23.374759 2580 scope.go:117] "RemoveContainer" containerID="70075de9d118c000a60ba09e3e2d09c79cc5458fde5406880291d01d84da8dc5" Jul 17 06:41:23.374966 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:41:23.374962 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2_odh-ai-gateway-infra(e9b50f59-e745-4e5f-9464-baa5fa0716a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" podUID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" Jul 17 06:41:24.381088 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:24.381052 2580 scope.go:117] "RemoveContainer" containerID="70075de9d118c000a60ba09e3e2d09c79cc5458fde5406880291d01d84da8dc5" Jul 17 06:41:24.381605 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:41:24.381295 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2_odh-ai-gateway-infra(e9b50f59-e745-4e5f-9464-baa5fa0716a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" podUID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" Jul 17 06:41:24.884755 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:24.884728 2580 scope.go:117] "RemoveContainer" containerID="0e883358942294b9103e0a6a7427c2516b3e9440c211831ff9378443a758485a" Jul 17 06:41:25.387410 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:25.387362 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" event={"ID":"373596fe-7b81-4e20-8e87-e1043ea2dfd6","Type":"ContainerStarted","Data":"5ab877614b7f4a7cc388d41c62808834ea673afe9da9d041f79f1e6f8bb60630"} Jul 17 06:41:25.404216 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:25.404091 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" podStartSLOduration=19.404071816 podStartE2EDuration="19.404071816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:41:25.4032899 +0000 UTC m=+3610.147969229" watchObservedRunningTime="2026-07-17 06:41:25.404071816 +0000 UTC m=+3610.148751145" Jul 17 06:41:25.495533 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:25.495493 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:41:25.690217 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:25.690126 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:41:25.797132 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:25.797083 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:41:26.395377 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:26.395336 2580 generic.go:356] "Generic (PLEG): container finished" podID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" containerID="5ab877614b7f4a7cc388d41c62808834ea673afe9da9d041f79f1e6f8bb60630" exitCode=6 Jul 17 06:41:26.395898 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:26.395422 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" event={"ID":"373596fe-7b81-4e20-8e87-e1043ea2dfd6","Type":"ContainerDied","Data":"5ab877614b7f4a7cc388d41c62808834ea673afe9da9d041f79f1e6f8bb60630"} Jul 17 06:41:26.395898 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:26.395543 2580 scope.go:117] "RemoveContainer" containerID="0e883358942294b9103e0a6a7427c2516b3e9440c211831ff9378443a758485a" Jul 17 06:41:26.395898 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:26.395890 2580 scope.go:117] "RemoveContainer" containerID="5ab877614b7f4a7cc388d41c62808834ea673afe9da9d041f79f1e6f8bb60630" Jul 17 06:41:26.396188 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:41:26.396160 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd_odh-ai-gateway-infra(373596fe-7b81-4e20-8e87-e1043ea2dfd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" podUID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" Jul 17 06:41:26.429379 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:26.429338 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd"] Jul 17 06:41:26.472251 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:26.472217 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:41:27.540184 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:27.540156 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" Jul 17 06:41:27.632872 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:27.632831 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/373596fe-7b81-4e20-8e87-e1043ea2dfd6-maas-api-service-ca\") pod \"373596fe-7b81-4e20-8e87-e1043ea2dfd6\" (UID: \"373596fe-7b81-4e20-8e87-e1043ea2dfd6\") " Jul 17 06:41:27.633268 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:27.633243 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373596fe-7b81-4e20-8e87-e1043ea2dfd6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "373596fe-7b81-4e20-8e87-e1043ea2dfd6" (UID: "373596fe-7b81-4e20-8e87-e1043ea2dfd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:41:27.734282 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:27.734181 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/373596fe-7b81-4e20-8e87-e1043ea2dfd6-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:41:28.407996 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:28.407966 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" Jul 17 06:41:28.408203 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:28.407973 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd" event={"ID":"373596fe-7b81-4e20-8e87-e1043ea2dfd6","Type":"ContainerDied","Data":"b07f93448ed002172350d3a2829327c3ca508a665a72b314316bd57e2f62b671"} Jul 17 06:41:28.408203 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:28.408103 2580 scope.go:117] "RemoveContainer" containerID="5ab877614b7f4a7cc388d41c62808834ea673afe9da9d041f79f1e6f8bb60630" Jul 17 06:41:28.448979 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:28.448942 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd"] Jul 17 06:41:28.461850 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:28.461817 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-bhqdd"] Jul 17 06:41:28.889786 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:28.889745 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" path="/var/lib/kubelet/pods/373596fe-7b81-4e20-8e87-e1043ea2dfd6/volumes" Jul 17 06:41:33.884078 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:33.884046 2580 scope.go:117] "RemoveContainer" containerID="2a9bf7877be23ae98fe6ec362f9098749c4c445aa95e2b94a0b3bb0f48c5e320" Jul 17 06:41:34.439576 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:34.439479 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" event={"ID":"c5d7a935-4201-4101-84fc-ba8bfd1eb8c9","Type":"ContainerStarted","Data":"fbca63044d6f852711d664c3bef8bcb46091d7c39b00f98648db324224698b23"} Jul 17 06:41:34.456322 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:34.456264 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" podStartSLOduration=20.456247516 podStartE2EDuration="20.456247516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:41:34.454887632 +0000 UTC m=+3619.199566961" watchObservedRunningTime="2026-07-17 06:41:34.456247516 +0000 UTC m=+3619.200926844" Jul 17 06:41:35.445547 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:35.445506 2580 generic.go:356] "Generic (PLEG): container finished" podID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" containerID="fbca63044d6f852711d664c3bef8bcb46091d7c39b00f98648db324224698b23" exitCode=6 Jul 17 06:41:35.445987 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:35.445581 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" event={"ID":"c5d7a935-4201-4101-84fc-ba8bfd1eb8c9","Type":"ContainerDied","Data":"fbca63044d6f852711d664c3bef8bcb46091d7c39b00f98648db324224698b23"} Jul 17 06:41:35.445987 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:35.445645 2580 scope.go:117] "RemoveContainer" containerID="2a9bf7877be23ae98fe6ec362f9098749c4c445aa95e2b94a0b3bb0f48c5e320" Jul 17 06:41:35.445987 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:35.445974 2580 scope.go:117] "RemoveContainer" containerID="fbca63044d6f852711d664c3bef8bcb46091d7c39b00f98648db324224698b23" Jul 17 06:41:35.446210 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:41:35.446190 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-67spm_odh-ai-gateway-infra(c5d7a935-4201-4101-84fc-ba8bfd1eb8c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" podUID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" Jul 17 06:41:35.470993 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:35.470960 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm"] Jul 17 06:41:36.589965 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:36.589930 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" Jul 17 06:41:36.611636 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:36.611604 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d7a935-4201-4101-84fc-ba8bfd1eb8c9-maas-api-service-ca\") pod \"c5d7a935-4201-4101-84fc-ba8bfd1eb8c9\" (UID: \"c5d7a935-4201-4101-84fc-ba8bfd1eb8c9\") " Jul 17 06:41:36.611968 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:36.611945 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d7a935-4201-4101-84fc-ba8bfd1eb8c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" (UID: "c5d7a935-4201-4101-84fc-ba8bfd1eb8c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:41:36.712254 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:36.712161 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d7a935-4201-4101-84fc-ba8bfd1eb8c9-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:41:37.457796 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:37.457763 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" Jul 17 06:41:37.458021 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:37.457811 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm" event={"ID":"c5d7a935-4201-4101-84fc-ba8bfd1eb8c9","Type":"ContainerDied","Data":"06ad25349b982d7e67126704fced6e82e0ef2c3786a5b2435e7a385b183b7fc9"} Jul 17 06:41:37.458021 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:37.457890 2580 scope.go:117] "RemoveContainer" containerID="fbca63044d6f852711d664c3bef8bcb46091d7c39b00f98648db324224698b23" Jul 17 06:41:37.475833 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:37.475793 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm"] Jul 17 06:41:37.481638 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:37.481610 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-67spm"] Jul 17 06:41:37.788503 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:37.788414 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:41:38.889573 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:38.889526 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" path="/var/lib/kubelet/pods/c5d7a935-4201-4101-84fc-ba8bfd1eb8c9/volumes" Jul 17 06:41:39.884398 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:39.884357 2580 scope.go:117] "RemoveContainer" containerID="70075de9d118c000a60ba09e3e2d09c79cc5458fde5406880291d01d84da8dc5" Jul 17 06:41:40.477352 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:40.477264 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" event={"ID":"e9b50f59-e745-4e5f-9464-baa5fa0716a1","Type":"ContainerStarted","Data":"aacf2a981887b4e54e426ec20fe2a7da107fe6761ead56b39c77f0c9f67fca10"} Jul 17 06:41:40.496697 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:40.496641 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" podStartSLOduration=20.496624087 podStartE2EDuration="20.496624087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:41:40.495533839 +0000 UTC m=+3625.240213167" watchObservedRunningTime="2026-07-17 06:41:40.496624087 +0000 UTC m=+3625.241303445" Jul 17 06:41:41.484931 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:41.484887 2580 generic.go:356] "Generic (PLEG): container finished" podID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" containerID="aacf2a981887b4e54e426ec20fe2a7da107fe6761ead56b39c77f0c9f67fca10" exitCode=6 Jul 17 06:41:41.485368 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:41.484955 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" event={"ID":"e9b50f59-e745-4e5f-9464-baa5fa0716a1","Type":"ContainerDied","Data":"aacf2a981887b4e54e426ec20fe2a7da107fe6761ead56b39c77f0c9f67fca10"} Jul 17 06:41:41.485368 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:41.485018 2580 scope.go:117] "RemoveContainer" containerID="70075de9d118c000a60ba09e3e2d09c79cc5458fde5406880291d01d84da8dc5" Jul 17 06:41:41.485368 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:41.485307 2580 scope.go:117] "RemoveContainer" containerID="aacf2a981887b4e54e426ec20fe2a7da107fe6761ead56b39c77f0c9f67fca10" Jul 17 06:41:41.485566 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:41:41.485544 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2_odh-ai-gateway-infra(e9b50f59-e745-4e5f-9464-baa5fa0716a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" podUID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" Jul 17 06:41:41.507234 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:41.507195 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2"] Jul 17 06:41:42.625987 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:42.625958 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" Jul 17 06:41:42.655321 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:42.655282 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b50f59-e745-4e5f-9464-baa5fa0716a1-maas-api-service-ca\") pod \"e9b50f59-e745-4e5f-9464-baa5fa0716a1\" (UID: \"e9b50f59-e745-4e5f-9464-baa5fa0716a1\") " Jul 17 06:41:42.655640 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:42.655616 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b50f59-e745-4e5f-9464-baa5fa0716a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9b50f59-e745-4e5f-9464-baa5fa0716a1" (UID: "e9b50f59-e745-4e5f-9464-baa5fa0716a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:41:42.756747 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:42.756637 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b50f59-e745-4e5f-9464-baa5fa0716a1-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:41:43.496851 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.496811 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" event={"ID":"e9b50f59-e745-4e5f-9464-baa5fa0716a1","Type":"ContainerDied","Data":"07b7aa5a20278ec0607bc04ba8ace166639b6eeef83575f6dbb256b875521b0c"} Jul 17 06:41:43.497054 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.496881 2580 scope.go:117] "RemoveContainer" containerID="aacf2a981887b4e54e426ec20fe2a7da107fe6761ead56b39c77f0c9f67fca10" Jul 17 06:41:43.497054 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.496822 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2" Jul 17 06:41:43.520799 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.520765 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2"] Jul 17 06:41:43.527568 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.527539 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-nl5l2"] Jul 17 06:41:43.578508 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.578469 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv"] Jul 17 06:41:43.578902 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.578885 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" containerName="revoke-keys" Jul 17 06:41:43.578991 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.578905 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" containerName="revoke-keys" Jul 17 06:41:43.578991 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.578918 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" containerName="revoke-keys" Jul 17 06:41:43.578991 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.578926 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" containerName="revoke-keys" Jul 17 06:41:43.578991 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.578941 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" containerName="revoke-keys" Jul 17 06:41:43.578991 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.578950 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" containerName="revoke-keys" Jul 17 06:41:43.578991 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.578960 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" containerName="revoke-keys" Jul 17 06:41:43.578991 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.578969 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" containerName="revoke-keys" Jul 17 06:41:43.578991 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.578981 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" containerName="revoke-keys" Jul 17 06:41:43.578991 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.578989 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" containerName="revoke-keys" Jul 17 06:41:43.579433 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.579005 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" containerName="revoke-keys" Jul 17 06:41:43.579433 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.579014 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" containerName="revoke-keys" Jul 17 06:41:43.579433 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.579026 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" containerName="revoke-keys" Jul 17 06:41:43.579433 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.579034 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" containerName="revoke-keys" Jul 17 06:41:43.579433 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.579127 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" containerName="revoke-keys" Jul 17 06:41:43.579433 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.579138 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" containerName="revoke-keys" Jul 17 06:41:43.579433 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.579147 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" containerName="revoke-keys" Jul 17 06:41:43.579433 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.579159 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" containerName="revoke-keys" Jul 17 06:41:43.579433 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.579169 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" containerName="revoke-keys" Jul 17 06:41:43.579433 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.579178 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" containerName="revoke-keys" Jul 17 06:41:43.579433 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.579186 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" containerName="revoke-keys" Jul 17 06:41:43.579433 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.579195 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" containerName="revoke-keys" Jul 17 06:41:43.582238 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.582218 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" Jul 17 06:41:43.584465 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.584449 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:41:43.589221 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.589157 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv"] Jul 17 06:41:43.664447 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.664408 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/791f470c-a0d2-4822-bb9e-029ca9bd036d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-mwxxv\" (UID: \"791f470c-a0d2-4822-bb9e-029ca9bd036d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" Jul 17 06:41:43.765279 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.765180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/791f470c-a0d2-4822-bb9e-029ca9bd036d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-mwxxv\" (UID: \"791f470c-a0d2-4822-bb9e-029ca9bd036d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" Jul 17 06:41:43.765869 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.765848 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/791f470c-a0d2-4822-bb9e-029ca9bd036d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-mwxxv\" (UID: \"791f470c-a0d2-4822-bb9e-029ca9bd036d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" Jul 17 06:41:43.893556 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:43.893521 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" Jul 17 06:41:44.029802 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:44.029776 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv"] Jul 17 06:41:44.032040 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:41:44.032003 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791f470c_a0d2_4822_bb9e_029ca9bd036d.slice/crio-37acffbad8603f8768681e2f082f1ceb4de039ad486cf61986469da074069866 WatchSource:0}: Error finding container 37acffbad8603f8768681e2f082f1ceb4de039ad486cf61986469da074069866: Status 404 returned error can't find the container with id 37acffbad8603f8768681e2f082f1ceb4de039ad486cf61986469da074069866 Jul 17 06:41:44.505534 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:44.505494 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" event={"ID":"791f470c-a0d2-4822-bb9e-029ca9bd036d","Type":"ContainerStarted","Data":"119520f70441de12d0a79300d9354454d11151135b3eb80784bd78cdf3e50e8e"} Jul 17 06:41:44.505726 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:44.505557 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" event={"ID":"791f470c-a0d2-4822-bb9e-029ca9bd036d","Type":"ContainerStarted","Data":"37acffbad8603f8768681e2f082f1ceb4de039ad486cf61986469da074069866"} Jul 17 06:41:44.522293 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:44.522235 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" podStartSLOduration=1.522218014 podStartE2EDuration="1.522218014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:41:44.520221532 +0000 UTC m=+3629.264900859" watchObservedRunningTime="2026-07-17 06:41:44.522218014 +0000 UTC m=+3629.266897343" Jul 17 06:41:44.890689 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:44.890645 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9b50f59-e745-4e5f-9464-baa5fa0716a1" path="/var/lib/kubelet/pods/e9b50f59-e745-4e5f-9464-baa5fa0716a1/volumes" Jul 17 06:41:45.402310 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.402260 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss"] Jul 17 06:41:45.402772 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.402751 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" containerName="revoke-keys" Jul 17 06:41:45.402881 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.402775 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="373596fe-7b81-4e20-8e87-e1043ea2dfd6" containerName="revoke-keys" Jul 17 06:41:45.402881 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.402788 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" containerName="revoke-keys" Jul 17 06:41:45.402881 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.402796 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" containerName="revoke-keys" Jul 17 06:41:45.403036 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.402879 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5d7a935-4201-4101-84fc-ba8bfd1eb8c9" containerName="revoke-keys" Jul 17 06:41:45.406103 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.406078 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss" Jul 17 06:41:45.413799 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.413760 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss"] Jul 17 06:41:45.480086 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.480049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5136a293-d4b9-4901-852e-dd5fe0a20f11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss\" (UID: \"5136a293-d4b9-4901-852e-dd5fe0a20f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss" Jul 17 06:41:45.512284 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.512245 2580 generic.go:356] "Generic (PLEG): container finished" podID="791f470c-a0d2-4822-bb9e-029ca9bd036d" containerID="119520f70441de12d0a79300d9354454d11151135b3eb80784bd78cdf3e50e8e" exitCode=6 Jul 17 06:41:45.512471 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.512334 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" event={"ID":"791f470c-a0d2-4822-bb9e-029ca9bd036d","Type":"ContainerDied","Data":"119520f70441de12d0a79300d9354454d11151135b3eb80784bd78cdf3e50e8e"} Jul 17 06:41:45.512535 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.512519 2580 scope.go:117] "RemoveContainer" containerID="119520f70441de12d0a79300d9354454d11151135b3eb80784bd78cdf3e50e8e" Jul 17 06:41:45.581422 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.581339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5136a293-d4b9-4901-852e-dd5fe0a20f11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss\" (UID: \"5136a293-d4b9-4901-852e-dd5fe0a20f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss" Jul 17 06:41:45.582050 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.582030 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5136a293-d4b9-4901-852e-dd5fe0a20f11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss\" (UID: \"5136a293-d4b9-4901-852e-dd5fe0a20f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss" Jul 17 06:41:45.718877 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.718839 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss" Jul 17 06:41:45.856868 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:45.856838 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss"] Jul 17 06:41:45.859343 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:41:45.859313 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5136a293_d4b9_4901_852e_dd5fe0a20f11.slice/crio-4eba3fe8169214629e1ee62978fe1b58948b9e027cc009c3bc88991921ef313a WatchSource:0}: Error finding container 4eba3fe8169214629e1ee62978fe1b58948b9e027cc009c3bc88991921ef313a: Status 404 returned error can't find the container with id 4eba3fe8169214629e1ee62978fe1b58948b9e027cc009c3bc88991921ef313a Jul 17 06:41:46.064330 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:46.064301 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:41:46.163878 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:46.163847 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:41:46.520893 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:46.520859 2580 generic.go:356] "Generic (PLEG): container finished" podID="791f470c-a0d2-4822-bb9e-029ca9bd036d" containerID="1891361b53091835213566539e1a9400e31622488488d43e2ddea1884ee7adb5" exitCode=6 Jul 17 06:41:46.521107 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:46.520943 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" event={"ID":"791f470c-a0d2-4822-bb9e-029ca9bd036d","Type":"ContainerDied","Data":"1891361b53091835213566539e1a9400e31622488488d43e2ddea1884ee7adb5"} Jul 17 06:41:46.521107 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:46.520999 2580 scope.go:117] "RemoveContainer" containerID="119520f70441de12d0a79300d9354454d11151135b3eb80784bd78cdf3e50e8e" Jul 17 06:41:46.521233 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:46.521215 2580 scope.go:117] "RemoveContainer" containerID="1891361b53091835213566539e1a9400e31622488488d43e2ddea1884ee7adb5" Jul 17 06:41:46.521502 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:41:46.521478 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-mwxxv_odh-ai-gateway-infra(791f470c-a0d2-4822-bb9e-029ca9bd036d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" podUID="791f470c-a0d2-4822-bb9e-029ca9bd036d" Jul 17 06:41:46.522852 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:46.522828 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss" event={"ID":"5136a293-d4b9-4901-852e-dd5fe0a20f11","Type":"ContainerStarted","Data":"2bbf794fd10eacac0dae588c4a511e75593194fc9d3c12d2e0531ebd7e756a14"} Jul 17 06:41:46.522960 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:46.522873 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss" event={"ID":"5136a293-d4b9-4901-852e-dd5fe0a20f11","Type":"ContainerStarted","Data":"4eba3fe8169214629e1ee62978fe1b58948b9e027cc009c3bc88991921ef313a"} Jul 17 06:41:46.553262 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:46.553208 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss" podStartSLOduration=1.553191118 podStartE2EDuration="1.553191118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:41:46.550958197 +0000 UTC m=+3631.295637540" watchObservedRunningTime="2026-07-17 06:41:46.553191118 +0000 UTC m=+3631.297870506" Jul 17 06:41:47.529797 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:47.529761 2580 scope.go:117] "RemoveContainer" containerID="1891361b53091835213566539e1a9400e31622488488d43e2ddea1884ee7adb5" Jul 17 06:41:47.530292 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:41:47.530041 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-mwxxv_odh-ai-gateway-infra(791f470c-a0d2-4822-bb9e-029ca9bd036d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" podUID="791f470c-a0d2-4822-bb9e-029ca9bd036d" Jul 17 06:41:47.531011 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:47.530985 2580 generic.go:356] "Generic (PLEG): container finished" podID="5136a293-d4b9-4901-852e-dd5fe0a20f11" containerID="2bbf794fd10eacac0dae588c4a511e75593194fc9d3c12d2e0531ebd7e756a14" exitCode=0 Jul 17 06:41:47.531133 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:47.531061 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss" event={"ID":"5136a293-d4b9-4901-852e-dd5fe0a20f11","Type":"ContainerDied","Data":"2bbf794fd10eacac0dae588c4a511e75593194fc9d3c12d2e0531ebd7e756a14"} Jul 17 06:41:48.668729 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:48.668701 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss" Jul 17 06:41:48.711642 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:48.711604 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5136a293-d4b9-4901-852e-dd5fe0a20f11-maas-api-service-ca\") pod \"5136a293-d4b9-4901-852e-dd5fe0a20f11\" (UID: \"5136a293-d4b9-4901-852e-dd5fe0a20f11\") " Jul 17 06:41:48.712030 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:48.712006 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5136a293-d4b9-4901-852e-dd5fe0a20f11-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5136a293-d4b9-4901-852e-dd5fe0a20f11" (UID: "5136a293-d4b9-4901-852e-dd5fe0a20f11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:41:48.813121 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:48.813023 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5136a293-d4b9-4901-852e-dd5fe0a20f11-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:41:49.542679 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:49.542648 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss" Jul 17 06:41:49.542862 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:49.542644 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss" event={"ID":"5136a293-d4b9-4901-852e-dd5fe0a20f11","Type":"ContainerDied","Data":"4eba3fe8169214629e1ee62978fe1b58948b9e027cc009c3bc88991921ef313a"} Jul 17 06:41:49.542862 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:49.542779 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eba3fe8169214629e1ee62978fe1b58948b9e027cc009c3bc88991921ef313a" Jul 17 06:41:56.004017 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.003976 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6"] Jul 17 06:41:56.004498 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.004197 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" podUID="a426f129-0f66-4f5d-9ee5-40fd5fee9581" containerName="maas-api" containerID="cri-o://d6352b7167894e24023cd38df55ec213e455bc19ae41a0b0641bce3c05290920" gracePeriod=30 Jul 17 06:41:56.062737 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.062692 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:41:56.252171 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.252132 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:41:56.381223 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.381177 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a426f129-0f66-4f5d-9ee5-40fd5fee9581-maas-api-tls\") pod \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\" (UID: \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\") " Jul 17 06:41:56.381223 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.381225 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhkl5\" (UniqueName: \"kubernetes.io/projected/a426f129-0f66-4f5d-9ee5-40fd5fee9581-kube-api-access-hhkl5\") pod \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\" (UID: \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\") " Jul 17 06:41:56.381545 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.381291 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a426f129-0f66-4f5d-9ee5-40fd5fee9581-openshift-service-ca\") pod \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\" (UID: \"a426f129-0f66-4f5d-9ee5-40fd5fee9581\") " Jul 17 06:41:56.381740 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.381703 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a426f129-0f66-4f5d-9ee5-40fd5fee9581-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a426f129-0f66-4f5d-9ee5-40fd5fee9581" (UID: "a426f129-0f66-4f5d-9ee5-40fd5fee9581"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:41:56.383520 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.383493 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a426f129-0f66-4f5d-9ee5-40fd5fee9581-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a426f129-0f66-4f5d-9ee5-40fd5fee9581" (UID: "a426f129-0f66-4f5d-9ee5-40fd5fee9581"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:41:56.383600 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.383526 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a426f129-0f66-4f5d-9ee5-40fd5fee9581-kube-api-access-hhkl5" (OuterVolumeSpecName: "kube-api-access-hhkl5") pod "a426f129-0f66-4f5d-9ee5-40fd5fee9581" (UID: "a426f129-0f66-4f5d-9ee5-40fd5fee9581"). InnerVolumeSpecName "kube-api-access-hhkl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:41:56.482162 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.482122 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a426f129-0f66-4f5d-9ee5-40fd5fee9581-maas-api-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:41:56.482162 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.482155 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhkl5\" (UniqueName: \"kubernetes.io/projected/a426f129-0f66-4f5d-9ee5-40fd5fee9581-kube-api-access-hhkl5\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:41:56.482162 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.482166 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a426f129-0f66-4f5d-9ee5-40fd5fee9581-openshift-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:41:56.576118 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.576077 2580 generic.go:356] "Generic (PLEG): container finished" podID="a426f129-0f66-4f5d-9ee5-40fd5fee9581" containerID="d6352b7167894e24023cd38df55ec213e455bc19ae41a0b0641bce3c05290920" exitCode=0 Jul 17 06:41:56.576118 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.576119 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" Jul 17 06:41:56.576414 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.576165 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" event={"ID":"a426f129-0f66-4f5d-9ee5-40fd5fee9581","Type":"ContainerDied","Data":"d6352b7167894e24023cd38df55ec213e455bc19ae41a0b0641bce3c05290920"} Jul 17 06:41:56.576414 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.576222 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6" event={"ID":"a426f129-0f66-4f5d-9ee5-40fd5fee9581","Type":"ContainerDied","Data":"7c6001f3c302978e0b87e47720bc6ff16913c414d259b1d1aafbedb1412461b9"} Jul 17 06:41:56.576414 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.576253 2580 scope.go:117] "RemoveContainer" containerID="d6352b7167894e24023cd38df55ec213e455bc19ae41a0b0641bce3c05290920" Jul 17 06:41:56.585604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.585584 2580 scope.go:117] "RemoveContainer" containerID="d6352b7167894e24023cd38df55ec213e455bc19ae41a0b0641bce3c05290920" Jul 17 06:41:56.585874 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:41:56.585855 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6352b7167894e24023cd38df55ec213e455bc19ae41a0b0641bce3c05290920\": container with ID starting with d6352b7167894e24023cd38df55ec213e455bc19ae41a0b0641bce3c05290920 not found: ID does not exist" containerID="d6352b7167894e24023cd38df55ec213e455bc19ae41a0b0641bce3c05290920" Jul 17 06:41:56.585911 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.585885 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6352b7167894e24023cd38df55ec213e455bc19ae41a0b0641bce3c05290920"} err="failed to get container status \"d6352b7167894e24023cd38df55ec213e455bc19ae41a0b0641bce3c05290920\": rpc error: code = NotFound desc = could not find container \"d6352b7167894e24023cd38df55ec213e455bc19ae41a0b0641bce3c05290920\": container with ID starting with d6352b7167894e24023cd38df55ec213e455bc19ae41a0b0641bce3c05290920 not found: ID does not exist" Jul 17 06:41:56.600398 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.600349 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6"] Jul 17 06:41:56.605614 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.605587 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ff4257-8496b6f986-2svj6"] Jul 17 06:41:56.889139 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:56.889106 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a426f129-0f66-4f5d-9ee5-40fd5fee9581" path="/var/lib/kubelet/pods/a426f129-0f66-4f5d-9ee5-40fd5fee9581/volumes" Jul 17 06:41:58.884449 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:58.884409 2580 scope.go:117] "RemoveContainer" containerID="1891361b53091835213566539e1a9400e31622488488d43e2ddea1884ee7adb5" Jul 17 06:41:59.594220 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:59.594187 2580 generic.go:356] "Generic (PLEG): container finished" podID="791f470c-a0d2-4822-bb9e-029ca9bd036d" containerID="ce4eb8252658a4119480ea1d1729975d49ba907d126a5fb860da8842ec02da25" exitCode=6 Jul 17 06:41:59.594443 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:59.594266 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" event={"ID":"791f470c-a0d2-4822-bb9e-029ca9bd036d","Type":"ContainerDied","Data":"ce4eb8252658a4119480ea1d1729975d49ba907d126a5fb860da8842ec02da25"} Jul 17 06:41:59.594443 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:59.594332 2580 scope.go:117] "RemoveContainer" containerID="1891361b53091835213566539e1a9400e31622488488d43e2ddea1884ee7adb5" Jul 17 06:41:59.594647 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:59.594631 2580 scope.go:117] "RemoveContainer" containerID="ce4eb8252658a4119480ea1d1729975d49ba907d126a5fb860da8842ec02da25" Jul 17 06:41:59.594882 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:41:59.594862 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-mwxxv_odh-ai-gateway-infra(791f470c-a0d2-4822-bb9e-029ca9bd036d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" podUID="791f470c-a0d2-4822-bb9e-029ca9bd036d" Jul 17 06:41:59.598421 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:41:59.598375 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:42:00.513632 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:00.513589 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:00.622477 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:00.622441 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv"] Jul 17 06:42:00.751112 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:00.751086 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" Jul 17 06:42:00.823561 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:00.823476 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/791f470c-a0d2-4822-bb9e-029ca9bd036d-maas-api-service-ca\") pod \"791f470c-a0d2-4822-bb9e-029ca9bd036d\" (UID: \"791f470c-a0d2-4822-bb9e-029ca9bd036d\") " Jul 17 06:42:00.823850 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:00.823827 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791f470c-a0d2-4822-bb9e-029ca9bd036d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "791f470c-a0d2-4822-bb9e-029ca9bd036d" (UID: "791f470c-a0d2-4822-bb9e-029ca9bd036d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:42:00.909408 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:00.909356 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:00.924685 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:00.924652 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/791f470c-a0d2-4822-bb9e-029ca9bd036d-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:42:01.611078 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:01.611029 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" event={"ID":"791f470c-a0d2-4822-bb9e-029ca9bd036d","Type":"ContainerDied","Data":"37acffbad8603f8768681e2f082f1ceb4de039ad486cf61986469da074069866"} Jul 17 06:42:01.611591 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:01.611118 2580 scope.go:117] "RemoveContainer" containerID="ce4eb8252658a4119480ea1d1729975d49ba907d126a5fb860da8842ec02da25" Jul 17 06:42:01.611591 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:01.611050 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv" Jul 17 06:42:01.649182 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:01.649152 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv"] Jul 17 06:42:01.657288 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:01.657256 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mwxxv"] Jul 17 06:42:02.889257 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:02.889221 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="791f470c-a0d2-4822-bb9e-029ca9bd036d" path="/var/lib/kubelet/pods/791f470c-a0d2-4822-bb9e-029ca9bd036d/volumes" Jul 17 06:42:07.176645 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:07.176611 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:07.256585 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:07.256553 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:07.502636 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:07.502553 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:07.689640 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:07.689608 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:08.359352 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:08.359306 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:08.460699 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:08.460662 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:14.747087 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.747049 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz"] Jul 17 06:42:14.747495 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.747409 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791f470c-a0d2-4822-bb9e-029ca9bd036d" containerName="revoke-keys" Jul 17 06:42:14.747495 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.747420 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="791f470c-a0d2-4822-bb9e-029ca9bd036d" containerName="revoke-keys" Jul 17 06:42:14.747495 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.747429 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791f470c-a0d2-4822-bb9e-029ca9bd036d" containerName="revoke-keys" Jul 17 06:42:14.747495 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.747435 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="791f470c-a0d2-4822-bb9e-029ca9bd036d" containerName="revoke-keys" Jul 17 06:42:14.747495 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.747444 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a426f129-0f66-4f5d-9ee5-40fd5fee9581" containerName="maas-api" Jul 17 06:42:14.747495 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.747450 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a426f129-0f66-4f5d-9ee5-40fd5fee9581" containerName="maas-api" Jul 17 06:42:14.747495 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.747461 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5136a293-d4b9-4901-852e-dd5fe0a20f11" containerName="revoke-keys" Jul 17 06:42:14.747495 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.747466 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5136a293-d4b9-4901-852e-dd5fe0a20f11" containerName="revoke-keys" Jul 17 06:42:14.747767 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.747519 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="791f470c-a0d2-4822-bb9e-029ca9bd036d" containerName="revoke-keys" Jul 17 06:42:14.747767 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.747526 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a426f129-0f66-4f5d-9ee5-40fd5fee9581" containerName="maas-api" Jul 17 06:42:14.747767 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.747533 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="791f470c-a0d2-4822-bb9e-029ca9bd036d" containerName="revoke-keys" Jul 17 06:42:14.747767 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.747539 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5136a293-d4b9-4901-852e-dd5fe0a20f11" containerName="revoke-keys" Jul 17 06:42:14.751995 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.751976 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" Jul 17 06:42:14.755294 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.755273 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:42:14.777181 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.777156 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz"] Jul 17 06:42:14.852935 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.852896 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66882bf2-9245-42f3-bac8-aafc51e4e4b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz\" (UID: \"66882bf2-9245-42f3-bac8-aafc51e4e4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" Jul 17 06:42:14.953767 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.953731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66882bf2-9245-42f3-bac8-aafc51e4e4b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz\" (UID: \"66882bf2-9245-42f3-bac8-aafc51e4e4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" Jul 17 06:42:14.954377 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:14.954354 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66882bf2-9245-42f3-bac8-aafc51e4e4b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz\" (UID: \"66882bf2-9245-42f3-bac8-aafc51e4e4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" Jul 17 06:42:15.062362 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:15.062273 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" Jul 17 06:42:15.201599 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:15.201570 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz"] Jul 17 06:42:15.203808 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:42:15.203777 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66882bf2_9245_42f3_bac8_aafc51e4e4b1.slice/crio-3c1076b88a59768f31aef23ece68e9fd9232fdc0fa5e378a46937b4ef10ffb2e WatchSource:0}: Error finding container 3c1076b88a59768f31aef23ece68e9fd9232fdc0fa5e378a46937b4ef10ffb2e: Status 404 returned error can't find the container with id 3c1076b88a59768f31aef23ece68e9fd9232fdc0fa5e378a46937b4ef10ffb2e Jul 17 06:42:15.681540 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:15.681505 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" event={"ID":"66882bf2-9245-42f3-bac8-aafc51e4e4b1","Type":"ContainerStarted","Data":"b472cf9addf332c077d643ea3212fd3e627337cf95ef44928cc404610669a3d3"} Jul 17 06:42:15.681724 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:15.681556 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" event={"ID":"66882bf2-9245-42f3-bac8-aafc51e4e4b1","Type":"ContainerStarted","Data":"3c1076b88a59768f31aef23ece68e9fd9232fdc0fa5e378a46937b4ef10ffb2e"} Jul 17 06:42:15.700612 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:15.700562 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" podStartSLOduration=1.700548996 podStartE2EDuration="1.700548996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:42:15.698976212 +0000 UTC m=+3660.443655552" watchObservedRunningTime="2026-07-17 06:42:15.700548996 +0000 UTC m=+3660.445228323" Jul 17 06:42:16.687907 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:16.687875 2580 generic.go:356] "Generic (PLEG): container finished" podID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" containerID="b472cf9addf332c077d643ea3212fd3e627337cf95ef44928cc404610669a3d3" exitCode=6 Jul 17 06:42:16.688283 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:16.687908 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" event={"ID":"66882bf2-9245-42f3-bac8-aafc51e4e4b1","Type":"ContainerDied","Data":"b472cf9addf332c077d643ea3212fd3e627337cf95ef44928cc404610669a3d3"} Jul 17 06:42:16.688283 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:16.688198 2580 scope.go:117] "RemoveContainer" containerID="b472cf9addf332c077d643ea3212fd3e627337cf95ef44928cc404610669a3d3" Jul 17 06:42:17.694889 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:17.694851 2580 generic.go:356] "Generic (PLEG): container finished" podID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" containerID="b42197c91d99a4202549b1687be81b911229b8e9f4c8b135281d9df0092722f7" exitCode=6 Jul 17 06:42:17.695373 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:17.694936 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" event={"ID":"66882bf2-9245-42f3-bac8-aafc51e4e4b1","Type":"ContainerDied","Data":"b42197c91d99a4202549b1687be81b911229b8e9f4c8b135281d9df0092722f7"} Jul 17 06:42:17.695373 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:17.695002 2580 scope.go:117] "RemoveContainer" containerID="b472cf9addf332c077d643ea3212fd3e627337cf95ef44928cc404610669a3d3" Jul 17 06:42:17.695373 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:17.695121 2580 scope.go:117] "RemoveContainer" containerID="b42197c91d99a4202549b1687be81b911229b8e9f4c8b135281d9df0092722f7" Jul 17 06:42:17.695571 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:42:17.695410 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz_odh-ai-gateway-infra(66882bf2-9245-42f3-bac8-aafc51e4e4b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" podUID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" Jul 17 06:42:18.607758 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:18.607723 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx"] Jul 17 06:42:18.608042 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:18.607990 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" podUID="87691a64-febc-4700-aec9-e965782925e9" containerName="maas-api" containerID="cri-o://fe3bc23836a0eceea3800e660b145525582c1f555069163d3957bb81b4f1d982" gracePeriod=30 Jul 17 06:42:18.650576 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:18.650544 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:18.701682 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:18.701650 2580 scope.go:117] "RemoveContainer" containerID="b42197c91d99a4202549b1687be81b911229b8e9f4c8b135281d9df0092722f7" Jul 17 06:42:18.702079 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:42:18.701842 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz_odh-ai-gateway-infra(66882bf2-9245-42f3-bac8-aafc51e4e4b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" podUID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" Jul 17 06:42:18.857896 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:18.857838 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:42:18.988626 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:18.988595 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87691a64-febc-4700-aec9-e965782925e9-maas-api-tls\") pod \"87691a64-febc-4700-aec9-e965782925e9\" (UID: \"87691a64-febc-4700-aec9-e965782925e9\") " Jul 17 06:42:18.988790 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:18.988654 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87691a64-febc-4700-aec9-e965782925e9-openshift-service-ca\") pod \"87691a64-febc-4700-aec9-e965782925e9\" (UID: \"87691a64-febc-4700-aec9-e965782925e9\") " Jul 17 06:42:18.988790 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:18.988734 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvps7\" (UniqueName: \"kubernetes.io/projected/87691a64-febc-4700-aec9-e965782925e9-kube-api-access-fvps7\") pod \"87691a64-febc-4700-aec9-e965782925e9\" (UID: \"87691a64-febc-4700-aec9-e965782925e9\") " Jul 17 06:42:18.989181 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:18.989073 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87691a64-febc-4700-aec9-e965782925e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87691a64-febc-4700-aec9-e965782925e9" (UID: "87691a64-febc-4700-aec9-e965782925e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:42:18.990731 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:18.990708 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87691a64-febc-4700-aec9-e965782925e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87691a64-febc-4700-aec9-e965782925e9" (UID: "87691a64-febc-4700-aec9-e965782925e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:42:18.990895 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:18.990876 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87691a64-febc-4700-aec9-e965782925e9-kube-api-access-fvps7" (OuterVolumeSpecName: "kube-api-access-fvps7") pod "87691a64-febc-4700-aec9-e965782925e9" (UID: "87691a64-febc-4700-aec9-e965782925e9"). InnerVolumeSpecName "kube-api-access-fvps7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:42:19.090092 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:19.090051 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvps7\" (UniqueName: \"kubernetes.io/projected/87691a64-febc-4700-aec9-e965782925e9-kube-api-access-fvps7\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:42:19.090092 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:19.090082 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87691a64-febc-4700-aec9-e965782925e9-maas-api-tls\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:42:19.090092 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:19.090093 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87691a64-febc-4700-aec9-e965782925e9-openshift-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:42:19.707719 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:19.707666 2580 generic.go:356] "Generic (PLEG): container finished" podID="87691a64-febc-4700-aec9-e965782925e9" containerID="fe3bc23836a0eceea3800e660b145525582c1f555069163d3957bb81b4f1d982" exitCode=0 Jul 17 06:42:19.708145 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:19.707729 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" Jul 17 06:42:19.708145 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:19.707760 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" event={"ID":"87691a64-febc-4700-aec9-e965782925e9","Type":"ContainerDied","Data":"fe3bc23836a0eceea3800e660b145525582c1f555069163d3957bb81b4f1d982"} Jul 17 06:42:19.708145 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:19.707821 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx" event={"ID":"87691a64-febc-4700-aec9-e965782925e9","Type":"ContainerDied","Data":"785feadf2b156108839a2a471efabb2412dea9906afa05040e09ded9bbafe10c"} Jul 17 06:42:19.708145 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:19.707843 2580 scope.go:117] "RemoveContainer" containerID="fe3bc23836a0eceea3800e660b145525582c1f555069163d3957bb81b4f1d982" Jul 17 06:42:19.717028 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:19.717011 2580 scope.go:117] "RemoveContainer" containerID="fe3bc23836a0eceea3800e660b145525582c1f555069163d3957bb81b4f1d982" Jul 17 06:42:19.717288 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:42:19.717269 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3bc23836a0eceea3800e660b145525582c1f555069163d3957bb81b4f1d982\": container with ID starting with fe3bc23836a0eceea3800e660b145525582c1f555069163d3957bb81b4f1d982 not found: ID does not exist" containerID="fe3bc23836a0eceea3800e660b145525582c1f555069163d3957bb81b4f1d982" Jul 17 06:42:19.717332 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:19.717295 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3bc23836a0eceea3800e660b145525582c1f555069163d3957bb81b4f1d982"} err="failed to get container status \"fe3bc23836a0eceea3800e660b145525582c1f555069163d3957bb81b4f1d982\": rpc error: code = NotFound desc = could not find container \"fe3bc23836a0eceea3800e660b145525582c1f555069163d3957bb81b4f1d982\": container with ID starting with fe3bc23836a0eceea3800e660b145525582c1f555069163d3957bb81b4f1d982 not found: ID does not exist" Jul 17 06:42:19.730378 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:19.730350 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx"] Jul 17 06:42:19.733815 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:19.733792 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-346c23-6886f6cbdd-4c4zx"] Jul 17 06:42:20.759913 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:20.759885 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk"] Jul 17 06:42:20.760293 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:20.760205 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87691a64-febc-4700-aec9-e965782925e9" containerName="maas-api" Jul 17 06:42:20.760293 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:20.760216 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="87691a64-febc-4700-aec9-e965782925e9" containerName="maas-api" Jul 17 06:42:20.760293 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:20.760229 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791f470c-a0d2-4822-bb9e-029ca9bd036d" containerName="revoke-keys" Jul 17 06:42:20.760293 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:20.760235 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="791f470c-a0d2-4822-bb9e-029ca9bd036d" containerName="revoke-keys" Jul 17 06:42:20.760293 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:20.760281 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="791f470c-a0d2-4822-bb9e-029ca9bd036d" containerName="revoke-keys" Jul 17 06:42:20.760293 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:20.760295 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="87691a64-febc-4700-aec9-e965782925e9" containerName="maas-api" Jul 17 06:42:20.764949 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:20.764931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" Jul 17 06:42:20.770683 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:20.770658 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk"] Jul 17 06:42:20.889351 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:20.889319 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87691a64-febc-4700-aec9-e965782925e9" path="/var/lib/kubelet/pods/87691a64-febc-4700-aec9-e965782925e9/volumes" Jul 17 06:42:20.907922 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:20.907890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65ba9db-2944-4e8b-979d-bd9ba3b38310-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk\" (UID: \"b65ba9db-2944-4e8b-979d-bd9ba3b38310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" Jul 17 06:42:21.008690 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:21.008649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65ba9db-2944-4e8b-979d-bd9ba3b38310-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk\" (UID: \"b65ba9db-2944-4e8b-979d-bd9ba3b38310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" Jul 17 06:42:21.009237 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:21.009217 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65ba9db-2944-4e8b-979d-bd9ba3b38310-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk\" (UID: \"b65ba9db-2944-4e8b-979d-bd9ba3b38310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" Jul 17 06:42:21.075968 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:21.075887 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" Jul 17 06:42:21.235252 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:21.235222 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk"] Jul 17 06:42:21.237002 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:42:21.236972 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65ba9db_2944_4e8b_979d_bd9ba3b38310.slice/crio-5db6c0f375a9b530ab439336555dc912abaaba4aaf207ed21c7a85ff78c73e58 WatchSource:0}: Error finding container 5db6c0f375a9b530ab439336555dc912abaaba4aaf207ed21c7a85ff78c73e58: Status 404 returned error can't find the container with id 5db6c0f375a9b530ab439336555dc912abaaba4aaf207ed21c7a85ff78c73e58 Jul 17 06:42:21.719641 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:21.719606 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" event={"ID":"b65ba9db-2944-4e8b-979d-bd9ba3b38310","Type":"ContainerStarted","Data":"b492de294a13ecdc3dae13b37c06733a1b91decdc9c72f991a769171ee2bbd90"} Jul 17 06:42:21.719833 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:21.719663 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" event={"ID":"b65ba9db-2944-4e8b-979d-bd9ba3b38310","Type":"ContainerStarted","Data":"5db6c0f375a9b530ab439336555dc912abaaba4aaf207ed21c7a85ff78c73e58"} Jul 17 06:42:21.736946 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:21.736900 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" podStartSLOduration=1.7368841879999999 podStartE2EDuration="1.736884188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:42:21.735315647 +0000 UTC m=+3666.479994975" watchObservedRunningTime="2026-07-17 06:42:21.736884188 +0000 UTC m=+3666.481563515" Jul 17 06:42:21.897099 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:21.897068 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lxk5j"] Jul 17 06:42:22.725286 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:22.725244 2580 generic.go:356] "Generic (PLEG): container finished" podID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" containerID="b492de294a13ecdc3dae13b37c06733a1b91decdc9c72f991a769171ee2bbd90" exitCode=6 Jul 17 06:42:22.725506 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:22.725305 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" event={"ID":"b65ba9db-2944-4e8b-979d-bd9ba3b38310","Type":"ContainerDied","Data":"b492de294a13ecdc3dae13b37c06733a1b91decdc9c72f991a769171ee2bbd90"} Jul 17 06:42:22.725580 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:22.725558 2580 scope.go:117] "RemoveContainer" containerID="b492de294a13ecdc3dae13b37c06733a1b91decdc9c72f991a769171ee2bbd90" Jul 17 06:42:22.910238 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:22.910203 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:23.383593 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:23.383562 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:23.731718 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:23.731629 2580 generic.go:356] "Generic (PLEG): container finished" podID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" containerID="fcfbb03ab9a08a0d3c32a89a2e0784ee526f2b95141de67ca18274ddb2246386" exitCode=6 Jul 17 06:42:23.731718 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:23.731707 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" event={"ID":"b65ba9db-2944-4e8b-979d-bd9ba3b38310","Type":"ContainerDied","Data":"fcfbb03ab9a08a0d3c32a89a2e0784ee526f2b95141de67ca18274ddb2246386"} Jul 17 06:42:23.731928 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:23.731749 2580 scope.go:117] "RemoveContainer" containerID="b492de294a13ecdc3dae13b37c06733a1b91decdc9c72f991a769171ee2bbd90" Jul 17 06:42:23.731973 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:23.731958 2580 scope.go:117] "RemoveContainer" containerID="fcfbb03ab9a08a0d3c32a89a2e0784ee526f2b95141de67ca18274ddb2246386" Jul 17 06:42:23.732156 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:42:23.732136 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk_odh-ai-gateway-infra(b65ba9db-2944-4e8b-979d-bd9ba3b38310)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" podUID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" Jul 17 06:42:24.738303 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:24.738272 2580 scope.go:117] "RemoveContainer" containerID="fcfbb03ab9a08a0d3c32a89a2e0784ee526f2b95141de67ca18274ddb2246386" Jul 17 06:42:24.738725 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:42:24.738495 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk_odh-ai-gateway-infra(b65ba9db-2944-4e8b-979d-bd9ba3b38310)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" podUID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" Jul 17 06:42:28.766551 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:28.766516 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:29.884093 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:29.884062 2580 scope.go:117] "RemoveContainer" containerID="b42197c91d99a4202549b1687be81b911229b8e9f4c8b135281d9df0092722f7" Jul 17 06:42:30.238954 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:30.238868 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:30.359041 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:30.359003 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:30.535249 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:30.535169 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:30.765876 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:30.765846 2580 generic.go:356] "Generic (PLEG): container finished" podID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" containerID="2f327323dd0ddf250b19f667505bb1906e8a21f296a41dbc98122658f1e65c0b" exitCode=6 Jul 17 06:42:30.766039 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:30.765913 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" event={"ID":"66882bf2-9245-42f3-bac8-aafc51e4e4b1","Type":"ContainerDied","Data":"2f327323dd0ddf250b19f667505bb1906e8a21f296a41dbc98122658f1e65c0b"} Jul 17 06:42:30.766039 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:30.765974 2580 scope.go:117] "RemoveContainer" containerID="b42197c91d99a4202549b1687be81b911229b8e9f4c8b135281d9df0092722f7" Jul 17 06:42:30.766323 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:30.766300 2580 scope.go:117] "RemoveContainer" containerID="2f327323dd0ddf250b19f667505bb1906e8a21f296a41dbc98122658f1e65c0b" Jul 17 06:42:30.766611 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:42:30.766586 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz_odh-ai-gateway-infra(66882bf2-9245-42f3-bac8-aafc51e4e4b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" podUID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" Jul 17 06:42:30.777886 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:30.777863 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:31.797607 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:31.797566 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz"] Jul 17 06:42:31.928001 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:31.927979 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" Jul 17 06:42:32.000102 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:32.000065 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66882bf2-9245-42f3-bac8-aafc51e4e4b1-maas-api-service-ca\") pod \"66882bf2-9245-42f3-bac8-aafc51e4e4b1\" (UID: \"66882bf2-9245-42f3-bac8-aafc51e4e4b1\") " Jul 17 06:42:32.000443 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:32.000420 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66882bf2-9245-42f3-bac8-aafc51e4e4b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "66882bf2-9245-42f3-bac8-aafc51e4e4b1" (UID: "66882bf2-9245-42f3-bac8-aafc51e4e4b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:42:32.101106 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:32.101021 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66882bf2-9245-42f3-bac8-aafc51e4e4b1-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:42:32.780125 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:32.780086 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" event={"ID":"66882bf2-9245-42f3-bac8-aafc51e4e4b1","Type":"ContainerDied","Data":"3c1076b88a59768f31aef23ece68e9fd9232fdc0fa5e378a46937b4ef10ffb2e"} Jul 17 06:42:32.780125 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:32.780106 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz" Jul 17 06:42:32.780377 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:32.780146 2580 scope.go:117] "RemoveContainer" containerID="2f327323dd0ddf250b19f667505bb1906e8a21f296a41dbc98122658f1e65c0b" Jul 17 06:42:32.804801 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:32.804774 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz"] Jul 17 06:42:32.809227 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:32.809202 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-jxxzz"] Jul 17 06:42:32.889612 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:32.889577 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" path="/var/lib/kubelet/pods/66882bf2-9245-42f3-bac8-aafc51e4e4b1/volumes" Jul 17 06:42:33.638336 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.638301 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz"] Jul 17 06:42:33.638651 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.638639 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" containerName="revoke-keys" Jul 17 06:42:33.638698 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.638653 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" containerName="revoke-keys" Jul 17 06:42:33.638698 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.638663 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" containerName="revoke-keys" Jul 17 06:42:33.638698 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.638669 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" containerName="revoke-keys" Jul 17 06:42:33.638698 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.638682 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" containerName="revoke-keys" Jul 17 06:42:33.638698 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.638687 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" containerName="revoke-keys" Jul 17 06:42:33.638847 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.638738 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" containerName="revoke-keys" Jul 17 06:42:33.638847 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.638746 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" containerName="revoke-keys" Jul 17 06:42:33.643109 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.643084 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" Jul 17 06:42:33.652253 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.652226 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz"] Jul 17 06:42:33.714894 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.714864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33265d76-0807-48c0-86ee-c6a8e87e7593-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-28jcz\" (UID: \"33265d76-0807-48c0-86ee-c6a8e87e7593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" Jul 17 06:42:33.816038 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.815989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33265d76-0807-48c0-86ee-c6a8e87e7593-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-28jcz\" (UID: \"33265d76-0807-48c0-86ee-c6a8e87e7593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" Jul 17 06:42:33.816667 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.816649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33265d76-0807-48c0-86ee-c6a8e87e7593-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-28jcz\" (UID: \"33265d76-0807-48c0-86ee-c6a8e87e7593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" Jul 17 06:42:33.955058 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:33.954979 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" Jul 17 06:42:34.095400 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:34.095276 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz"] Jul 17 06:42:34.097915 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:42:34.097889 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33265d76_0807_48c0_86ee_c6a8e87e7593.slice/crio-138eb46007746308f1f059287dee76dac96cd2ebe6a6329169c116841137bb62 WatchSource:0}: Error finding container 138eb46007746308f1f059287dee76dac96cd2ebe6a6329169c116841137bb62: Status 404 returned error can't find the container with id 138eb46007746308f1f059287dee76dac96cd2ebe6a6329169c116841137bb62 Jul 17 06:42:34.792198 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:34.792163 2580 generic.go:356] "Generic (PLEG): container finished" podID="33265d76-0807-48c0-86ee-c6a8e87e7593" containerID="8391a59f8b26c159a825d65c8bc513599db4ab68c17a8a404c16f329be62f30c" exitCode=6 Jul 17 06:42:34.792380 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:34.792203 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" event={"ID":"33265d76-0807-48c0-86ee-c6a8e87e7593","Type":"ContainerDied","Data":"8391a59f8b26c159a825d65c8bc513599db4ab68c17a8a404c16f329be62f30c"} Jul 17 06:42:34.792380 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:34.792241 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" event={"ID":"33265d76-0807-48c0-86ee-c6a8e87e7593","Type":"ContainerStarted","Data":"138eb46007746308f1f059287dee76dac96cd2ebe6a6329169c116841137bb62"} Jul 17 06:42:34.792490 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:34.792421 2580 scope.go:117] "RemoveContainer" containerID="8391a59f8b26c159a825d65c8bc513599db4ab68c17a8a404c16f329be62f30c" Jul 17 06:42:35.799250 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:35.799214 2580 generic.go:356] "Generic (PLEG): container finished" podID="33265d76-0807-48c0-86ee-c6a8e87e7593" containerID="73295cef0335770169edf45ac1f25f15914d32a20c32b18f3ffa5ad0cab59504" exitCode=6 Jul 17 06:42:35.799662 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:35.799295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" event={"ID":"33265d76-0807-48c0-86ee-c6a8e87e7593","Type":"ContainerDied","Data":"73295cef0335770169edf45ac1f25f15914d32a20c32b18f3ffa5ad0cab59504"} Jul 17 06:42:35.799662 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:35.799356 2580 scope.go:117] "RemoveContainer" containerID="8391a59f8b26c159a825d65c8bc513599db4ab68c17a8a404c16f329be62f30c" Jul 17 06:42:35.799662 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:35.799573 2580 scope.go:117] "RemoveContainer" containerID="73295cef0335770169edf45ac1f25f15914d32a20c32b18f3ffa5ad0cab59504" Jul 17 06:42:35.799813 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:42:35.799797 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-28jcz_odh-ai-gateway-infra(33265d76-0807-48c0-86ee-c6a8e87e7593)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" podUID="33265d76-0807-48c0-86ee-c6a8e87e7593" Jul 17 06:42:35.884556 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:35.884524 2580 scope.go:117] "RemoveContainer" containerID="fcfbb03ab9a08a0d3c32a89a2e0784ee526f2b95141de67ca18274ddb2246386" Jul 17 06:42:36.450525 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:42:36.450495 2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65ba9db_2944_4e8b_979d_bd9ba3b38310.slice/crio-a682b16eb6a7a4d260c1f902443aac6bef2253a7afb29fd8560dc627637798cc.scope\": RecentStats: unable to find data in memory cache]" Jul 17 06:42:36.805803 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:36.805708 2580 generic.go:356] "Generic (PLEG): container finished" podID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" containerID="a682b16eb6a7a4d260c1f902443aac6bef2253a7afb29fd8560dc627637798cc" exitCode=6 Jul 17 06:42:36.805803 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:36.805787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" event={"ID":"b65ba9db-2944-4e8b-979d-bd9ba3b38310","Type":"ContainerDied","Data":"a682b16eb6a7a4d260c1f902443aac6bef2253a7afb29fd8560dc627637798cc"} Jul 17 06:42:36.806304 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:36.805896 2580 scope.go:117] "RemoveContainer" containerID="fcfbb03ab9a08a0d3c32a89a2e0784ee526f2b95141de67ca18274ddb2246386" Jul 17 06:42:36.806304 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:36.806202 2580 scope.go:117] "RemoveContainer" containerID="a682b16eb6a7a4d260c1f902443aac6bef2253a7afb29fd8560dc627637798cc" Jul 17 06:42:36.806506 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:42:36.806477 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk_odh-ai-gateway-infra(b65ba9db-2944-4e8b-979d-bd9ba3b38310)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" podUID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" Jul 17 06:42:36.807604 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:36.807590 2580 scope.go:117] "RemoveContainer" containerID="73295cef0335770169edf45ac1f25f15914d32a20c32b18f3ffa5ad0cab59504" Jul 17 06:42:36.807813 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:42:36.807795 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-28jcz_odh-ai-gateway-infra(33265d76-0807-48c0-86ee-c6a8e87e7593)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" podUID="33265d76-0807-48c0-86ee-c6a8e87e7593" Jul 17 06:42:37.255837 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:37.255752 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554df894d5-bklwn_4f0798af-769e-4f03-a93b-e020ef58711e/manager/0.log" Jul 17 06:42:37.267138 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:37.267105 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554df894d5-bklwn_4f0798af-769e-4f03-a93b-e020ef58711e/copy-manifests/0.log" Jul 17 06:42:37.848829 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:37.848796 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk"] Jul 17 06:42:37.981576 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:37.981551 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" Jul 17 06:42:38.055765 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.055724 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65ba9db-2944-4e8b-979d-bd9ba3b38310-maas-api-service-ca\") pod \"b65ba9db-2944-4e8b-979d-bd9ba3b38310\" (UID: \"b65ba9db-2944-4e8b-979d-bd9ba3b38310\") " Jul 17 06:42:38.056111 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.056091 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65ba9db-2944-4e8b-979d-bd9ba3b38310-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b65ba9db-2944-4e8b-979d-bd9ba3b38310" (UID: "b65ba9db-2944-4e8b-979d-bd9ba3b38310"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:42:38.156345 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.156316 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65ba9db-2944-4e8b-979d-bd9ba3b38310-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:42:38.268491 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.268463 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737830-2rgdb_1940eb7c-a67a-4956-89ba-8cfa548b7a10/cleanup/0.log" Jul 17 06:42:38.374687 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.374657 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-572ab047-28jcz_33265d76-0807-48c0-86ee-c6a8e87e7593/revoke-keys/1.log" Jul 17 06:42:38.596520 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.596444 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-1b3bd1-mhf7q_48b2bbef-6ea9-4c4f-9b48-41bc8b7a7a76/revoke-keys/0.log" Jul 17 06:42:38.708325 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.708297 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk_b65ba9db-2944-4e8b-979d-bd9ba3b38310/revoke-keys/2.log" Jul 17 06:42:38.818911 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.818872 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" event={"ID":"b65ba9db-2944-4e8b-979d-bd9ba3b38310","Type":"ContainerDied","Data":"5db6c0f375a9b530ab439336555dc912abaaba4aaf207ed21c7a85ff78c73e58"} Jul 17 06:42:38.818911 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.818901 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk" Jul 17 06:42:38.819153 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.818946 2580 scope.go:117] "RemoveContainer" containerID="a682b16eb6a7a4d260c1f902443aac6bef2253a7afb29fd8560dc627637798cc" Jul 17 06:42:38.820566 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.820538 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-e77d3d-wrb99_1784b9e9-4a76-4357-917a-c214afd77cf0/revoke-keys/0.log" Jul 17 06:42:38.849314 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.849256 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk"] Jul 17 06:42:38.861315 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.861292 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-mfpgk"] Jul 17 06:42:38.888882 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.888856 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" path="/var/lib/kubelet/pods/b65ba9db-2944-4e8b-979d-bd9ba3b38310/volumes" Jul 17 06:42:38.938793 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:38.938763 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-142438-d8tzs_19cdc854-30c1-4bb1-a199-2693b4b7c69e/revoke-keys/0.log" Jul 17 06:42:39.054054 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:39.054026 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-3b164bbd-c28lm_ed250e97-e673-43b0-9db0-25c99a2079a8/revoke-keys/0.log" Jul 17 06:42:39.170279 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:39.170238 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-bd7a398a-vscfq_b2282024-e221-4907-8e2d-2a9775f525d7/revoke-keys/0.log" Jul 17 06:42:39.302222 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:39.302192 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-ff4257-jqvss_5136a293-d4b9-4901-852e-dd5fe0a20f11/revoke-keys/0.log" Jul 17 06:42:41.607079 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:41.607050 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-lxk5j_77e7d293-2e85-4011-a049-7ae1d8cb7d9e/limitador/0.log" Jul 17 06:42:42.199070 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:42.199037 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-nmbk2_c9a34069-bd84-4eeb-bce4-5469f2236571/discovery/0.log" Jul 17 06:42:43.255859 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:43.255828 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-bmtcj_f8a37b28-4186-4b97-80fb-73dfd1342ebe/main/0.log" Jul 17 06:42:43.741444 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:43.741413 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-xmx2g_1230e234-d26f-4d28-a830-9e850c2c1ed7/main/0.log" Jul 17 06:42:48.884506 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:48.884461 2580 scope.go:117] "RemoveContainer" containerID="73295cef0335770169edf45ac1f25f15914d32a20c32b18f3ffa5ad0cab59504" Jul 17 06:42:49.101577 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:49.101541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:49.201616 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:49.201534 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:42:49.871626 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:49.871596 2580 generic.go:356] "Generic (PLEG): container finished" podID="33265d76-0807-48c0-86ee-c6a8e87e7593" containerID="0c1431b18e46919970d1410a0caf4a4548ba8defd1b5c14ceac6848a4dd45fae" exitCode=6 Jul 17 06:42:49.871804 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:49.871647 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" event={"ID":"33265d76-0807-48c0-86ee-c6a8e87e7593","Type":"ContainerDied","Data":"0c1431b18e46919970d1410a0caf4a4548ba8defd1b5c14ceac6848a4dd45fae"} Jul 17 06:42:49.871804 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:49.871694 2580 scope.go:117] "RemoveContainer" containerID="73295cef0335770169edf45ac1f25f15914d32a20c32b18f3ffa5ad0cab59504" Jul 17 06:42:49.872007 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:49.871989 2580 scope.go:117] "RemoveContainer" containerID="0c1431b18e46919970d1410a0caf4a4548ba8defd1b5c14ceac6848a4dd45fae" Jul 17 06:42:49.872222 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:42:49.872200 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-28jcz_odh-ai-gateway-infra(33265d76-0807-48c0-86ee-c6a8e87e7593)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" podUID="33265d76-0807-48c0-86ee-c6a8e87e7593" Jul 17 06:42:50.912963 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:50.912927 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz"] Jul 17 06:42:51.041029 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:51.041002 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" Jul 17 06:42:51.174025 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:51.173924 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33265d76-0807-48c0-86ee-c6a8e87e7593-maas-api-service-ca\") pod \"33265d76-0807-48c0-86ee-c6a8e87e7593\" (UID: \"33265d76-0807-48c0-86ee-c6a8e87e7593\") " Jul 17 06:42:51.174340 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:51.174319 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33265d76-0807-48c0-86ee-c6a8e87e7593-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33265d76-0807-48c0-86ee-c6a8e87e7593" (UID: "33265d76-0807-48c0-86ee-c6a8e87e7593"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:42:51.275522 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:51.275491 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33265d76-0807-48c0-86ee-c6a8e87e7593-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:42:51.884489 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:51.884461 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" Jul 17 06:42:51.884673 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:51.884459 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz" event={"ID":"33265d76-0807-48c0-86ee-c6a8e87e7593","Type":"ContainerDied","Data":"138eb46007746308f1f059287dee76dac96cd2ebe6a6329169c116841137bb62"} Jul 17 06:42:51.884673 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:51.884597 2580 scope.go:117] "RemoveContainer" containerID="0c1431b18e46919970d1410a0caf4a4548ba8defd1b5c14ceac6848a4dd45fae" Jul 17 06:42:51.910464 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:51.910431 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz"] Jul 17 06:42:51.915092 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:51.915070 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-28jcz"] Jul 17 06:42:52.055403 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:52.055353 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t6rpv_55c6188c-0a2e-4607-8c0e-7a5685859c4d/global-pull-secret-syncer/0.log" Jul 17 06:42:52.107994 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:52.107959 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" Jul 17 06:42:52.287727 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:52.287634 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-12.ec2.internal_ba0e125b5dfaf98ff0a6e6b6f46e0aae/haproxy/0.log" Jul 17 06:42:52.890020 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:52.889988 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33265d76-0807-48c0-86ee-c6a8e87e7593" path="/var/lib/kubelet/pods/33265d76-0807-48c0-86ee-c6a8e87e7593/volumes" Jul 17 06:42:56.924979 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.924945 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd"] Jul 17 06:42:56.925356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925255 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" containerName="revoke-keys" Jul 17 06:42:56.925356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925265 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" containerName="revoke-keys" Jul 17 06:42:56.925356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925276 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33265d76-0807-48c0-86ee-c6a8e87e7593" containerName="revoke-keys" Jul 17 06:42:56.925356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925282 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="33265d76-0807-48c0-86ee-c6a8e87e7593" containerName="revoke-keys" Jul 17 06:42:56.925356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925289 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" containerName="revoke-keys" Jul 17 06:42:56.925356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925295 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" containerName="revoke-keys" Jul 17 06:42:56.925356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925304 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" containerName="revoke-keys" Jul 17 06:42:56.925356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925309 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" containerName="revoke-keys" Jul 17 06:42:56.925356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925316 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33265d76-0807-48c0-86ee-c6a8e87e7593" containerName="revoke-keys" Jul 17 06:42:56.925356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925321 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="33265d76-0807-48c0-86ee-c6a8e87e7593" containerName="revoke-keys" Jul 17 06:42:56.925356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925330 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33265d76-0807-48c0-86ee-c6a8e87e7593" containerName="revoke-keys" Jul 17 06:42:56.925356 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925335 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="33265d76-0807-48c0-86ee-c6a8e87e7593" containerName="revoke-keys" Jul 17 06:42:56.925747 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925398 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="33265d76-0807-48c0-86ee-c6a8e87e7593" containerName="revoke-keys" Jul 17 06:42:56.925747 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925406 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" containerName="revoke-keys" Jul 17 06:42:56.925747 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925414 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="66882bf2-9245-42f3-bac8-aafc51e4e4b1" containerName="revoke-keys" Jul 17 06:42:56.925747 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925420 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" containerName="revoke-keys" Jul 17 06:42:56.925747 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925425 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="33265d76-0807-48c0-86ee-c6a8e87e7593" containerName="revoke-keys" Jul 17 06:42:56.925747 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.925430 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65ba9db-2944-4e8b-979d-bd9ba3b38310" containerName="revoke-keys" Jul 17 06:42:56.929872 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.929855 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" Jul 17 06:42:56.934028 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.934007 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:42:56.934135 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.934097 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 06:42:56.940811 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:56.940756 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd"] Jul 17 06:42:57.024862 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:57.024817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd5038-6bcc-4fef-8e95-9b4092e670c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd\" (UID: \"0cbd5038-6bcc-4fef-8e95-9b4092e670c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" Jul 17 06:42:57.125704 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:57.125666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd5038-6bcc-4fef-8e95-9b4092e670c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd\" (UID: \"0cbd5038-6bcc-4fef-8e95-9b4092e670c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" Jul 17 06:42:57.126347 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:57.126328 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd5038-6bcc-4fef-8e95-9b4092e670c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd\" (UID: \"0cbd5038-6bcc-4fef-8e95-9b4092e670c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" Jul 17 06:42:57.240368 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:57.240282 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" Jul 17 06:42:57.381237 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:57.381208 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd"] Jul 17 06:42:57.384034 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:42:57.384002 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbd5038_6bcc_4fef_8e95_9b4092e670c9.slice/crio-d64166acae839068d7037a5e4e13f8df827232d387b09b67b424cffcf4b3c2e9 WatchSource:0}: Error finding container d64166acae839068d7037a5e4e13f8df827232d387b09b67b424cffcf4b3c2e9: Status 404 returned error can't find the container with id d64166acae839068d7037a5e4e13f8df827232d387b09b67b424cffcf4b3c2e9 Jul 17 06:42:57.519605 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:57.519520 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-lxk5j_77e7d293-2e85-4011-a049-7ae1d8cb7d9e/limitador/0.log" Jul 17 06:42:57.914081 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:57.914049 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" event={"ID":"0cbd5038-6bcc-4fef-8e95-9b4092e670c9","Type":"ContainerStarted","Data":"8fb78d9e124618e9a6500873e4bc3f25647719a5dfc65222989af083489f799b"} Jul 17 06:42:57.914261 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:57.914104 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" event={"ID":"0cbd5038-6bcc-4fef-8e95-9b4092e670c9","Type":"ContainerStarted","Data":"d64166acae839068d7037a5e4e13f8df827232d387b09b67b424cffcf4b3c2e9"} Jul 17 06:42:57.934591 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:57.934536 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" podStartSLOduration=1.934521926 podStartE2EDuration="1.934521926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:42:57.93337963 +0000 UTC m=+3702.678058982" watchObservedRunningTime="2026-07-17 06:42:57.934521926 +0000 UTC m=+3702.679201253" Jul 17 06:42:58.920122 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:58.920085 2580 generic.go:356] "Generic (PLEG): container finished" podID="0cbd5038-6bcc-4fef-8e95-9b4092e670c9" containerID="8fb78d9e124618e9a6500873e4bc3f25647719a5dfc65222989af083489f799b" exitCode=6 Jul 17 06:42:58.920292 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:58.920151 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" event={"ID":"0cbd5038-6bcc-4fef-8e95-9b4092e670c9","Type":"ContainerDied","Data":"8fb78d9e124618e9a6500873e4bc3f25647719a5dfc65222989af083489f799b"} Jul 17 06:42:58.920371 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:58.920359 2580 scope.go:117] "RemoveContainer" containerID="8fb78d9e124618e9a6500873e4bc3f25647719a5dfc65222989af083489f799b" Jul 17 06:42:59.132926 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.132898 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0243024d-ed96-44ac-8b41-e334fbd21648/alertmanager/0.log" Jul 17 06:42:59.157551 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.157528 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0243024d-ed96-44ac-8b41-e334fbd21648/config-reloader/0.log" Jul 17 06:42:59.186513 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.186491 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0243024d-ed96-44ac-8b41-e334fbd21648/kube-rbac-proxy-web/0.log" Jul 17 06:42:59.208250 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.208224 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0243024d-ed96-44ac-8b41-e334fbd21648/kube-rbac-proxy/0.log" Jul 17 06:42:59.237582 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.237557 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0243024d-ed96-44ac-8b41-e334fbd21648/kube-rbac-proxy-metric/0.log" Jul 17 06:42:59.281188 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.281160 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0243024d-ed96-44ac-8b41-e334fbd21648/prom-label-proxy/0.log" Jul 17 06:42:59.315569 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.315542 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0243024d-ed96-44ac-8b41-e334fbd21648/init-config-reloader/0.log" Jul 17 06:42:59.703167 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.703140 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q9fj7_377bf2c4-8523-4d6d-a342-ffe1ec22daf2/node-exporter/0.log" Jul 17 06:42:59.726273 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.726226 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q9fj7_377bf2c4-8523-4d6d-a342-ffe1ec22daf2/kube-rbac-proxy/0.log" Jul 17 06:42:59.762473 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.762448 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q9fj7_377bf2c4-8523-4d6d-a342-ffe1ec22daf2/init-textfile/0.log" Jul 17 06:42:59.794133 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.794042 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-hlp5g_f8d44361-f45e-42bb-bdb6-d4c6a071f8d6/kube-rbac-proxy-main/0.log" Jul 17 06:42:59.816203 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.816179 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-hlp5g_f8d44361-f45e-42bb-bdb6-d4c6a071f8d6/kube-rbac-proxy-self/0.log" Jul 17 06:42:59.839857 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.839820 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-hlp5g_f8d44361-f45e-42bb-bdb6-d4c6a071f8d6/openshift-state-metrics/0.log" Jul 17 06:42:59.883793 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.883762 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_15cb2621-c194-4892-ae7b-dc9ee28f2db2/prometheus/0.log" Jul 17 06:42:59.906697 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.906669 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_15cb2621-c194-4892-ae7b-dc9ee28f2db2/config-reloader/0.log" Jul 17 06:42:59.926293 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.926261 2580 generic.go:356] "Generic (PLEG): container finished" podID="0cbd5038-6bcc-4fef-8e95-9b4092e670c9" containerID="b35f17cee57c6defb94a3b9583e4dc893708f8ebfea3ea85ad6a27ac714f1eb5" exitCode=6 Jul 17 06:42:59.926472 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.926352 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" event={"ID":"0cbd5038-6bcc-4fef-8e95-9b4092e670c9","Type":"ContainerDied","Data":"b35f17cee57c6defb94a3b9583e4dc893708f8ebfea3ea85ad6a27ac714f1eb5"} Jul 17 06:42:59.926472 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.926458 2580 scope.go:117] "RemoveContainer" containerID="8fb78d9e124618e9a6500873e4bc3f25647719a5dfc65222989af083489f799b" Jul 17 06:42:59.926674 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.926654 2580 scope.go:117] "RemoveContainer" containerID="b35f17cee57c6defb94a3b9583e4dc893708f8ebfea3ea85ad6a27ac714f1eb5" Jul 17 06:42:59.926882 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:42:59.926863 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd_odh-ai-gateway-infra(0cbd5038-6bcc-4fef-8e95-9b4092e670c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" podUID="0cbd5038-6bcc-4fef-8e95-9b4092e670c9" Jul 17 06:42:59.929627 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.929573 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_15cb2621-c194-4892-ae7b-dc9ee28f2db2/thanos-sidecar/0.log" Jul 17 06:42:59.957815 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.957795 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_15cb2621-c194-4892-ae7b-dc9ee28f2db2/kube-rbac-proxy-web/0.log" Jul 17 06:42:59.983002 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:42:59.982969 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_15cb2621-c194-4892-ae7b-dc9ee28f2db2/kube-rbac-proxy/0.log" Jul 17 06:43:00.011792 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:00.011763 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_15cb2621-c194-4892-ae7b-dc9ee28f2db2/kube-rbac-proxy-thanos/0.log" Jul 17 06:43:00.034019 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:00.033994 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_15cb2621-c194-4892-ae7b-dc9ee28f2db2/init-config-reloader/0.log" Jul 17 06:43:00.932888 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:00.932861 2580 scope.go:117] "RemoveContainer" containerID="b35f17cee57c6defb94a3b9583e4dc893708f8ebfea3ea85ad6a27ac714f1eb5" Jul 17 06:43:00.933276 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:43:00.933031 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd_odh-ai-gateway-infra(0cbd5038-6bcc-4fef-8e95-9b4092e670c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" podUID="0cbd5038-6bcc-4fef-8e95-9b4092e670c9" Jul 17 06:43:02.469666 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.469623 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg"] Jul 17 06:43:02.470110 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.469999 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="33265d76-0807-48c0-86ee-c6a8e87e7593" containerName="revoke-keys" Jul 17 06:43:02.473969 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.473948 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.476592 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.476567 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bbz7r\"/\"kube-root-ca.crt\"" Jul 17 06:43:02.477453 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.477432 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bbz7r\"/\"default-dockercfg-rdkgz\"" Jul 17 06:43:02.477536 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.477453 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bbz7r\"/\"openshift-service-ca.crt\"" Jul 17 06:43:02.498103 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.498069 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg"] Jul 17 06:43:02.576934 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.576902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-proc\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.576934 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.576936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-lib-modules\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.577157 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.576962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-podres\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.577157 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.577026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8mr\" (UniqueName: \"kubernetes.io/projected/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-kube-api-access-ch8mr\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.577157 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.577119 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-sys\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.677751 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.677712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-sys\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.677924 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.677788 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-proc\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.677924 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.677812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-lib-modules\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.677924 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.677845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-podres\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.677924 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.677849 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-sys\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.678062 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.677922 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-proc\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.678062 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.677920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8mr\" (UniqueName: \"kubernetes.io/projected/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-kube-api-access-ch8mr\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.678062 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.677955 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-lib-modules\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.678062 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.677963 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-podres\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.686764 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.686733 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8mr\" (UniqueName: \"kubernetes.io/projected/c7d97224-88ef-49c9-9d73-9ed45b96cb4e-kube-api-access-ch8mr\") pod \"perf-node-gather-daemonset-9xftg\" (UID: \"c7d97224-88ef-49c9-9d73-9ed45b96cb4e\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.783557 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.783472 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:02.935116 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:02.935073 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg"] Jul 17 06:43:02.939743 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:43:02.939718 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc7d97224_88ef_49c9_9d73_9ed45b96cb4e.slice/crio-ab20135d10395e3f8a613fccb1a3dc62ab33357a1ba266699d0b304a029f0bdb WatchSource:0}: Error finding container ab20135d10395e3f8a613fccb1a3dc62ab33357a1ba266699d0b304a029f0bdb: Status 404 returned error can't find the container with id ab20135d10395e3f8a613fccb1a3dc62ab33357a1ba266699d0b304a029f0bdb Jul 17 06:43:03.948155 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:03.948120 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" event={"ID":"c7d97224-88ef-49c9-9d73-9ed45b96cb4e","Type":"ContainerStarted","Data":"a9278995dce4a8ad608e1487a5d2f9b0b085e38435f0158a811dcdcc1d30ecae"} Jul 17 06:43:03.948622 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:03.948172 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" event={"ID":"c7d97224-88ef-49c9-9d73-9ed45b96cb4e","Type":"ContainerStarted","Data":"ab20135d10395e3f8a613fccb1a3dc62ab33357a1ba266699d0b304a029f0bdb"} Jul 17 06:43:03.948622 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:03.948238 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:03.977708 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:03.977559 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" podStartSLOduration=1.9775407569999999 podStartE2EDuration="1.977540757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 06:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:43:03.972863961 +0000 UTC m=+3708.717543289" watchObservedRunningTime="2026-07-17 06:43:03.977540757 +0000 UTC m=+3708.722220085" Jul 17 06:43:04.776720 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:04.776682 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg"] Jul 17 06:43:04.780132 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:04.780109 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg" Jul 17 06:43:04.791820 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:04.791791 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg"] Jul 17 06:43:04.841953 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:04.841925 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nhv84_f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe/dns/0.log" Jul 17 06:43:04.876096 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:04.876069 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nhv84_f83f0f4a-a71a-4ce1-a4f6-8ce7ce17f3fe/kube-rbac-proxy/0.log" Jul 17 06:43:04.898395 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:04.898364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/469ed3b2-6f3c-4207-8164-bb3b581cc2cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg\" (UID: \"469ed3b2-6f3c-4207-8164-bb3b581cc2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg" Jul 17 06:43:04.987332 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:04.987298 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gw9qs_ca6d89cf-4331-4c48-91b6-5e6b570d8927/dns-node-resolver/0.log" Jul 17 06:43:04.999450 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:04.999421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/469ed3b2-6f3c-4207-8164-bb3b581cc2cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg\" (UID: \"469ed3b2-6f3c-4207-8164-bb3b581cc2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg" Jul 17 06:43:05.000060 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:05.000037 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/469ed3b2-6f3c-4207-8164-bb3b581cc2cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg\" (UID: \"469ed3b2-6f3c-4207-8164-bb3b581cc2cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg" Jul 17 06:43:05.090526 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:05.090428 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg" Jul 17 06:43:05.221843 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:05.221811 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg"] Jul 17 06:43:05.226135 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:43:05.226110 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469ed3b2_6f3c_4207_8164_bb3b581cc2cd.slice/crio-952f2edb793c1a90b0b876af1725b171026e6188fa13aa64d9c7633a18ac5a38 WatchSource:0}: Error finding container 952f2edb793c1a90b0b876af1725b171026e6188fa13aa64d9c7633a18ac5a38: Status 404 returned error can't find the container with id 952f2edb793c1a90b0b876af1725b171026e6188fa13aa64d9c7633a18ac5a38 Jul 17 06:43:05.667296 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:05.667266 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xqr2c_e3a2951e-e5a2-49fc-a938-a5617748d2bb/node-ca/0.log" Jul 17 06:43:05.960445 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:05.960411 2580 generic.go:356] "Generic (PLEG): container finished" podID="469ed3b2-6f3c-4207-8164-bb3b581cc2cd" containerID="79acc0fc8c84760a3f35929161a1b3ea17cd94d2269c27438873f0c9704ba20c" exitCode=6 Jul 17 06:43:05.960596 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:05.960472 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg" event={"ID":"469ed3b2-6f3c-4207-8164-bb3b581cc2cd","Type":"ContainerDied","Data":"79acc0fc8c84760a3f35929161a1b3ea17cd94d2269c27438873f0c9704ba20c"} Jul 17 06:43:05.960596 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:05.960514 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg" event={"ID":"469ed3b2-6f3c-4207-8164-bb3b581cc2cd","Type":"ContainerStarted","Data":"952f2edb793c1a90b0b876af1725b171026e6188fa13aa64d9c7633a18ac5a38"} Jul 17 06:43:05.960711 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:05.960696 2580 scope.go:117] "RemoveContainer" containerID="79acc0fc8c84760a3f35929161a1b3ea17cd94d2269c27438873f0c9704ba20c" Jul 17 06:43:06.884066 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:06.884027 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-nmbk2_c9a34069-bd84-4eeb-bce4-5469f2236571/discovery/0.log" Jul 17 06:43:06.967408 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:06.967310 2580 generic.go:356] "Generic (PLEG): container finished" podID="469ed3b2-6f3c-4207-8164-bb3b581cc2cd" containerID="f158a000610a701f58b1086d3fab3b7066c6dcc49f076f95ca9628def649eff2" exitCode=6 Jul 17 06:43:06.967408 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:06.967371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg" event={"ID":"469ed3b2-6f3c-4207-8164-bb3b581cc2cd","Type":"ContainerDied","Data":"f158a000610a701f58b1086d3fab3b7066c6dcc49f076f95ca9628def649eff2"} Jul 17 06:43:06.967614 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:06.967439 2580 scope.go:117] "RemoveContainer" containerID="79acc0fc8c84760a3f35929161a1b3ea17cd94d2269c27438873f0c9704ba20c" Jul 17 06:43:06.967666 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:06.967653 2580 scope.go:117] "RemoveContainer" containerID="f158a000610a701f58b1086d3fab3b7066c6dcc49f076f95ca9628def649eff2" Jul 17 06:43:06.967926 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:43:06.967906 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg_odh-ai-gateway-infra(469ed3b2-6f3c-4207-8164-bb3b581cc2cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg" podUID="469ed3b2-6f3c-4207-8164-bb3b581cc2cd" Jul 17 06:43:07.974396 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:07.974365 2580 scope.go:117] "RemoveContainer" containerID="f158a000610a701f58b1086d3fab3b7066c6dcc49f076f95ca9628def649eff2" Jul 17 06:43:07.974817 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:43:07.974575 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg_odh-ai-gateway-infra(469ed3b2-6f3c-4207-8164-bb3b581cc2cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-z8tgg" podUID="469ed3b2-6f3c-4207-8164-bb3b581cc2cd" Jul 17 06:43:08.008231 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:08.008202 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dsrvg_3fde696a-f615-4e9f-b866-84073d4a923b/serve-healthcheck-canary/0.log" Jul 17 06:43:09.093677 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:09.093651 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vfk9k_9d52e702-a8bf-44dd-887b-f16b64fa0e7a/kube-rbac-proxy/0.log" Jul 17 06:43:09.165706 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:09.165677 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vfk9k_9d52e702-a8bf-44dd-887b-f16b64fa0e7a/exporter/0.log" Jul 17 06:43:09.220325 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:09.220293 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vfk9k_9d52e702-a8bf-44dd-887b-f16b64fa0e7a/extractor/0.log" Jul 17 06:43:09.963222 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:09.963197 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-9xftg" Jul 17 06:43:10.840258 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:10.840223 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-svrmx"] Jul 17 06:43:10.845283 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:10.845264 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-svrmx" Jul 17 06:43:10.861473 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:10.861437 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-svrmx"] Jul 17 06:43:10.945171 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:10.945137 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b11a5d9-7595-4b97-bf6a-eb1e5ecae349-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-svrmx\" (UID: \"2b11a5d9-7595-4b97-bf6a-eb1e5ecae349\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-svrmx" Jul 17 06:43:11.046342 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:11.046308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b11a5d9-7595-4b97-bf6a-eb1e5ecae349-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-svrmx\" (UID: \"2b11a5d9-7595-4b97-bf6a-eb1e5ecae349\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-svrmx" Jul 17 06:43:11.046916 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:11.046897 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b11a5d9-7595-4b97-bf6a-eb1e5ecae349-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-svrmx\" (UID: \"2b11a5d9-7595-4b97-bf6a-eb1e5ecae349\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-svrmx" Jul 17 06:43:11.155239 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:11.155198 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-svrmx" Jul 17 06:43:11.298437 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:11.298401 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-svrmx"] Jul 17 06:43:11.304541 ip-10-0-142-12 kubenswrapper[2580]: W0717 06:43:11.304515 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b11a5d9_7595_4b97_bf6a_eb1e5ecae349.slice/crio-dfa759f9606648e7fb2b8444781b5af1024c43ffbf7fce212c69d6a952856c7c WatchSource:0}: Error finding container dfa759f9606648e7fb2b8444781b5af1024c43ffbf7fce212c69d6a952856c7c: Status 404 returned error can't find the container with id dfa759f9606648e7fb2b8444781b5af1024c43ffbf7fce212c69d6a952856c7c Jul 17 06:43:11.884740 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:11.884714 2580 scope.go:117] "RemoveContainer" containerID="b35f17cee57c6defb94a3b9583e4dc893708f8ebfea3ea85ad6a27ac714f1eb5" Jul 17 06:43:11.993950 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:11.993915 2580 generic.go:356] "Generic (PLEG): container finished" podID="2b11a5d9-7595-4b97-bf6a-eb1e5ecae349" containerID="17fcb4b811e4a36330f26a610a1798ce75d6d6d4dfd15ae01c97272f9b014d1d" exitCode=6 Jul 17 06:43:11.994146 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:11.993966 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-svrmx" event={"ID":"2b11a5d9-7595-4b97-bf6a-eb1e5ecae349","Type":"ContainerDied","Data":"17fcb4b811e4a36330f26a610a1798ce75d6d6d4dfd15ae01c97272f9b014d1d"} Jul 17 06:43:11.994146 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:11.994012 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-svrmx" event={"ID":"2b11a5d9-7595-4b97-bf6a-eb1e5ecae349","Type":"ContainerStarted","Data":"dfa759f9606648e7fb2b8444781b5af1024c43ffbf7fce212c69d6a952856c7c"} Jul 17 06:43:11.994277 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:11.994190 2580 scope.go:117] "RemoveContainer" containerID="17fcb4b811e4a36330f26a610a1798ce75d6d6d4dfd15ae01c97272f9b014d1d" Jul 17 06:43:12.271069 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:12.271036 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:43:12.417626 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:12.417546 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7phbz_12d8ad5a-f134-43f0-a882-1e4c6eaa9b5c/konnectivity-agent/0.log" limit=1024 Jul 17 06:43:12.639769 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:12.639734 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-nmbk2_c9a34069-bd84-4eeb-bce4-5469f2236571/discovery/0.log" Jul 17 06:43:12.774578 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:12.774548 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554df894d5-bklwn_4f0798af-769e-4f03-a93b-e020ef58711e/manager/0.log" Jul 17 06:43:12.849227 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:12.849199 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554df894d5-bklwn_4f0798af-769e-4f03-a93b-e020ef58711e/copy-manifests/0.log" Jul 17 06:43:13.000544 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:13.000446 2580 generic.go:356] "Generic (PLEG): container finished" podID="2b11a5d9-7595-4b97-bf6a-eb1e5ecae349" containerID="6f0e1f923e78b2a7c38ed401fd6f028e3b5c44e007ba6a99ea13af5eba5ad941" exitCode=6 Jul 17 06:43:13.000993 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:13.000537 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-svrmx" event={"ID":"2b11a5d9-7595-4b97-bf6a-eb1e5ecae349","Type":"ContainerDied","Data":"6f0e1f923e78b2a7c38ed401fd6f028e3b5c44e007ba6a99ea13af5eba5ad941"} Jul 17 06:43:13.000993 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:13.000596 2580 scope.go:117] "RemoveContainer" containerID="17fcb4b811e4a36330f26a610a1798ce75d6d6d4dfd15ae01c97272f9b014d1d" Jul 17 06:43:13.000993 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:13.000786 2580 scope.go:117] "RemoveContainer" containerID="6f0e1f923e78b2a7c38ed401fd6f028e3b5c44e007ba6a99ea13af5eba5ad941" Jul 17 06:43:13.001163 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:43:13.001009 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-svrmx_odh-ai-gateway-infra(2b11a5d9-7595-4b97-bf6a-eb1e5ecae349)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-svrmx" podUID="2b11a5d9-7595-4b97-bf6a-eb1e5ecae349" Jul 17 06:43:13.002475 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:13.002457 2580 generic.go:356] "Generic (PLEG): container finished" podID="0cbd5038-6bcc-4fef-8e95-9b4092e670c9" containerID="c2aedf186fa356923b1885c9383f885d81af711bbf5d8b087d5864dd5eee9ed1" exitCode=6 Jul 17 06:43:13.002580 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:13.002493 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" event={"ID":"0cbd5038-6bcc-4fef-8e95-9b4092e670c9","Type":"ContainerDied","Data":"c2aedf186fa356923b1885c9383f885d81af711bbf5d8b087d5864dd5eee9ed1"} Jul 17 06:43:13.002740 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:13.002727 2580 scope.go:117] "RemoveContainer" containerID="c2aedf186fa356923b1885c9383f885d81af711bbf5d8b087d5864dd5eee9ed1" Jul 17 06:43:13.002916 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:43:13.002901 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd_odh-ai-gateway-infra(0cbd5038-6bcc-4fef-8e95-9b4092e670c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" podUID="0cbd5038-6bcc-4fef-8e95-9b4092e670c9" Jul 17 06:43:13.011473 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:13.011454 2580 scope.go:117] "RemoveContainer" containerID="b35f17cee57c6defb94a3b9583e4dc893708f8ebfea3ea85ad6a27ac714f1eb5" Jul 17 06:43:14.011615 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:14.011585 2580 scope.go:117] "RemoveContainer" containerID="6f0e1f923e78b2a7c38ed401fd6f028e3b5c44e007ba6a99ea13af5eba5ad941" Jul 17 06:43:14.012138 ip-10-0-142-12 kubenswrapper[2580]: E0717 06:43:14.011817 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-svrmx_odh-ai-gateway-infra(2b11a5d9-7595-4b97-bf6a-eb1e5ecae349)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-svrmx" podUID="2b11a5d9-7595-4b97-bf6a-eb1e5ecae349" Jul 17 06:43:14.074949 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:14.074916 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd"] Jul 17 06:43:14.206120 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:14.206061 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" Jul 17 06:43:14.272018 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:14.271984 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd5038-6bcc-4fef-8e95-9b4092e670c9-maas-api-service-ca\") pod \"0cbd5038-6bcc-4fef-8e95-9b4092e670c9\" (UID: \"0cbd5038-6bcc-4fef-8e95-9b4092e670c9\") " Jul 17 06:43:14.272338 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:14.272315 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbd5038-6bcc-4fef-8e95-9b4092e670c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0cbd5038-6bcc-4fef-8e95-9b4092e670c9" (UID: "0cbd5038-6bcc-4fef-8e95-9b4092e670c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:43:14.372969 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:14.372937 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd5038-6bcc-4fef-8e95-9b4092e670c9-maas-api-service-ca\") on node \"ip-10-0-142-12.ec2.internal\" DevicePath \"\"" Jul 17 06:43:14.464980 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:14.464891 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-nmbk2_c9a34069-bd84-4eeb-bce4-5469f2236571/discovery/0.log" Jul 17 06:43:14.722911 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:14.722832 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d9bdd98d9-rt4lx_f9010838-12a1-4913-bf38-cb360367770f/manager/0.log" Jul 17 06:43:15.017828 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:15.017746 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" event={"ID":"0cbd5038-6bcc-4fef-8e95-9b4092e670c9","Type":"ContainerDied","Data":"d64166acae839068d7037a5e4e13f8df827232d387b09b67b424cffcf4b3c2e9"} Jul 17 06:43:15.017828 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:15.017796 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd" Jul 17 06:43:15.017828 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:15.017817 2580 scope.go:117] "RemoveContainer" containerID="c2aedf186fa356923b1885c9383f885d81af711bbf5d8b087d5864dd5eee9ed1" Jul 17 06:43:15.040738 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:15.040712 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd"] Jul 17 06:43:15.043946 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:15.043923 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-m8kgd"] Jul 17 06:43:16.890262 ip-10-0-142-12 kubenswrapper[2580]: I0717 06:43:16.890228 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cbd5038-6bcc-4fef-8e95-9b4092e670c9" path="/var/lib/kubelet/pods/0cbd5038-6bcc-4fef-8e95-9b4092e670c9/volumes"