Aug 04 10:36:50.775820 ip-10-0-131-159 systemd[1]: Starting Kubernetes Kubelet... Aug 04 10:36:51.222164 ip-10-0-131-159 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 10:36:51.222164 ip-10-0-131-159 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 10:36:51.222164 ip-10-0-131-159 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 10:36:51.222164 ip-10-0-131-159 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 10:36:51.222164 ip-10-0-131-159 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 10:36:51.224283 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.224166 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 10:36:51.234291 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234261 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:36:51.234291 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234284 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:36:51.234291 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234288 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:36:51.234291 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234292 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:36:51.234291 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234295 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:36:51.234291 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234298 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:36:51.234291 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234301 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234306 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234309 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234312 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234314 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234317 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234320 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234323 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234326 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234328 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234331 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234334 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234336 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234339 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234342 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234344 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234347 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234350 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234352 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234360 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:36:51.234589 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234363 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234366 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234369 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234371 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234374 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234376 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234379 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234382 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234384 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234387 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234391 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234395 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234399 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234403 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234407 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234410 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234413 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234416 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:36:51.235064 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234419 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234422 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234424 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234427 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234430 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234433 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234435 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234438 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234441 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234443 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234446 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234449 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234452 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234455 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234457 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234460 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234463 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234465 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234468 2576 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234471 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:36:51.235518 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234474 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234477 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234480 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234482 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234485 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234489 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234492 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234495 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234513 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234517 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234520 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234523 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234526 2576 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234529 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234532 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234535 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234537 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234540 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234543 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:36:51.236044 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234546 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234549 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234552 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234554 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234557 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234560 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234563 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234565 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234570 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234573 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234576 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234578 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234581 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234584 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234586 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.234589 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235320 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235326 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235329 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235332 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:36:51.236515 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235335 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235340 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235345 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235348 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235351 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235354 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235357 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235360 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235362 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235365 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235367 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235370 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235373 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235376 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235379 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235382 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235384 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235387 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235390 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:36:51.236998 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235392 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235395 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235397 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235401 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235404 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235407 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235410 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235412 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235414 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235417 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235420 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235423 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235426 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235429 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235431 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235434 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235436 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235439 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235441 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:36:51.237467 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235444 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235447 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235450 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235452 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235455 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235457 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235462 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235466 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235469 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235471 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235474 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235477 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235479 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235482 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235484 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235487 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235490 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235492 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235495 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235498 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:36:51.237945 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235517 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235520 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235523 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235526 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235528 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235532 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235534 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235537 2576 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235540 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235543 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235545 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235548 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235551 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235555 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235558 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235560 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235564 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235568 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235571 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235574 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:36:51.238437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235577 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235581 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235584 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235587 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235590 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235592 2576 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235595 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235597 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235600 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235603 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235605 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235608 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235610 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235613 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235615 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235619 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.235621 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236028 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236035 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:36:51.238952 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236039 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236043 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236046 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236050 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236053 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236056 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236058 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236061 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236064 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236066 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236069 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236072 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236075 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236078 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236125 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236129 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236132 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236135 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236138 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:36:51.239411 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236140 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236143 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236146 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236148 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236152 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236154 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236157 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236160 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236163 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236165 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236168 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236171 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236173 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236176 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236178 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236181 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236184 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236186 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236189 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236192 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:36:51.239896 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236195 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236197 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236200 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236203 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236205 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236208 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236211 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236214 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236217 2576 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236220 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236223 2576 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236225 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236229 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236232 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236235 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236237 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236240 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236242 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236245 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236248 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:36:51.240427 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236251 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236253 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236256 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236259 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236261 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236264 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236266 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236270 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236274 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236276 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236279 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236282 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236285 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236288 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236290 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236294 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236298 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236301 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236307 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236310 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:36:51.240921 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236313 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236316 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236318 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236321 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236324 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236326 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236329 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236332 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236334 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236337 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236340 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236343 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236346 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236349 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236352 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236355 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236357 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.236360 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236434 2576 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236442 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236448 2576 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 10:36:51.241395 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236453 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236457 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236460 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236465 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236469 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236473 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236478 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236481 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236485 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236489 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236492 2576 flags.go:64] FLAG: --cgroup-root="" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236495 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236511 2576 flags.go:64] FLAG: --client-ca-file="" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236515 2576 flags.go:64] FLAG: --cloud-provider="external" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236518 2576 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236524 2576 flags.go:64] FLAG: --cluster-domain="" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236527 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236530 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236533 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236537 2576 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236540 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236544 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236547 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 10:36:51.241944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236551 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236555 2576 flags.go:64] FLAG: --contention-profiling="false" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236558 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236561 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236564 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236567 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236571 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236575 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236578 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236581 2576 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236584 2576 flags.go:64] FLAG: --enable-server="true" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236587 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236591 2576 flags.go:64] FLAG: --event-burst="100" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236594 2576 flags.go:64] FLAG: --event-qps="50" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236597 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236601 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236604 2576 flags.go:64] FLAG: --eviction-hard="" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236613 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236616 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236619 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236623 2576 flags.go:64] FLAG: --eviction-soft="" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236626 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236628 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236632 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236635 2576 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 10:36:51.242520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236638 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236641 2576 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236644 2576 flags.go:64] FLAG: --feature-gates="" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236648 2576 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236650 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236654 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236657 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236660 2576 flags.go:64] FLAG: --healthz-port="10248" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236663 2576 flags.go:64] FLAG: --help="false" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236666 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236669 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236671 2576 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236674 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236677 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236681 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236684 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236687 2576 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236689 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236692 2576 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236695 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236698 2576 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236701 2576 flags.go:64] FLAG: --kube-reserved="" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236705 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236707 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 10:36:51.243140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236711 2576 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236714 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236717 2576 flags.go:64] FLAG: --lock-file="" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236720 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236723 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236726 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236732 2576 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236735 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236738 2576 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236741 2576 flags.go:64] FLAG: --logging-format="text" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236744 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236747 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236750 2576 flags.go:64] FLAG: --manifest-url="" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236753 2576 flags.go:64] FLAG: --manifest-url-header="" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236757 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236760 2576 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236764 2576 flags.go:64] FLAG: --max-pods="110" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236767 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236770 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236773 2576 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236776 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236779 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236782 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236785 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236793 2576 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 10:36:51.243761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236796 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236799 2576 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236802 2576 flags.go:64] FLAG: --pod-cidr="" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236805 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236811 2576 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236814 2576 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236818 2576 flags.go:64] FLAG: --pods-per-core="0" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236821 2576 flags.go:64] FLAG: --port="10250" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236824 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236827 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03c7793113e85c359" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236830 2576 flags.go:64] FLAG: --qos-reserved="" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236834 2576 flags.go:64] FLAG: --read-only-port="10255" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236837 2576 flags.go:64] FLAG: --register-node="true" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236839 2576 flags.go:64] FLAG: --register-with-taints="" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236843 2576 flags.go:64] FLAG: --registry-burst="10" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236847 2576 flags.go:64] FLAG: --registry-qps="5" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236850 2576 flags.go:64] FLAG: --reserved-cpus="" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236853 2576 flags.go:64] FLAG: --reserved-memory="" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236856 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236859 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236863 2576 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236865 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236869 2576 flags.go:64] FLAG: --runonce="false" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236872 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236875 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 10:36:51.244770 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236878 2576 flags.go:64] FLAG: --seccomp-default="false" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236881 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236884 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236887 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236890 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236893 2576 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236896 2576 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236899 2576 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236902 2576 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236905 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236908 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236910 2576 flags.go:64] FLAG: --system-cgroups="" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236913 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236919 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236922 2576 flags.go:64] FLAG: --tls-cert-file="" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236924 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236929 2576 flags.go:64] FLAG: --tls-min-version="" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236932 2576 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236935 2576 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236938 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236941 2576 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236944 2576 flags.go:64] FLAG: --v="2" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236948 2576 flags.go:64] FLAG: --version="false" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236953 2576 flags.go:64] FLAG: --vmodule="" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236956 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 10:36:51.245933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.236960 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237078 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237082 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237086 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237089 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237092 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237095 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237099 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237102 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237105 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237108 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237110 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237113 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237116 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237118 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237121 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237124 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237126 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237129 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:36:51.246992 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237131 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237134 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237137 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237139 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237143 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237146 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237149 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237151 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237154 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237157 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237159 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237162 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237166 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237171 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237175 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237178 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237182 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237185 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237187 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:36:51.247855 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237190 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237193 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237196 2576 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237199 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237201 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237204 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237206 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237209 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237212 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237214 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237217 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237220 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237223 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237225 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237228 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237230 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237233 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237237 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237240 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237243 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:36:51.248686 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237245 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237248 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237251 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237254 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237257 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237259 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237262 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237265 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237268 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237272 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237274 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237277 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237280 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237282 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237285 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237287 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237290 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237293 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237296 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237298 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:36:51.249231 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237301 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237303 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237306 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237309 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237311 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237314 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237317 2576 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237319 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237322 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237325 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237328 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237331 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237333 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237336 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237339 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237342 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237344 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237347 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237350 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237354 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:36:51.249849 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237357 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.237361 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.238098 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.246704 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.246731 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.246760 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.246775 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246871 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246879 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246884 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246889 2576 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246893 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246898 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246902 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246907 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:36:51.250452 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246912 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246916 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246920 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246925 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246929 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246934 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246938 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246943 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246947 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246952 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246956 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246961 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246965 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246972 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246978 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246983 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246987 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246991 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.246996 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:36:51.251104 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247000 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247006 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247020 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247026 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247031 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247036 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247040 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247045 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247050 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247054 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247059 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247066 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247071 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247076 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247081 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247087 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247093 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247098 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247102 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:36:51.251895 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247107 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247112 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247116 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247120 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247125 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247129 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247134 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247138 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247142 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247146 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247150 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247155 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247159 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247163 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247168 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247173 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247186 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247190 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247194 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247198 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:36:51.252365 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247203 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247208 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247213 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247217 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247221 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247226 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247230 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247234 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247238 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247242 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247247 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247251 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247256 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247260 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247264 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247269 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247273 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247278 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247282 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247286 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:36:51.253061 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247290 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247295 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247299 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247304 2576 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247308 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247313 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247317 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247322 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247326 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247338 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247343 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247347 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247351 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.247360 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247617 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:36:51.253642 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247628 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247633 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247637 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247642 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247647 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247652 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247656 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247661 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247665 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247670 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247674 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247678 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247682 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247687 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247691 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247696 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247700 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247704 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247709 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247713 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:36:51.254035 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247717 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247721 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247725 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247732 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247740 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247745 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247749 2576 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247761 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247766 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247770 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247774 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247778 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247783 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247787 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247792 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247796 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247800 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247804 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247809 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247813 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:36:51.254563 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247817 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247821 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247826 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247830 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247835 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247839 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247843 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247848 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247852 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247857 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247861 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247865 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247869 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247873 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247877 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247885 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247891 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247895 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247901 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:36:51.255046 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247905 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247917 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247922 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247926 2576 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247933 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247938 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247943 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247948 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247953 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247957 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247962 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247966 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247971 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247975 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247979 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247983 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247988 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247992 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.247997 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248001 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:36:51.255590 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248005 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248010 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248015 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248019 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248023 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248028 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248032 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248036 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248041 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248045 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248050 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248054 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248058 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248063 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248075 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248080 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248084 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248088 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:36:51.256073 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:51.248093 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:36:51.256658 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.248101 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 10:36:51.256658 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.248305 2576 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 10:36:51.256658 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.252683 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 10:36:51.256658 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.255296 2576 server.go:1019] "Starting client certificate rotation" Aug 04 10:36:51.256658 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.255410 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 10:36:51.256658 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.255450 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 10:36:51.258878 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.258857 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 10:36:51.260879 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.260857 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 10:36:51.272577 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.272557 2576 log.go:25] "Validated CRI v1 runtime API" Aug 04 10:36:51.276492 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.276470 2576 log.go:25] "Validated CRI v1 image API" Aug 04 10:36:51.278173 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.277946 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 10:36:51.281397 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.281373 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b1a96232-e958-41d2-b6e1-641f0257c2f1:/dev/nvme0n1p3 bd97903c-ce7b-4a42-aa5f-faeab2c4b291:/dev/nvme0n1p4] Aug 04 10:36:51.281488 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.281397 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 10:36:51.286685 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.286571 2576 manager.go:216] Machine: {Timestamp:2026-08-04 10:36:51.286050743 +0000 UTC m=+0.383844129 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098733 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec216d8203b349550e365b762b363a19 SystemUUID:ec216d82-03b3-4955-0e36-5b762b363a19 BootID:b2b76e18-fc32-4d88-9701-a412ebe8b55f Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:9c:5e:ad:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:9c:5e:ad:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:ef:e5:2b:df:47 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 10:36:51.286685 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.286679 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 10:36:51.286782 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.286767 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 10:36:51.290256 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.290222 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 10:36:51.290402 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.290259 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-159.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 10:36:51.291125 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.291112 2576 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 10:36:51.291162 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.291129 2576 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 10:36:51.291162 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.291137 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 10:36:51.291162 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.291149 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 10:36:51.293033 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.293022 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 10:36:51.293907 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.293892 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 10:36:51.298343 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.298331 2576 state_mem.go:36] "Initialized new in-memory state store" Aug 04 10:36:51.298479 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.298469 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 10:36:51.298528 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.298514 2576 kubelet.go:486] "Attempting to sync node with API server" Aug 04 10:36:51.298528 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.298527 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 10:36:51.298589 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.298542 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 10:36:51.298589 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.298553 2576 kubelet.go:398] "Adding apiserver pod source" Aug 04 10:36:51.298589 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.298563 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 10:36:51.299007 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.298995 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 10:36:51.299049 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.299018 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 10:36:51.301438 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.301422 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 10:36:51.303561 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.303547 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 10:36:51.303606 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.303578 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 10:36:51.304379 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.304368 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 10:36:51.304417 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.304388 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 10:36:51.304417 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.304398 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 10:36:51.304417 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.304405 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 10:36:51.304417 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.304413 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 10:36:51.304535 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.304430 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 10:36:51.304535 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.304461 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 10:36:51.304535 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.304469 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 10:36:51.304535 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.304476 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 10:36:51.304535 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.304482 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 10:36:51.304535 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.304494 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 10:36:51.304535 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.304513 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 10:36:51.305305 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.305294 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 10:36:51.305338 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.305307 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 10:36:51.308762 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.308740 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-159.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 10:36:51.308875 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.308769 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 10:36:51.308875 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.308782 2576 server.go:1268] "Started kubelet" Aug 04 10:36:51.308875 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.308813 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 10:36:51.308875 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.308858 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 10:36:51.309109 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.309090 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-159.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 10:36:51.309160 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.309138 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 10:36:51.309718 ip-10-0-131-159 systemd[1]: Started Kubernetes Kubelet. Aug 04 10:36:51.310444 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.310304 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 10:36:51.311945 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.311928 2576 server.go:310] "Adding debug handlers to kubelet server" Aug 04 10:36:51.313879 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.313858 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 10:36:51.318318 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.317190 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-159.ec2.internal.18c893d8bacf1dd9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-159.ec2.internal,UID:ip-10-0-131-159.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-159.ec2.internal,},FirstTimestamp:2026-08-04 10:36:51.308756441 +0000 UTC m=+0.406549826,LastTimestamp:2026-08-04 10:36:51.308756441 +0000 UTC m=+0.406549826,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-159.ec2.internal,}" Aug 04 10:36:51.319088 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.319070 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 10:36:51.319169 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.319116 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 10:36:51.319218 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.319170 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-159.ec2.internal\" not found" Aug 04 10:36:51.320058 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.319967 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 10:36:51.320058 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.320061 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 10:36:51.320188 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.320071 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 10:36:51.320352 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.320338 2576 reconstruct.go:97] "Volume reconstruction finished" Aug 04 10:36:51.320352 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.320352 2576 reconciler.go:29] "Reconciler: start to sync state" Aug 04 10:36:51.320603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.320586 2576 factory.go:55] Registering systemd factory Aug 04 10:36:51.320672 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.320634 2576 factory.go:223] Registration of the systemd container factory successfully Aug 04 10:36:51.320986 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.320970 2576 factory.go:153] Registering CRI-O factory Aug 04 10:36:51.321050 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.320989 2576 factory.go:223] Registration of the crio container factory successfully Aug 04 10:36:51.321095 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.321048 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 10:36:51.321145 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.321097 2576 factory.go:103] Registering Raw factory Aug 04 10:36:51.321145 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.321113 2576 manager.go:1196] Started watching for new ooms in manager Aug 04 10:36:51.322024 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.322002 2576 manager.go:349] Starting recovery of all containers Aug 04 10:36:51.331590 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.331543 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-159.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 10:36:51.331694 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.331585 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 10:36:51.334367 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.334348 2576 manager.go:354] Recovery completed Aug 04 10:36:51.339910 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.339895 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:36:51.341842 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.341820 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:36:51.341918 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.341861 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:36:51.341918 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.341878 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:36:51.342463 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.342448 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 10:36:51.342463 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.342460 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 10:36:51.342571 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.342465 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:36:51.342571 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.342477 2576 state_mem.go:36] "Initialized new in-memory state store" Aug 04 10:36:51.343166 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.343147 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:36:51.343229 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.343181 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:36:51.343229 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.343197 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:36:51.344659 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.344646 2576 policy_none.go:49] "None policy: Start" Aug 04 10:36:51.344739 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.344664 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 10:36:51.344739 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.344677 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 10:36:51.346783 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.346769 2576 policy_none.go:47] "Start" Aug 04 10:36:51.351450 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.351380 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-159.ec2.internal.18c893d8bcc7fd9f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-159.ec2.internal,UID:ip-10-0-131-159.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-159.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-159.ec2.internal,},FirstTimestamp:2026-08-04 10:36:51.341843871 +0000 UTC m=+0.439637267,LastTimestamp:2026-08-04 10:36:51.341843871 +0000 UTC m=+0.439637267,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-159.ec2.internal,}" Aug 04 10:36:51.354515 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.354428 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-159.ec2.internal.18c893d8bcc85ba8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-159.ec2.internal,UID:ip-10-0-131-159.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-159.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-159.ec2.internal,},FirstTimestamp:2026-08-04 10:36:51.341867944 +0000 UTC m=+0.439661330,LastTimestamp:2026-08-04 10:36:51.341867944 +0000 UTC m=+0.439661330,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-159.ec2.internal,}" Aug 04 10:36:51.359391 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.359371 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z958l" Aug 04 10:36:51.360392 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.360326 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-159.ec2.internal.18c893d8bcc89a25 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-159.ec2.internal,UID:ip-10-0-131-159.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-159.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-159.ec2.internal,},FirstTimestamp:2026-08-04 10:36:51.341883941 +0000 UTC m=+0.439677333,LastTimestamp:2026-08-04 10:36:51.341883941 +0000 UTC m=+0.439677333,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-159.ec2.internal,}" Aug 04 10:36:51.367637 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.367548 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-159.ec2.internal.18c893d8bcc7fd9f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-159.ec2.internal.18c893d8bcc7fd9f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-159.ec2.internal,UID:ip-10-0-131-159.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-159.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-159.ec2.internal,},FirstTimestamp:2026-08-04 10:36:51.341843871 +0000 UTC m=+0.439637267,LastTimestamp:2026-08-04 10:36:51.34316474 +0000 UTC m=+0.440958127,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-159.ec2.internal,}" Aug 04 10:36:51.376005 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.375853 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z958l" Aug 04 10:36:51.378306 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.378225 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-159.ec2.internal.18c893d8bcc85ba8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-159.ec2.internal.18c893d8bcc85ba8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-159.ec2.internal,UID:ip-10-0-131-159.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-159.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-159.ec2.internal,},FirstTimestamp:2026-08-04 10:36:51.341867944 +0000 UTC m=+0.439661330,LastTimestamp:2026-08-04 10:36:51.343188667 +0000 UTC m=+0.440982059,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-159.ec2.internal,}" Aug 04 10:36:51.382517 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.382436 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-159.ec2.internal.18c893d8bcc89a25\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-159.ec2.internal.18c893d8bcc89a25 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-159.ec2.internal,UID:ip-10-0-131-159.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-159.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-159.ec2.internal,},FirstTimestamp:2026-08-04 10:36:51.341883941 +0000 UTC m=+0.439677333,LastTimestamp:2026-08-04 10:36:51.34320246 +0000 UTC m=+0.440995846,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-159.ec2.internal,}" Aug 04 10:36:51.406284 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.387143 2576 manager.go:337] "Starting Device Plugin manager" Aug 04 10:36:51.406284 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.387174 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 10:36:51.406284 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.387187 2576 server.go:87] "Starting device plugin registration server" Aug 04 10:36:51.406284 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.387479 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 10:36:51.406284 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.387490 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 10:36:51.406284 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.387619 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 10:36:51.406284 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.387763 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 10:36:51.406284 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.387773 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 10:36:51.406284 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.388258 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 10:36:51.406284 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.388314 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-159.ec2.internal\" not found" Aug 04 10:36:51.429630 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.429595 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 10:36:51.430880 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.430853 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 10:36:51.430995 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.430889 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 10:36:51.430995 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.430913 2576 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 10:36:51.430995 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.430949 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 10:36:51.433744 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.433726 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:36:51.487753 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.487671 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:36:51.488687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.488663 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:36:51.488800 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.488697 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:36:51.488800 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.488708 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:36:51.488800 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.488733 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.500658 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.500639 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.500717 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.500666 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-159.ec2.internal\": node \"ip-10-0-131-159.ec2.internal\" not found" Aug 04 10:36:51.527613 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.527582 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-159.ec2.internal\" not found" Aug 04 10:36:51.532290 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.531662 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-159.ec2.internal"] Aug 04 10:36:51.532357 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.532332 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:36:51.533296 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.533275 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:36:51.533352 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.533309 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:36:51.533352 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.533320 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:36:51.535684 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.535672 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:36:51.535802 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.535785 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.535853 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.535834 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:36:51.536453 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.536435 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:36:51.536554 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.536463 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:36:51.536554 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.536473 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:36:51.536554 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.536436 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:36:51.536661 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.536592 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:36:51.536661 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.536606 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:36:51.538745 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.538726 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.538842 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.538757 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:36:51.539435 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.539420 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:36:51.539527 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.539450 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:36:51.539527 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.539463 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:36:51.563464 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.563435 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-159.ec2.internal\" not found" node="ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.567984 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.567966 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-159.ec2.internal\" not found" node="ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.622847 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.622808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6c23ed269e5911d443d635b804efbb1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal\" (UID: \"f6c23ed269e5911d443d635b804efbb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.623002 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.622867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57e7e3404a3ff3539a1dc706d1378691-config\") pod \"kube-apiserver-proxy-ip-10-0-131-159.ec2.internal\" (UID: \"57e7e3404a3ff3539a1dc706d1378691\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.623002 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.622923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6c23ed269e5911d443d635b804efbb1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal\" (UID: \"f6c23ed269e5911d443d635b804efbb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.627921 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.627902 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-159.ec2.internal\" not found" Aug 04 10:36:51.723321 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.723287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6c23ed269e5911d443d635b804efbb1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal\" (UID: \"f6c23ed269e5911d443d635b804efbb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.723321 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.723322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6c23ed269e5911d443d635b804efbb1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal\" (UID: \"f6c23ed269e5911d443d635b804efbb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.723446 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.723339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57e7e3404a3ff3539a1dc706d1378691-config\") pod \"kube-apiserver-proxy-ip-10-0-131-159.ec2.internal\" (UID: \"57e7e3404a3ff3539a1dc706d1378691\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.723446 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.723382 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57e7e3404a3ff3539a1dc706d1378691-config\") pod \"kube-apiserver-proxy-ip-10-0-131-159.ec2.internal\" (UID: \"57e7e3404a3ff3539a1dc706d1378691\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.723446 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.723394 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6c23ed269e5911d443d635b804efbb1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal\" (UID: \"f6c23ed269e5911d443d635b804efbb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.723446 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.723394 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6c23ed269e5911d443d635b804efbb1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal\" (UID: \"f6c23ed269e5911d443d635b804efbb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.728421 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.728392 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-159.ec2.internal\" not found" Aug 04 10:36:51.828749 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.828711 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-159.ec2.internal\" not found" Aug 04 10:36:51.866276 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.866232 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.870820 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:51.870794 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-159.ec2.internal" Aug 04 10:36:51.929301 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:51.929258 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-159.ec2.internal\" not found" Aug 04 10:36:52.029938 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:52.029876 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-159.ec2.internal\" not found" Aug 04 10:36:52.131109 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:52.131071 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-159.ec2.internal\" not found" Aug 04 10:36:52.231166 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:52.231133 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-159.ec2.internal\" not found" Aug 04 10:36:52.253270 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.253221 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 10:36:52.253493 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.253459 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 10:36:52.281031 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.280942 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:36:52.331967 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:52.331927 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-159.ec2.internal\" not found" Aug 04 10:36:52.382342 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.382286 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 10:31:52 +0000 UTC" deadline="2028-01-09 18:45:18.441036086 +0000 UTC" Aug 04 10:36:52.382342 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.382348 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12560h8m26.058709201s" Aug 04 10:36:52.432250 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:52.432189 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-159.ec2.internal\" not found" Aug 04 10:36:52.441324 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.441294 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:36:52.444176 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.444153 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:36:52.444333 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.444192 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:36:52.444333 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.444207 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:36:52.513071 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.513039 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:36:52.520800 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.520765 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal" Aug 04 10:36:52.536685 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.536618 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 10:36:52.537553 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.537537 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-159.ec2.internal" Aug 04 10:36:52.548222 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.548195 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 10:36:52.583264 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.583239 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:36:52.583693 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:52.583667 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e7e3404a3ff3539a1dc706d1378691.slice/crio-1ef41d802e15d64927f72886d27abfe6e58784985a883306d1f27d105e199077 WatchSource:0}: Error finding container 1ef41d802e15d64927f72886d27abfe6e58784985a883306d1f27d105e199077: Status 404 returned error can't find the container with id 1ef41d802e15d64927f72886d27abfe6e58784985a883306d1f27d105e199077 Aug 04 10:36:52.589134 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:52.589118 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 10:36:53.300703 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.300652 2576 apiserver.go:52] "Watching apiserver" Aug 04 10:36:53.308693 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.308653 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 10:36:53.310950 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.310923 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-ldmtm","openshift-network-diagnostics/network-check-target-qxldr","openshift-network-operator/iptables-alerter-l4r2b","openshift-ovn-kubernetes/ovnkube-node-hmgnh","kube-system/global-pull-secret-syncer-gr6kd","kube-system/konnectivity-agent-8qppj","kube-system/kube-apiserver-proxy-ip-10-0-131-159.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8","openshift-cluster-node-tuning-operator/tuned-psr7q","openshift-dns/node-resolver-r9kvv","openshift-multus/multus-additional-cni-plugins-f65wd","openshift-multus/multus-dk4zq","openshift-image-registry/node-ca-c9kzp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal"] Aug 04 10:36:53.313746 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.313725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r9kvv" Aug 04 10:36:53.316155 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.316129 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c9kzp" Aug 04 10:36:53.318635 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.318554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l4r2b" Aug 04 10:36:53.319901 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.319880 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 10:36:53.319901 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.319892 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 10:36:53.320072 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.319938 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 10:36:53.320072 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.320022 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qp6sn\"" Aug 04 10:36:53.321215 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.321194 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.323405 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.323383 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:36:53.324075 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.324057 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c7z65\"" Aug 04 10:36:53.324328 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.324298 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 10:36:53.324783 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.324760 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 10:36:53.324904 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.324881 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 10:36:53.325019 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.325001 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 10:36:53.325151 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.325135 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 10:36:53.325246 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.325229 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 10:36:53.325392 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.325378 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 10:36:53.325631 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.325605 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cpp52\"" Aug 04 10:36:53.326642 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.326098 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:53.326642 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.326177 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 10:36:53.326642 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.326206 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z2nlr\"" Aug 04 10:36:53.326642 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.326291 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 10:36:53.326642 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:53.326203 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:36:53.326989 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.326974 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 10:36:53.327071 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.327053 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 10:36:53.327594 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.327399 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8qppj" Aug 04 10:36:53.330739 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.330720 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.333123 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.333102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.336143 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.336121 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:36:53.336240 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:53.336196 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:36:53.336931 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.336910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b19bc6d-13f7-4db1-bc59-34eba05608ff-hosts-file\") pod \"node-resolver-r9kvv\" (UID: \"7b19bc6d-13f7-4db1-bc59-34eba05608ff\") " pod="openshift-dns/node-resolver-r9kvv" Aug 04 10:36:53.337022 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.336935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa44517a-0e03-4dad-b3b0-79ee2001d9cd-host\") pod \"node-ca-c9kzp\" (UID: \"fa44517a-0e03-4dad-b3b0-79ee2001d9cd\") " pod="openshift-image-registry/node-ca-c9kzp" Aug 04 10:36:53.337022 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.336956 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t4f6\" (UniqueName: \"kubernetes.io/projected/fa44517a-0e03-4dad-b3b0-79ee2001d9cd-kube-api-access-6t4f6\") pod \"node-ca-c9kzp\" (UID: \"fa44517a-0e03-4dad-b3b0-79ee2001d9cd\") " pod="openshift-image-registry/node-ca-c9kzp" Aug 04 10:36:53.337022 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.336974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f-iptables-alerter-script\") pod \"iptables-alerter-l4r2b\" (UID: \"5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f\") " pod="openshift-network-operator/iptables-alerter-l4r2b" Aug 04 10:36:53.337022 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.336994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-run-ovn\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337228 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337228 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-ovnkube-config\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337228 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-env-overrides\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337228 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-ovnkube-script-lib\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337228 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-systemd-units\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337228 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-kubelet\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337228 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337215 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-run-netns\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337240 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-etc-openvswitch\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-cni-netd\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-run-systemd\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-var-lib-openvswitch\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337380 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b19bc6d-13f7-4db1-bc59-34eba05608ff-tmp-dir\") pod \"node-resolver-r9kvv\" (UID: \"7b19bc6d-13f7-4db1-bc59-34eba05608ff\") " pod="openshift-dns/node-resolver-r9kvv" Aug 04 10:36:53.337603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-slash\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337413 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-run-openvswitch\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-node-log\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-ovn-node-metrics-cert\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsgcn\" (UniqueName: \"kubernetes.io/projected/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-kube-api-access-zsgcn\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.337603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa44517a-0e03-4dad-b3b0-79ee2001d9cd-serviceca\") pod \"node-ca-c9kzp\" (UID: \"fa44517a-0e03-4dad-b3b0-79ee2001d9cd\") " pod="openshift-image-registry/node-ca-c9kzp" Aug 04 10:36:53.337603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f-host-slash\") pod \"iptables-alerter-l4r2b\" (UID: \"5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f\") " pod="openshift-network-operator/iptables-alerter-l4r2b" Aug 04 10:36:53.337603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337567 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64s6c\" (UniqueName: \"kubernetes.io/projected/5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f-kube-api-access-64s6c\") pod \"iptables-alerter-l4r2b\" (UID: \"5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f\") " pod="openshift-network-operator/iptables-alerter-l4r2b" Aug 04 10:36:53.337603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-log-socket\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.338312 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.338312 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qrbb\" (UniqueName: \"kubernetes.io/projected/7b19bc6d-13f7-4db1-bc59-34eba05608ff-kube-api-access-7qrbb\") pod \"node-resolver-r9kvv\" (UID: \"7b19bc6d-13f7-4db1-bc59-34eba05608ff\") " pod="openshift-dns/node-resolver-r9kvv" Aug 04 10:36:53.338312 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.337669 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-cni-bin\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.339122 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.339100 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:36:53.339213 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:53.339186 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:36:53.341480 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.341450 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.343425 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.343402 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 10:36:53.343575 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.343410 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 10:36:53.343575 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.343466 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nqdv6\"" Aug 04 10:36:53.343802 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.343781 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 10:36:53.343917 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.343898 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 10:36:53.343990 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.343973 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 10:36:53.344230 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.344199 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.344303 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.344232 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4frx8\"" Aug 04 10:36:53.344303 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.344276 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:36:53.344409 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.344319 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 10:36:53.344465 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.344448 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-49rzq\"" Aug 04 10:36:53.349371 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.349349 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 10:36:53.349479 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.349428 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 10:36:53.355465 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.355440 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wxd4t\"" Aug 04 10:36:53.355465 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.355454 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lkxqm\"" Aug 04 10:36:53.355900 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.355870 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 10:36:53.356003 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.355872 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 10:36:53.356003 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.355949 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 10:36:53.356003 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.355872 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 10:36:53.361662 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.361640 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 10:36:53.368630 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.368602 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8gm6h" Aug 04 10:36:53.385493 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.385460 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8gm6h" Aug 04 10:36:53.422028 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.421996 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 10:36:53.437337 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.437276 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-159.ec2.internal" event={"ID":"57e7e3404a3ff3539a1dc706d1378691","Type":"ContainerStarted","Data":"1ef41d802e15d64927f72886d27abfe6e58784985a883306d1f27d105e199077"} Aug 04 10:36:53.438555 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.438527 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f-iptables-alerter-script\") pod \"iptables-alerter-l4r2b\" (UID: \"5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f\") " pod="openshift-network-operator/iptables-alerter-l4r2b" Aug 04 10:36:53.438684 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.438568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.438684 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.438602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-etc-selinux\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.438782 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.438714 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-var-lib-cni-bin\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.438782 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.438736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-run-ovn\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.438782 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.438764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.438912 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.438823 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.438912 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.438848 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-run-ovn\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.438912 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.438891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-env-overrides\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.439047 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.438921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-device-dir\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.439047 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.438947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:36:53.439047 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.438973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f9830de-6c97-47d6-99d0-ed151a33791b-tmp\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.439047 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-cnibin\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.439252 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439185 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4d7923f6-df0e-4f83-b6da-34d93041a405-konnectivity-ca\") pod \"konnectivity-agent-8qppj\" (UID: \"4d7923f6-df0e-4f83-b6da-34d93041a405\") " pod="kube-system/konnectivity-agent-8qppj" Aug 04 10:36:53.439252 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439199 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f-iptables-alerter-script\") pod \"iptables-alerter-l4r2b\" (UID: \"5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f\") " pod="openshift-network-operator/iptables-alerter-l4r2b" Aug 04 10:36:53.439252 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzmj\" (UniqueName: \"kubernetes.io/projected/3b065ca8-d420-4ffa-9da9-ab3e60c06322-kube-api-access-nlzmj\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.439409 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-sys\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.439409 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439318 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-kubelet\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.439409 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-kubelet\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.439409 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-etc-openvswitch\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.439666 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439410 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-lib-modules\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.439666 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439429 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-etc-openvswitch\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.439666 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-var-lib-kubelet\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.439666 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-run-systemd\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.439666 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d71b33c-089f-4d23-be91-d48f28a9c333-dbus\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:53.439666 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439558 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-run-systemd\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.439666 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439573 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/516d0545-a943-4d35-91f4-b8ac8b819560-multus-daemon-config\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.439666 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbl69\" (UniqueName: \"kubernetes.io/projected/516d0545-a943-4d35-91f4-b8ac8b819560-kube-api-access-tbl69\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.439666 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-slash\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.439666 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-node-log\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439665 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-slash\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-ovn-node-metrics-cert\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439727 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsgcn\" (UniqueName: \"kubernetes.io/projected/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-kube-api-access-zsgcn\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b065ca8-d420-4ffa-9da9-ab3e60c06322-system-cni-dir\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa44517a-0e03-4dad-b3b0-79ee2001d9cd-serviceca\") pod \"node-ca-c9kzp\" (UID: \"fa44517a-0e03-4dad-b3b0-79ee2001d9cd\") " pod="openshift-image-registry/node-ca-c9kzp" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439795 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-node-log\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b065ca8-d420-4ffa-9da9-ab3e60c06322-cni-binary-copy\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-run\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/516d0545-a943-4d35-91f4-b8ac8b819560-cni-binary-copy\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-cni-bin\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439925 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-socket-dir\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439933 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-cni-bin\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.439989 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-tuned\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b065ca8-d420-4ffa-9da9-ab3e60c06322-os-release\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa44517a-0e03-4dad-b3b0-79ee2001d9cd-host\") pod \"node-ca-c9kzp\" (UID: \"fa44517a-0e03-4dad-b3b0-79ee2001d9cd\") " pod="openshift-image-registry/node-ca-c9kzp" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa44517a-0e03-4dad-b3b0-79ee2001d9cd-host\") pod \"node-ca-c9kzp\" (UID: \"fa44517a-0e03-4dad-b3b0-79ee2001d9cd\") " pod="openshift-image-registry/node-ca-c9kzp" Aug 04 10:36:53.440227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440203 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4d7923f6-df0e-4f83-b6da-34d93041a405-agent-certs\") pod \"konnectivity-agent-8qppj\" (UID: \"4d7923f6-df0e-4f83-b6da-34d93041a405\") " pod="kube-system/konnectivity-agent-8qppj" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zd25\" (UniqueName: \"kubernetes.io/projected/81e633a3-39f4-477f-b771-27f727e61657-kube-api-access-8zd25\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440221 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-env-overrides\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-host\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-multus-cni-dir\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440312 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-hostroot\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d71b33c-089f-4d23-be91-d48f28a9c333-kubelet-config\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-ovnkube-config\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-ovnkube-script-lib\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440507 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa44517a-0e03-4dad-b3b0-79ee2001d9cd-serviceca\") pod \"node-ca-c9kzp\" (UID: \"fa44517a-0e03-4dad-b3b0-79ee2001d9cd\") " pod="openshift-image-registry/node-ca-c9kzp" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-run-multus-certs\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-systemd-units\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440698 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-systemd\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440718 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-systemd-units\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440759 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-os-release\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440790 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-var-lib-kubelet\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.441093 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-run-netns\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-cni-netd\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-registration-dir\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rft2q\" (UniqueName: \"kubernetes.io/projected/f67d2ed4-7812-45de-95c0-32cf86d9eef2-kube-api-access-rft2q\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440947 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-run-netns\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440970 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b065ca8-d420-4ffa-9da9-ab3e60c06322-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441015 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-ovnkube-config\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-var-lib-openvswitch\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-ovnkube-script-lib\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441063 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-var-lib-openvswitch\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.440973 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-cni-netd\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-sys-fs\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b065ca8-d420-4ffa-9da9-ab3e60c06322-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441138 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-system-cni-dir\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-run-k8s-cni-cncf-io\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441193 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-multus-conf-dir\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b19bc6d-13f7-4db1-bc59-34eba05608ff-tmp-dir\") pod \"node-resolver-r9kvv\" (UID: \"7b19bc6d-13f7-4db1-bc59-34eba05608ff\") " pod="openshift-dns/node-resolver-r9kvv" Aug 04 10:36:53.442043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441246 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-run-openvswitch\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441347 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b065ca8-d420-4ffa-9da9-ab3e60c06322-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441369 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-run-openvswitch\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-sysctl-d\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441420 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-sysctl-conf\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfzf2\" (UniqueName: \"kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2\") pod \"network-check-target-qxldr\" (UID: \"5b51f136-2d2f-4652-840c-54033e0df787\") " pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441483 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-multus-socket-dir-parent\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-run-netns\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441565 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b19bc6d-13f7-4db1-bc59-34eba05608ff-tmp-dir\") pod \"node-resolver-r9kvv\" (UID: \"7b19bc6d-13f7-4db1-bc59-34eba05608ff\") " pod="openshift-dns/node-resolver-r9kvv" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f-host-slash\") pod \"iptables-alerter-l4r2b\" (UID: \"5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f\") " pod="openshift-network-operator/iptables-alerter-l4r2b" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64s6c\" (UniqueName: \"kubernetes.io/projected/5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f-kube-api-access-64s6c\") pod \"iptables-alerter-l4r2b\" (UID: \"5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f\") " pod="openshift-network-operator/iptables-alerter-l4r2b" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-log-socket\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441655 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f-host-slash\") pod \"iptables-alerter-l4r2b\" (UID: \"5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f\") " pod="openshift-network-operator/iptables-alerter-l4r2b" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441728 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-log-socket\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-var-lib-cni-multus\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.442687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441795 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-etc-kubernetes\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.443302 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.443302 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qrbb\" (UniqueName: \"kubernetes.io/projected/7b19bc6d-13f7-4db1-bc59-34eba05608ff-kube-api-access-7qrbb\") pod \"node-resolver-r9kvv\" (UID: \"7b19bc6d-13f7-4db1-bc59-34eba05608ff\") " pod="openshift-dns/node-resolver-r9kvv" Aug 04 10:36:53.443302 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.441891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-modprobe-d\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.443302 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.442018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-sysconfig\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.443302 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.442062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-kubernetes\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.443302 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.442095 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pphl\" (UniqueName: \"kubernetes.io/projected/7f9830de-6c97-47d6-99d0-ed151a33791b-kube-api-access-9pphl\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.443302 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.442122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b065ca8-d420-4ffa-9da9-ab3e60c06322-cnibin\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.443302 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.442152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b19bc6d-13f7-4db1-bc59-34eba05608ff-hosts-file\") pod \"node-resolver-r9kvv\" (UID: \"7b19bc6d-13f7-4db1-bc59-34eba05608ff\") " pod="openshift-dns/node-resolver-r9kvv" Aug 04 10:36:53.443302 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.442178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t4f6\" (UniqueName: \"kubernetes.io/projected/fa44517a-0e03-4dad-b3b0-79ee2001d9cd-kube-api-access-6t4f6\") pod \"node-ca-c9kzp\" (UID: \"fa44517a-0e03-4dad-b3b0-79ee2001d9cd\") " pod="openshift-image-registry/node-ca-c9kzp" Aug 04 10:36:53.443302 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.442241 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b19bc6d-13f7-4db1-bc59-34eba05608ff-hosts-file\") pod \"node-resolver-r9kvv\" (UID: \"7b19bc6d-13f7-4db1-bc59-34eba05608ff\") " pod="openshift-dns/node-resolver-r9kvv" Aug 04 10:36:53.445248 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.445222 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-ovn-node-metrics-cert\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.454798 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.454765 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsgcn\" (UniqueName: \"kubernetes.io/projected/068755a5-9b7a-4e8d-964d-5fe08b65aeb8-kube-api-access-zsgcn\") pod \"ovnkube-node-hmgnh\" (UID: \"068755a5-9b7a-4e8d-964d-5fe08b65aeb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.473723 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.473655 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qrbb\" (UniqueName: \"kubernetes.io/projected/7b19bc6d-13f7-4db1-bc59-34eba05608ff-kube-api-access-7qrbb\") pod \"node-resolver-r9kvv\" (UID: \"7b19bc6d-13f7-4db1-bc59-34eba05608ff\") " pod="openshift-dns/node-resolver-r9kvv" Aug 04 10:36:53.474310 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.474280 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64s6c\" (UniqueName: \"kubernetes.io/projected/5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f-kube-api-access-64s6c\") pod \"iptables-alerter-l4r2b\" (UID: \"5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f\") " pod="openshift-network-operator/iptables-alerter-l4r2b" Aug 04 10:36:53.476950 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.476926 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t4f6\" (UniqueName: \"kubernetes.io/projected/fa44517a-0e03-4dad-b3b0-79ee2001d9cd-kube-api-access-6t4f6\") pod \"node-ca-c9kzp\" (UID: \"fa44517a-0e03-4dad-b3b0-79ee2001d9cd\") " pod="openshift-image-registry/node-ca-c9kzp" Aug 04 10:36:53.542477 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542440 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-os-release\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.542477 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-var-lib-kubelet\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.542740 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-registration-dir\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.542740 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rft2q\" (UniqueName: \"kubernetes.io/projected/f67d2ed4-7812-45de-95c0-32cf86d9eef2-kube-api-access-rft2q\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:36:53.542740 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542611 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-os-release\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.542740 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b065ca8-d420-4ffa-9da9-ab3e60c06322-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.542740 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-sys-fs\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.542740 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542612 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-var-lib-kubelet\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.542740 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b065ca8-d420-4ffa-9da9-ab3e60c06322-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-system-cni-dir\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542753 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-sys-fs\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542612 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-registration-dir\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542806 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-system-cni-dir\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-run-k8s-cni-cncf-io\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542862 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-multus-conf-dir\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542895 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b065ca8-d420-4ffa-9da9-ab3e60c06322-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-run-k8s-cni-cncf-io\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-sysctl-d\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542930 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-multus-conf-dir\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-sysctl-conf\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.542979 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzf2\" (UniqueName: \"kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2\") pod \"network-check-target-qxldr\" (UID: \"5b51f136-2d2f-4652-840c-54033e0df787\") " pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-multus-socket-dir-parent\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-run-netns\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543044 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b065ca8-d420-4ffa-9da9-ab3e60c06322-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-var-lib-cni-multus\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.543062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-sysctl-d\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-etc-kubernetes\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-modprobe-d\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543111 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-sysctl-conf\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543116 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-run-netns\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-sysconfig\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543152 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-var-lib-cni-multus\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-kubernetes\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543161 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-etc-kubernetes\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543174 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-multus-socket-dir-parent\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543207 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-sysconfig\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543217 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-kubernetes\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543228 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-modprobe-d\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pphl\" (UniqueName: \"kubernetes.io/projected/7f9830de-6c97-47d6-99d0-ed151a33791b-kube-api-access-9pphl\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b065ca8-d420-4ffa-9da9-ab3e60c06322-cnibin\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-etc-selinux\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543347 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b065ca8-d420-4ffa-9da9-ab3e60c06322-cnibin\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.543952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-var-lib-cni-bin\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-device-dir\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543409 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543416 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b065ca8-d420-4ffa-9da9-ab3e60c06322-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543425 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f9830de-6c97-47d6-99d0-ed151a33791b-tmp\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b065ca8-d420-4ffa-9da9-ab3e60c06322-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-etc-selinux\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543475 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-device-dir\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543451 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-var-lib-cni-bin\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-cnibin\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:53.543503 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4d7923f6-df0e-4f83-b6da-34d93041a405-konnectivity-ca\") pod \"konnectivity-agent-8qppj\" (UID: \"4d7923f6-df0e-4f83-b6da-34d93041a405\") " pod="kube-system/konnectivity-agent-8qppj" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543527 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-cnibin\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzmj\" (UniqueName: \"kubernetes.io/projected/3b065ca8-d420-4ffa-9da9-ab3e60c06322-kube-api-access-nlzmj\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:53.543601 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs podName:f67d2ed4-7812-45de-95c0-32cf86d9eef2 nodeName:}" failed. No retries permitted until 2026-08-04 10:36:54.043560798 +0000 UTC m=+3.141354181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs") pod "network-metrics-daemon-ldmtm" (UID: "f67d2ed4-7812-45de-95c0-32cf86d9eef2") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-sys\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.544871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543664 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-lib-modules\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-var-lib-kubelet\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d71b33c-089f-4d23-be91-d48f28a9c333-dbus\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/516d0545-a943-4d35-91f4-b8ac8b819560-multus-daemon-config\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543751 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-sys\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbl69\" (UniqueName: \"kubernetes.io/projected/516d0545-a943-4d35-91f4-b8ac8b819560-kube-api-access-tbl69\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b065ca8-d420-4ffa-9da9-ab3e60c06322-system-cni-dir\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543816 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-lib-modules\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b065ca8-d420-4ffa-9da9-ab3e60c06322-cni-binary-copy\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-run\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/516d0545-a943-4d35-91f4-b8ac8b819560-cni-binary-copy\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543898 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d71b33c-089f-4d23-be91-d48f28a9c333-dbus\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543906 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-socket-dir\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-tuned\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543946 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-var-lib-kubelet\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b065ca8-d420-4ffa-9da9-ab3e60c06322-os-release\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.543995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4d7923f6-df0e-4f83-b6da-34d93041a405-agent-certs\") pod \"konnectivity-agent-8qppj\" (UID: \"4d7923f6-df0e-4f83-b6da-34d93041a405\") " pod="kube-system/konnectivity-agent-8qppj" Aug 04 10:36:53.545761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544001 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-run\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zd25\" (UniqueName: \"kubernetes.io/projected/81e633a3-39f4-477f-b771-27f727e61657-kube-api-access-8zd25\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-host\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4d7923f6-df0e-4f83-b6da-34d93041a405-konnectivity-ca\") pod \"konnectivity-agent-8qppj\" (UID: \"4d7923f6-df0e-4f83-b6da-34d93041a405\") " pod="kube-system/konnectivity-agent-8qppj" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-multus-cni-dir\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-hostroot\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d71b33c-089f-4d23-be91-d48f28a9c333-kubelet-config\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-run-multus-certs\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-systemd\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-systemd\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b065ca8-d420-4ffa-9da9-ab3e60c06322-os-release\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544414 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/516d0545-a943-4d35-91f4-b8ac8b819560-multus-daemon-config\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544458 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b065ca8-d420-4ffa-9da9-ab3e60c06322-cni-binary-copy\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544525 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d71b33c-089f-4d23-be91-d48f28a9c333-kubelet-config\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544546 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/81e633a3-39f4-477f-b771-27f727e61657-socket-dir\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544576 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-hostroot\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544611 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-host-run-multus-certs\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.546536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544648 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/516d0545-a943-4d35-91f4-b8ac8b819560-multus-cni-dir\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.547248 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544658 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f9830de-6c97-47d6-99d0-ed151a33791b-host\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.547248 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:53.544696 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:36:53.547248 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.544702 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b065ca8-d420-4ffa-9da9-ab3e60c06322-system-cni-dir\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.547248 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:53.544763 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret podName:0d71b33c-089f-4d23-be91-d48f28a9c333 nodeName:}" failed. No retries permitted until 2026-08-04 10:36:54.044744542 +0000 UTC m=+3.142537930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret") pod "global-pull-secret-syncer-gr6kd" (UID: "0d71b33c-089f-4d23-be91-d48f28a9c333") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:36:53.547248 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.545171 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/516d0545-a943-4d35-91f4-b8ac8b819560-cni-binary-copy\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.547248 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.546379 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f9830de-6c97-47d6-99d0-ed151a33791b-tmp\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.547248 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.546441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f9830de-6c97-47d6-99d0-ed151a33791b-etc-tuned\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.547248 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.546943 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4d7923f6-df0e-4f83-b6da-34d93041a405-agent-certs\") pod \"konnectivity-agent-8qppj\" (UID: \"4d7923f6-df0e-4f83-b6da-34d93041a405\") " pod="kube-system/konnectivity-agent-8qppj" Aug 04 10:36:53.566413 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.566323 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzmj\" (UniqueName: \"kubernetes.io/projected/3b065ca8-d420-4ffa-9da9-ab3e60c06322-kube-api-access-nlzmj\") pod \"multus-additional-cni-plugins-f65wd\" (UID: \"3b065ca8-d420-4ffa-9da9-ab3e60c06322\") " pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.567810 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:53.567786 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:36:53.567810 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:53.567809 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:36:53.567993 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:53.567821 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cfzf2 for pod openshift-network-diagnostics/network-check-target-qxldr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:36:53.567993 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:53.567887 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2 podName:5b51f136-2d2f-4652-840c-54033e0df787 nodeName:}" failed. No retries permitted until 2026-08-04 10:36:54.067871 +0000 UTC m=+3.165664379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cfzf2" (UniqueName: "kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2") pod "network-check-target-qxldr" (UID: "5b51f136-2d2f-4652-840c-54033e0df787") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:36:53.568881 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.568857 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rft2q\" (UniqueName: \"kubernetes.io/projected/f67d2ed4-7812-45de-95c0-32cf86d9eef2-kube-api-access-rft2q\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:36:53.572012 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.571985 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zd25\" (UniqueName: \"kubernetes.io/projected/81e633a3-39f4-477f-b771-27f727e61657-kube-api-access-8zd25\") pod \"aws-ebs-csi-driver-node-6b6n8\" (UID: \"81e633a3-39f4-477f-b771-27f727e61657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.572396 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.572377 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pphl\" (UniqueName: \"kubernetes.io/projected/7f9830de-6c97-47d6-99d0-ed151a33791b-kube-api-access-9pphl\") pod \"tuned-psr7q\" (UID: \"7f9830de-6c97-47d6-99d0-ed151a33791b\") " pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.577414 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.577387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbl69\" (UniqueName: \"kubernetes.io/projected/516d0545-a943-4d35-91f4-b8ac8b819560-kube-api-access-tbl69\") pod \"multus-dk4zq\" (UID: \"516d0545-a943-4d35-91f4-b8ac8b819560\") " pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.627821 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.627776 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r9kvv" Aug 04 10:36:53.635200 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.635170 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l4r2b" Aug 04 10:36:53.642308 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.642281 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c9kzp" Aug 04 10:36:53.648517 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.648492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:36:53.655351 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.655316 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8qppj" Aug 04 10:36:53.660041 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.660018 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-psr7q" Aug 04 10:36:53.668202 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.668173 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" Aug 04 10:36:53.675372 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.675348 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f65wd" Aug 04 10:36:53.679480 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.679455 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dk4zq" Aug 04 10:36:53.752033 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:53.751996 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:36:54.047769 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.047687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:54.047912 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.047775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:36:54.047912 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:54.047853 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:36:54.047912 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:54.047870 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:36:54.048017 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:54.047927 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs podName:f67d2ed4-7812-45de-95c0-32cf86d9eef2 nodeName:}" failed. No retries permitted until 2026-08-04 10:36:55.047911448 +0000 UTC m=+4.145704827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs") pod "network-metrics-daemon-ldmtm" (UID: "f67d2ed4-7812-45de-95c0-32cf86d9eef2") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:36:54.048017 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:54.047942 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret podName:0d71b33c-089f-4d23-be91-d48f28a9c333 nodeName:}" failed. No retries permitted until 2026-08-04 10:36:55.047935528 +0000 UTC m=+4.145728902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret") pod "global-pull-secret-syncer-gr6kd" (UID: "0d71b33c-089f-4d23-be91-d48f28a9c333") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:36:54.079470 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:54.079433 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e633a3_39f4_477f_b771_27f727e61657.slice/crio-4c1e869ff715f504ae49d476cc08c0b2bc52baf6afce1bda956b95fd34c1d942 WatchSource:0}: Error finding container 4c1e869ff715f504ae49d476cc08c0b2bc52baf6afce1bda956b95fd34c1d942: Status 404 returned error can't find the container with id 4c1e869ff715f504ae49d476cc08c0b2bc52baf6afce1bda956b95fd34c1d942 Aug 04 10:36:54.080768 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:54.080743 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f50bae9_b79c_4a6a_96ec_1d6edecf5f3f.slice/crio-5acecaac97dba6078c080ba4fc36c87db90a76272a8b3306cea567bd9548680e WatchSource:0}: Error finding container 5acecaac97dba6078c080ba4fc36c87db90a76272a8b3306cea567bd9548680e: Status 404 returned error can't find the container with id 5acecaac97dba6078c080ba4fc36c87db90a76272a8b3306cea567bd9548680e Aug 04 10:36:54.081588 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:54.081452 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7923f6_df0e_4f83_b6da_34d93041a405.slice/crio-7db6b01601860df0f3d95062f3fb493ae2d658a51bdc3d0adf12f2fd0dc50e9c WatchSource:0}: Error finding container 7db6b01601860df0f3d95062f3fb493ae2d658a51bdc3d0adf12f2fd0dc50e9c: Status 404 returned error can't find the container with id 7db6b01601860df0f3d95062f3fb493ae2d658a51bdc3d0adf12f2fd0dc50e9c Aug 04 10:36:54.084437 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:54.084252 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b065ca8_d420_4ffa_9da9_ab3e60c06322.slice/crio-a9fa232ae8ddc7dd27312837eca40836598b8a2c1ec2a6ee90cb916676464448 WatchSource:0}: Error finding container a9fa232ae8ddc7dd27312837eca40836598b8a2c1ec2a6ee90cb916676464448: Status 404 returned error can't find the container with id a9fa232ae8ddc7dd27312837eca40836598b8a2c1ec2a6ee90cb916676464448 Aug 04 10:36:54.085875 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:54.085853 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa44517a_0e03_4dad_b3b0_79ee2001d9cd.slice/crio-608649729a0d6b844a017603bc504bf8a8fb02305b89c5e3ac00a9b8123d49c1 WatchSource:0}: Error finding container 608649729a0d6b844a017603bc504bf8a8fb02305b89c5e3ac00a9b8123d49c1: Status 404 returned error can't find the container with id 608649729a0d6b844a017603bc504bf8a8fb02305b89c5e3ac00a9b8123d49c1 Aug 04 10:36:54.086756 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:54.086733 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516d0545_a943_4d35_91f4_b8ac8b819560.slice/crio-e67a4620a6fe15add65918e82553d98b9a6b918e13e971562623aeb0aa06345f WatchSource:0}: Error finding container e67a4620a6fe15add65918e82553d98b9a6b918e13e971562623aeb0aa06345f: Status 404 returned error can't find the container with id e67a4620a6fe15add65918e82553d98b9a6b918e13e971562623aeb0aa06345f Aug 04 10:36:54.134966 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:36:54.134936 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c23ed269e5911d443d635b804efbb1.slice/crio-2be6fa285b94a153c22af7c9f8452c460a69577d59edeb042820853e28b59159 WatchSource:0}: Error finding container 2be6fa285b94a153c22af7c9f8452c460a69577d59edeb042820853e28b59159: Status 404 returned error can't find the container with id 2be6fa285b94a153c22af7c9f8452c460a69577d59edeb042820853e28b59159 Aug 04 10:36:54.148959 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.148936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzf2\" (UniqueName: \"kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2\") pod \"network-check-target-qxldr\" (UID: \"5b51f136-2d2f-4652-840c-54033e0df787\") " pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:36:54.149073 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:54.149060 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:36:54.149114 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:54.149077 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:36:54.149114 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:54.149087 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cfzf2 for pod openshift-network-diagnostics/network-check-target-qxldr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:36:54.149182 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:54.149137 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2 podName:5b51f136-2d2f-4652-840c-54033e0df787 nodeName:}" failed. No retries permitted until 2026-08-04 10:36:55.14912199 +0000 UTC m=+4.246915363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfzf2" (UniqueName: "kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2") pod "network-check-target-qxldr" (UID: "5b51f136-2d2f-4652-840c-54033e0df787") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:36:54.388951 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.388829 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 10:31:53 +0000 UTC" deadline="2028-04-20 23:19:36.029346128 +0000 UTC" Aug 04 10:36:54.388951 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.388888 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15012h42m41.640461337s" Aug 04 10:36:54.443846 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.443687 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-159.ec2.internal" event={"ID":"57e7e3404a3ff3539a1dc706d1378691","Type":"ContainerStarted","Data":"7b865e400e4c0d44aaaf135c2ef9c40328eae2e3b5b2d85692670dc953856685"} Aug 04 10:36:54.445269 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.445225 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-psr7q" event={"ID":"7f9830de-6c97-47d6-99d0-ed151a33791b","Type":"ContainerStarted","Data":"c21f3e7fdc3011d7f37bb724d1abf5a7d3843ceb441a4c2789878d4a100e475b"} Aug 04 10:36:54.446798 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.446768 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r9kvv" event={"ID":"7b19bc6d-13f7-4db1-bc59-34eba05608ff","Type":"ContainerStarted","Data":"5aa891236b0739fecc0c6c63b17398ca8aa3e8f8e8fdd83deb4df756f500d694"} Aug 04 10:36:54.448206 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.448176 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" event={"ID":"068755a5-9b7a-4e8d-964d-5fe08b65aeb8","Type":"ContainerStarted","Data":"f8277370dd1cbfc02c26a914dba82e484931dea9cfaaff4aff588e43da07cf06"} Aug 04 10:36:54.449573 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.449544 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dk4zq" event={"ID":"516d0545-a943-4d35-91f4-b8ac8b819560","Type":"ContainerStarted","Data":"e67a4620a6fe15add65918e82553d98b9a6b918e13e971562623aeb0aa06345f"} Aug 04 10:36:54.450944 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.450914 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c9kzp" event={"ID":"fa44517a-0e03-4dad-b3b0-79ee2001d9cd","Type":"ContainerStarted","Data":"608649729a0d6b844a017603bc504bf8a8fb02305b89c5e3ac00a9b8123d49c1"} Aug 04 10:36:54.452260 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.452232 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f65wd" event={"ID":"3b065ca8-d420-4ffa-9da9-ab3e60c06322","Type":"ContainerStarted","Data":"a9fa232ae8ddc7dd27312837eca40836598b8a2c1ec2a6ee90cb916676464448"} Aug 04 10:36:54.453812 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.453780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8qppj" event={"ID":"4d7923f6-df0e-4f83-b6da-34d93041a405","Type":"ContainerStarted","Data":"7db6b01601860df0f3d95062f3fb493ae2d658a51bdc3d0adf12f2fd0dc50e9c"} Aug 04 10:36:54.455166 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.455129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal" event={"ID":"f6c23ed269e5911d443d635b804efbb1","Type":"ContainerStarted","Data":"2be6fa285b94a153c22af7c9f8452c460a69577d59edeb042820853e28b59159"} Aug 04 10:36:54.457357 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.457192 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l4r2b" event={"ID":"5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f","Type":"ContainerStarted","Data":"5acecaac97dba6078c080ba4fc36c87db90a76272a8b3306cea567bd9548680e"} Aug 04 10:36:54.458527 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:54.458488 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" event={"ID":"81e633a3-39f4-477f-b771-27f727e61657","Type":"ContainerStarted","Data":"4c1e869ff715f504ae49d476cc08c0b2bc52baf6afce1bda956b95fd34c1d942"} Aug 04 10:36:55.059021 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:55.058971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:36:55.059235 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:55.059047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:55.059235 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:55.059192 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:36:55.059356 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:55.059260 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret podName:0d71b33c-089f-4d23-be91-d48f28a9c333 nodeName:}" failed. No retries permitted until 2026-08-04 10:36:57.059240131 +0000 UTC m=+6.157033515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret") pod "global-pull-secret-syncer-gr6kd" (UID: "0d71b33c-089f-4d23-be91-d48f28a9c333") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:36:55.059763 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:55.059741 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:36:55.059860 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:55.059806 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs podName:f67d2ed4-7812-45de-95c0-32cf86d9eef2 nodeName:}" failed. No retries permitted until 2026-08-04 10:36:57.059787159 +0000 UTC m=+6.157580551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs") pod "network-metrics-daemon-ldmtm" (UID: "f67d2ed4-7812-45de-95c0-32cf86d9eef2") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:36:55.159767 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:55.159724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzf2\" (UniqueName: \"kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2\") pod \"network-check-target-qxldr\" (UID: \"5b51f136-2d2f-4652-840c-54033e0df787\") " pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:36:55.159969 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:55.159932 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:36:55.159969 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:55.159953 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:36:55.159969 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:55.159967 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cfzf2 for pod openshift-network-diagnostics/network-check-target-qxldr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:36:55.160140 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:55.160032 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2 podName:5b51f136-2d2f-4652-840c-54033e0df787 nodeName:}" failed. No retries permitted until 2026-08-04 10:36:57.160012039 +0000 UTC m=+6.257805417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfzf2" (UniqueName: "kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2") pod "network-check-target-qxldr" (UID: "5b51f136-2d2f-4652-840c-54033e0df787") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:36:55.392462 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:55.389352 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 10:31:53 +0000 UTC" deadline="2028-03-07 03:01:21.033913703 +0000 UTC" Aug 04 10:36:55.392462 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:55.389435 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13936h24m25.644483499s" Aug 04 10:36:55.433142 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:55.433102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:36:55.433320 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:55.433252 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:36:55.433593 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:55.433555 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:55.433707 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:55.433680 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:36:55.433860 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:55.433841 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:36:55.434000 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:55.433957 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:36:57.077272 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:57.076225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:36:57.077272 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:57.076541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:57.077272 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:57.076703 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:36:57.077272 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:57.076765 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret podName:0d71b33c-089f-4d23-be91-d48f28a9c333 nodeName:}" failed. No retries permitted until 2026-08-04 10:37:01.076746241 +0000 UTC m=+10.174539622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret") pod "global-pull-secret-syncer-gr6kd" (UID: "0d71b33c-089f-4d23-be91-d48f28a9c333") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:36:57.077272 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:57.077171 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:36:57.077272 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:57.077223 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs podName:f67d2ed4-7812-45de-95c0-32cf86d9eef2 nodeName:}" failed. No retries permitted until 2026-08-04 10:37:01.0772055 +0000 UTC m=+10.174998878 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs") pod "network-metrics-daemon-ldmtm" (UID: "f67d2ed4-7812-45de-95c0-32cf86d9eef2") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:36:57.177104 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:57.176958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzf2\" (UniqueName: \"kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2\") pod \"network-check-target-qxldr\" (UID: \"5b51f136-2d2f-4652-840c-54033e0df787\") " pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:36:57.177314 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:57.177129 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:36:57.177314 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:57.177163 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:36:57.177314 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:57.177178 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cfzf2 for pod openshift-network-diagnostics/network-check-target-qxldr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:36:57.177314 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:57.177246 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2 podName:5b51f136-2d2f-4652-840c-54033e0df787 nodeName:}" failed. No retries permitted until 2026-08-04 10:37:01.177224572 +0000 UTC m=+10.275017949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfzf2" (UniqueName: "kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2") pod "network-check-target-qxldr" (UID: "5b51f136-2d2f-4652-840c-54033e0df787") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:36:57.432219 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:57.432182 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:36:57.432421 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:57.432356 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:36:57.432547 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:57.432509 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:36:57.432739 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:57.432710 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:57.432807 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:57.432792 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:36:57.432864 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:57.432819 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:36:59.431692 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:59.431647 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:36:59.432208 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:59.431866 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:36:59.432525 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:59.432490 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:36:59.432653 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:59.432628 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:36:59.432797 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:36:59.432780 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:36:59.432904 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:36:59.432884 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:37:01.116967 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:01.116874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:01.117414 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:01.117050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:01.117414 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:01.117075 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:37:01.117414 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:01.117151 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs podName:f67d2ed4-7812-45de-95c0-32cf86d9eef2 nodeName:}" failed. No retries permitted until 2026-08-04 10:37:09.117130223 +0000 UTC m=+18.214923601 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs") pod "network-metrics-daemon-ldmtm" (UID: "f67d2ed4-7812-45de-95c0-32cf86d9eef2") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:37:01.117414 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:01.117195 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:37:01.117414 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:01.117260 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret podName:0d71b33c-089f-4d23-be91-d48f28a9c333 nodeName:}" failed. No retries permitted until 2026-08-04 10:37:09.117243476 +0000 UTC m=+18.215036854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret") pod "global-pull-secret-syncer-gr6kd" (UID: "0d71b33c-089f-4d23-be91-d48f28a9c333") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:37:01.218112 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:01.218066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzf2\" (UniqueName: \"kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2\") pod \"network-check-target-qxldr\" (UID: \"5b51f136-2d2f-4652-840c-54033e0df787\") " pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:01.218309 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:01.218257 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:37:01.218309 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:01.218273 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:37:01.218309 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:01.218282 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cfzf2 for pod openshift-network-diagnostics/network-check-target-qxldr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:37:01.218477 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:01.218332 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2 podName:5b51f136-2d2f-4652-840c-54033e0df787 nodeName:}" failed. No retries permitted until 2026-08-04 10:37:09.218314302 +0000 UTC m=+18.316107675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfzf2" (UniqueName: "kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2") pod "network-check-target-qxldr" (UID: "5b51f136-2d2f-4652-840c-54033e0df787") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:37:01.434350 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:01.434265 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:01.434526 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:01.434376 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:37:01.434764 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:01.434744 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:01.434857 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:01.434842 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:37:01.434916 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:01.434890 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:01.434968 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:01.434935 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:37:03.435550 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:03.435518 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:03.435894 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:03.435638 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:37:03.436010 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:03.435991 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:03.436087 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:03.436069 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:37:03.436141 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:03.436116 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:03.436187 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:03.436173 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:37:05.431337 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:05.431297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:05.431734 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:05.431297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:05.431734 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:05.431416 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:37:05.431734 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:05.431306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:05.431734 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:05.431481 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:37:05.431734 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:05.431577 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:37:07.435072 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:07.435038 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:07.435532 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:07.435076 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:07.435532 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:07.435076 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:07.435532 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:07.435177 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:37:07.435532 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:07.435300 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:37:07.435532 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:07.435370 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:37:09.175659 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:09.175612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:09.176187 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:09.175709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:09.176187 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:09.175786 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:37:09.176187 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:09.175823 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:37:09.176187 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:09.175852 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret podName:0d71b33c-089f-4d23-be91-d48f28a9c333 nodeName:}" failed. No retries permitted until 2026-08-04 10:37:25.175834598 +0000 UTC m=+34.273627971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret") pod "global-pull-secret-syncer-gr6kd" (UID: "0d71b33c-089f-4d23-be91-d48f28a9c333") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:37:09.176187 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:09.175877 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs podName:f67d2ed4-7812-45de-95c0-32cf86d9eef2 nodeName:}" failed. No retries permitted until 2026-08-04 10:37:25.175858715 +0000 UTC m=+34.273652107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs") pod "network-metrics-daemon-ldmtm" (UID: "f67d2ed4-7812-45de-95c0-32cf86d9eef2") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:37:09.276034 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:09.275995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzf2\" (UniqueName: \"kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2\") pod \"network-check-target-qxldr\" (UID: \"5b51f136-2d2f-4652-840c-54033e0df787\") " pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:09.276188 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:09.276173 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:37:09.276251 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:09.276193 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:37:09.276251 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:09.276202 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cfzf2 for pod openshift-network-diagnostics/network-check-target-qxldr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:37:09.276312 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:09.276256 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2 podName:5b51f136-2d2f-4652-840c-54033e0df787 nodeName:}" failed. No retries permitted until 2026-08-04 10:37:25.276240206 +0000 UTC m=+34.374033622 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfzf2" (UniqueName: "kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2") pod "network-check-target-qxldr" (UID: "5b51f136-2d2f-4652-840c-54033e0df787") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:37:09.431606 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:09.431492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:09.431606 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:09.431543 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:09.431828 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:09.431491 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:09.431828 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:09.431635 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:37:09.431828 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:09.431697 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:37:09.431828 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:09.431762 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:37:11.432215 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:11.432173 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:11.432700 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:11.432285 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:37:11.432700 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:11.432302 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:11.432700 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:11.432457 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:37:11.432700 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:11.432527 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:11.432700 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:11.432605 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:37:13.435156 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:13.434490 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:13.435156 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:13.434639 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:37:13.435156 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:13.434764 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:13.435156 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:13.434847 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:37:13.435156 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:13.434900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:13.435156 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:13.434996 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:37:13.520137 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:13.520026 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-psr7q" event={"ID":"7f9830de-6c97-47d6-99d0-ed151a33791b","Type":"ContainerStarted","Data":"4010029c0bcbdcaa7fea702ae65bcd684cc3bdf0fa2680255020bb8e088d6d8b"} Aug 04 10:37:13.538783 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:13.538567 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-159.ec2.internal" podStartSLOduration=20.538546433 podStartE2EDuration="20.538546433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:36:54.463082075 +0000 UTC m=+3.560875474" watchObservedRunningTime="2026-08-04 10:37:13.538546433 +0000 UTC m=+22.636339829" Aug 04 10:37:14.527011 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.526693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r9kvv" event={"ID":"7b19bc6d-13f7-4db1-bc59-34eba05608ff","Type":"ContainerStarted","Data":"ad097833d562acdcedd8631bcbe40f743d076d844518b87c7acd502ff309a74e"} Aug 04 10:37:14.530186 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.530151 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" event={"ID":"068755a5-9b7a-4e8d-964d-5fe08b65aeb8","Type":"ContainerStarted","Data":"6f33332790467fbb3142f64c48cc469525acea613f3cf31ff08cbe2b62ba964a"} Aug 04 10:37:14.530327 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.530231 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" event={"ID":"068755a5-9b7a-4e8d-964d-5fe08b65aeb8","Type":"ContainerStarted","Data":"5cce6d3ec4a9a7f10b0095d217de88912726d611e5f685551af1f5120ad37d22"} Aug 04 10:37:14.530327 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.530264 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" event={"ID":"068755a5-9b7a-4e8d-964d-5fe08b65aeb8","Type":"ContainerStarted","Data":"7245033ff24099e03aff8b49215178ac85a778537167db3135433f0313c29844"} Aug 04 10:37:14.530327 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.530309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" event={"ID":"068755a5-9b7a-4e8d-964d-5fe08b65aeb8","Type":"ContainerStarted","Data":"80e6f92f57d12638dcac5738e20522760da246f431427596fdb78ad93d160148"} Aug 04 10:37:14.530470 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.530342 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" event={"ID":"068755a5-9b7a-4e8d-964d-5fe08b65aeb8","Type":"ContainerStarted","Data":"e48bbb830d4128376db0985f62d1fc465e71d7b304407a7c0af63754ec07aa3a"} Aug 04 10:37:14.530470 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.530381 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" event={"ID":"068755a5-9b7a-4e8d-964d-5fe08b65aeb8","Type":"ContainerStarted","Data":"b145bd11babd5bbb3d770444e180b0d7056762f888236528045854b963c12db5"} Aug 04 10:37:14.531832 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.531803 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dk4zq" event={"ID":"516d0545-a943-4d35-91f4-b8ac8b819560","Type":"ContainerStarted","Data":"45c2a48e68f5ac11beac68033daf86a332593a663e32c31f5e878ee0fd57458a"} Aug 04 10:37:14.533488 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.533461 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c9kzp" event={"ID":"fa44517a-0e03-4dad-b3b0-79ee2001d9cd","Type":"ContainerStarted","Data":"5c8c10d14e6864dd92cf23f0c6e7d095411e3467487bbfeaa21bb78797e84f21"} Aug 04 10:37:14.535103 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.535080 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b065ca8-d420-4ffa-9da9-ab3e60c06322" containerID="2f050a52df1d5c50f72abc8a96c25f1f5486e8243fc9a1f9bbc9e243a9df271a" exitCode=0 Aug 04 10:37:14.535206 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.535154 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f65wd" event={"ID":"3b065ca8-d420-4ffa-9da9-ab3e60c06322","Type":"ContainerDied","Data":"2f050a52df1d5c50f72abc8a96c25f1f5486e8243fc9a1f9bbc9e243a9df271a"} Aug 04 10:37:14.536937 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.536899 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8qppj" event={"ID":"4d7923f6-df0e-4f83-b6da-34d93041a405","Type":"ContainerStarted","Data":"61a851dcad189bf67baccb69218d29508728e47b98df393ea16514dd0520db7b"} Aug 04 10:37:14.538627 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.538603 2576 generic.go:356] "Generic (PLEG): container finished" podID="f6c23ed269e5911d443d635b804efbb1" containerID="400a798b426563aef2c16da94835dd91c6aaf50342e5de390c2c0718f1308661" exitCode=0 Aug 04 10:37:14.538728 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.538652 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal" event={"ID":"f6c23ed269e5911d443d635b804efbb1","Type":"ContainerDied","Data":"400a798b426563aef2c16da94835dd91c6aaf50342e5de390c2c0718f1308661"} Aug 04 10:37:14.540263 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.540245 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" event={"ID":"81e633a3-39f4-477f-b771-27f727e61657","Type":"ContainerStarted","Data":"472b7a9a94429ce17cca5fe280666ae5706ac03ba4abfa52f0dc698a3816cdfa"} Aug 04 10:37:14.548726 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.548685 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-psr7q" podStartSLOduration=3.29965887 podStartE2EDuration="22.548670432s" totalImagesPullingTime="19.249011562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:37:13.538902498 +0000 UTC m=+22.636695894" watchObservedRunningTime="2026-08-04 10:37:14.548670432 +0000 UTC m=+23.646463867" Aug 04 10:37:14.568301 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.568259 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r9kvv" podStartSLOduration=3.317231247 podStartE2EDuration="22.568241266s" totalImagesPullingTime="19.251010019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:37:14.548805344 +0000 UTC m=+23.646598742" watchObservedRunningTime="2026-08-04 10:37:14.568241266 +0000 UTC m=+23.666034660" Aug 04 10:37:14.568803 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.568777 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8qppj" podStartSLOduration=3.287932126 podStartE2EDuration="22.568769117s" totalImagesPullingTime="19.280836991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:37:14.568331265 +0000 UTC m=+23.666124660" watchObservedRunningTime="2026-08-04 10:37:14.568769117 +0000 UTC m=+23.666562554" Aug 04 10:37:14.661451 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:14.661232 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c9kzp" podStartSLOduration=3.384999698 podStartE2EDuration="22.661214736s" totalImagesPullingTime="19.276215038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:37:14.660903753 +0000 UTC m=+23.758697151" watchObservedRunningTime="2026-08-04 10:37:14.661214736 +0000 UTC m=+23.759008130" Aug 04 10:37:15.218397 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:15.218362 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 10:37:15.398912 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:15.398807 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T10:37:15.218390765Z","UUID":"5bd1dc8e-4692-4358-b33b-d12ee45aa680","Handler":null,"Name":"","Endpoint":""} Aug 04 10:37:15.400654 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:15.400627 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 10:37:15.400809 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:15.400675 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 10:37:15.432176 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:15.432144 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:15.432380 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:15.432186 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:15.432380 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:15.432274 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:37:15.432380 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:15.432353 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:37:15.432644 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:15.432420 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:15.432644 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:15.432478 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:37:15.545070 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:15.545032 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal" event={"ID":"f6c23ed269e5911d443d635b804efbb1","Type":"ContainerStarted","Data":"5c8dec69e71b644fc3918081d4dc39039258337e4e5a4201bfd4d181ec418fd8"} Aug 04 10:37:15.547002 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:15.546968 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l4r2b" event={"ID":"5f50bae9-b79c-4a6a-96ec-1d6edecf5f3f","Type":"ContainerStarted","Data":"768ce0ddeebae4161ea71c96f4154a91065540ceeef4596a7ed0e151d608e314"} Aug 04 10:37:15.549033 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:15.548964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" event={"ID":"81e633a3-39f4-477f-b771-27f727e61657","Type":"ContainerStarted","Data":"47db723bebde6ae0c0af0532aff405e4376cb9937c38af43cf5a14ff2ad4af24"} Aug 04 10:37:15.568982 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:15.568918 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dk4zq" podStartSLOduration=4.238353074 podStartE2EDuration="23.56889769s" totalImagesPullingTime="19.330544616s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:37:14.685713125 +0000 UTC m=+23.783506522" watchObservedRunningTime="2026-08-04 10:37:15.56889769 +0000 UTC m=+24.666691088" Aug 04 10:37:15.596127 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:15.596071 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-159.ec2.internal" podStartSLOduration=22.596049084 podStartE2EDuration="22.596049084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:37:15.568709652 +0000 UTC m=+24.666503048" watchObservedRunningTime="2026-08-04 10:37:15.596049084 +0000 UTC m=+24.693842481" Aug 04 10:37:15.596343 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:15.596253 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l4r2b" podStartSLOduration=4.314942422 podStartE2EDuration="23.59624122s" totalImagesPullingTime="19.281298798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:37:15.595778571 +0000 UTC m=+24.693571967" watchObservedRunningTime="2026-08-04 10:37:15.59624122 +0000 UTC m=+24.694034617" Aug 04 10:37:16.555124 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:16.555037 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" event={"ID":"068755a5-9b7a-4e8d-964d-5fe08b65aeb8","Type":"ContainerStarted","Data":"90651235283dc14a33f513dc5156e28f698efe5718b8003d6472d76fb269124d"} Aug 04 10:37:16.557380 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:16.557350 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" event={"ID":"81e633a3-39f4-477f-b771-27f727e61657","Type":"ContainerStarted","Data":"96d30d56ed617ef5716a44774ec87e2dedc1ddff34b0d7773367fd51f11c03d3"} Aug 04 10:37:16.582984 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:16.582930 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6b6n8" podStartSLOduration=3.28554082 podStartE2EDuration="24.582913661s" totalImagesPullingTime="21.297372841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:36:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:37:16.582601707 +0000 UTC m=+25.680395103" watchObservedRunningTime="2026-08-04 10:37:16.582913661 +0000 UTC m=+25.680707056" Aug 04 10:37:17.432007 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:17.431973 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:17.432230 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:17.431973 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:17.432230 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:17.432131 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:37:17.432230 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:17.431977 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:17.432400 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:17.432220 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:37:17.432400 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:17.432282 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:37:18.947153 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:18.946893 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8qppj" Aug 04 10:37:18.948074 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:18.948055 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8qppj" Aug 04 10:37:19.431695 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:19.431660 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:19.431695 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:19.431660 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:19.431695 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:19.431660 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:19.431973 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:19.431775 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:37:19.431973 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:19.431847 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:37:19.431973 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:19.431892 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:37:19.567393 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:19.567356 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" event={"ID":"068755a5-9b7a-4e8d-964d-5fe08b65aeb8","Type":"ContainerStarted","Data":"07d429ea41d5524abac07cc51c92cd490ba00e5985cbaee90e123dc7c013dba0"} Aug 04 10:37:19.567651 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:19.567627 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:37:19.567746 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:19.567697 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:37:19.567805 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:19.567747 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:37:19.569298 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:19.569269 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b065ca8-d420-4ffa-9da9-ab3e60c06322" containerID="1964f7da534841dd29f3c8cb5b781456472c140a58874be8e79fd546a13aaed0" exitCode=0 Aug 04 10:37:19.569442 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:19.569348 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f65wd" event={"ID":"3b065ca8-d420-4ffa-9da9-ab3e60c06322","Type":"ContainerDied","Data":"1964f7da534841dd29f3c8cb5b781456472c140a58874be8e79fd546a13aaed0"} Aug 04 10:37:19.569554 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:19.569532 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8qppj" Aug 04 10:37:19.570548 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:19.570040 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8qppj" Aug 04 10:37:19.583704 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:19.583678 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:37:19.583837 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:19.583776 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:37:19.601538 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:19.601475 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" podStartSLOduration=8.287734185 podStartE2EDuration="27.601460587s" totalImagesPullingTime="19.313726402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:37:19.601020087 +0000 UTC m=+28.698813482" watchObservedRunningTime="2026-08-04 10:37:19.601460587 +0000 UTC m=+28.699253982" Aug 04 10:37:20.975385 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:20.975111 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qxldr"] Aug 04 10:37:20.975990 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:20.975465 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:20.975990 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:20.975628 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:37:20.975990 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:20.975743 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gr6kd"] Aug 04 10:37:20.975990 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:20.975846 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:20.975990 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:20.975934 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:37:20.976608 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:20.976584 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldmtm"] Aug 04 10:37:20.976715 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:20.976678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:20.976785 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:20.976766 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:37:21.576382 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:21.576347 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b065ca8-d420-4ffa-9da9-ab3e60c06322" containerID="3a9104c83866b4e33819ba9a268d277244a05fe75a6f29667c1a0ab8c122d112" exitCode=0 Aug 04 10:37:21.576596 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:21.576430 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f65wd" event={"ID":"3b065ca8-d420-4ffa-9da9-ab3e60c06322","Type":"ContainerDied","Data":"3a9104c83866b4e33819ba9a268d277244a05fe75a6f29667c1a0ab8c122d112"} Aug 04 10:37:22.431667 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:22.431633 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:22.432048 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:22.431633 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:22.432048 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:22.431736 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:37:22.432048 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:22.431814 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:37:23.431419 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:23.431381 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:23.431633 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:23.431551 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:37:23.584092 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:23.584059 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b065ca8-d420-4ffa-9da9-ab3e60c06322" containerID="54d52eba4e92a9a1109cd38a39ce48c990205e0c5c1ad1cb2d12ff3b0efb70fe" exitCode=0 Aug 04 10:37:23.584475 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:23.584109 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f65wd" event={"ID":"3b065ca8-d420-4ffa-9da9-ab3e60c06322","Type":"ContainerDied","Data":"54d52eba4e92a9a1109cd38a39ce48c990205e0c5c1ad1cb2d12ff3b0efb70fe"} Aug 04 10:37:24.431756 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:24.431559 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:24.431955 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:24.431559 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:24.431955 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:24.431828 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxldr" podUID="5b51f136-2d2f-4652-840c-54033e0df787" Aug 04 10:37:24.432201 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:24.432168 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr6kd" podUID="0d71b33c-089f-4d23-be91-d48f28a9c333" Aug 04 10:37:25.199205 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:25.199163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:25.199701 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:25.199275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:25.199701 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:25.199325 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:37:25.199701 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:25.199401 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs podName:f67d2ed4-7812-45de-95c0-32cf86d9eef2 nodeName:}" failed. No retries permitted until 2026-08-04 10:37:57.199379712 +0000 UTC m=+66.297173086 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs") pod "network-metrics-daemon-ldmtm" (UID: "f67d2ed4-7812-45de-95c0-32cf86d9eef2") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:37:25.199701 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:25.199422 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:37:25.199701 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:25.199526 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret podName:0d71b33c-089f-4d23-be91-d48f28a9c333 nodeName:}" failed. No retries permitted until 2026-08-04 10:37:57.199462695 +0000 UTC m=+66.297256073 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret") pod "global-pull-secret-syncer-gr6kd" (UID: "0d71b33c-089f-4d23-be91-d48f28a9c333") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:37:25.300249 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:25.300215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzf2\" (UniqueName: \"kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2\") pod \"network-check-target-qxldr\" (UID: \"5b51f136-2d2f-4652-840c-54033e0df787\") " pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:25.300440 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:25.300404 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:37:25.300440 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:25.300430 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:37:25.300440 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:25.300442 2576 projected.go:196] Error preparing data for projected volume kube-api-access-cfzf2 for pod openshift-network-diagnostics/network-check-target-qxldr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:37:25.300641 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:25.300526 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2 podName:5b51f136-2d2f-4652-840c-54033e0df787 nodeName:}" failed. No retries permitted until 2026-08-04 10:37:57.300485834 +0000 UTC m=+66.398279212 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfzf2" (UniqueName: "kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2") pod "network-check-target-qxldr" (UID: "5b51f136-2d2f-4652-840c-54033e0df787") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:37:25.434946 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:25.434915 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:25.435125 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:25.435033 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:37:26.186529 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.186437 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-159.ec2.internal" event="NodeReady" Aug 04 10:37:26.186691 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.186602 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 10:37:26.236470 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.236435 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vlqv6"] Aug 04 10:37:26.238565 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.238539 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:26.241411 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.241371 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 10:37:26.241611 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.241425 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f4ggs\"" Aug 04 10:37:26.241711 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.241693 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 10:37:26.242109 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.242092 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xwpcd"] Aug 04 10:37:26.243928 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.243908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xwpcd" Aug 04 10:37:26.246527 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.246405 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 10:37:26.247107 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.247078 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l5h9c\"" Aug 04 10:37:26.247107 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.247099 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 10:37:26.247367 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.247351 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 10:37:26.250728 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.250707 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vlqv6"] Aug 04 10:37:26.263927 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.263898 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xwpcd"] Aug 04 10:37:26.309044 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.309007 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:26.309221 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.309055 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-tmp-dir\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:26.309221 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.309187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk2nt\" (UniqueName: \"kubernetes.io/projected/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-kube-api-access-bk2nt\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:26.309221 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.309209 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-config-volume\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:26.410479 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.410445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk2nt\" (UniqueName: \"kubernetes.io/projected/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-kube-api-access-bk2nt\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:26.410479 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.410484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhxgc\" (UniqueName: \"kubernetes.io/projected/b86857b2-2acf-4829-b4fb-2d5ac4bf5236-kube-api-access-dhxgc\") pod \"ingress-canary-xwpcd\" (UID: \"b86857b2-2acf-4829-b4fb-2d5ac4bf5236\") " pod="openshift-ingress-canary/ingress-canary-xwpcd" Aug 04 10:37:26.410686 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.410522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-config-volume\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:26.410686 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.410654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:26.410750 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.410688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-tmp-dir\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:26.410782 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.410752 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b86857b2-2acf-4829-b4fb-2d5ac4bf5236-cert\") pod \"ingress-canary-xwpcd\" (UID: \"b86857b2-2acf-4829-b4fb-2d5ac4bf5236\") " pod="openshift-ingress-canary/ingress-canary-xwpcd" Aug 04 10:37:26.410837 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:26.410814 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:37:26.410917 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:26.410906 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls podName:683c4f2b-d66c-4b5c-b6ee-96d4007f49dc nodeName:}" failed. No retries permitted until 2026-08-04 10:37:26.910884367 +0000 UTC m=+36.008677754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls") pod "dns-default-vlqv6" (UID: "683c4f2b-d66c-4b5c-b6ee-96d4007f49dc") : secret "dns-default-metrics-tls" not found Aug 04 10:37:26.411155 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.411136 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-tmp-dir\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:26.411312 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.411297 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-config-volume\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:26.422293 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.422260 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk2nt\" (UniqueName: \"kubernetes.io/projected/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-kube-api-access-bk2nt\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:26.432115 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.432087 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:26.432281 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.432132 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:26.435144 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.435120 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 10:37:26.435286 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.435240 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 10:37:26.435350 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.435313 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 10:37:26.435402 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.435364 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qf5ds\"" Aug 04 10:37:26.511810 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.511777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b86857b2-2acf-4829-b4fb-2d5ac4bf5236-cert\") pod \"ingress-canary-xwpcd\" (UID: \"b86857b2-2acf-4829-b4fb-2d5ac4bf5236\") " pod="openshift-ingress-canary/ingress-canary-xwpcd" Aug 04 10:37:26.511991 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.511831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhxgc\" (UniqueName: \"kubernetes.io/projected/b86857b2-2acf-4829-b4fb-2d5ac4bf5236-kube-api-access-dhxgc\") pod \"ingress-canary-xwpcd\" (UID: \"b86857b2-2acf-4829-b4fb-2d5ac4bf5236\") " pod="openshift-ingress-canary/ingress-canary-xwpcd" Aug 04 10:37:26.515610 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.515546 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b86857b2-2acf-4829-b4fb-2d5ac4bf5236-cert\") pod \"ingress-canary-xwpcd\" (UID: \"b86857b2-2acf-4829-b4fb-2d5ac4bf5236\") " pod="openshift-ingress-canary/ingress-canary-xwpcd" Aug 04 10:37:26.523188 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.523162 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhxgc\" (UniqueName: \"kubernetes.io/projected/b86857b2-2acf-4829-b4fb-2d5ac4bf5236-kube-api-access-dhxgc\") pod \"ingress-canary-xwpcd\" (UID: \"b86857b2-2acf-4829-b4fb-2d5ac4bf5236\") " pod="openshift-ingress-canary/ingress-canary-xwpcd" Aug 04 10:37:26.558268 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.558232 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xwpcd" Aug 04 10:37:26.915768 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:26.915733 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:26.915951 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:26.915874 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:37:26.915951 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:26.915945 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls podName:683c4f2b-d66c-4b5c-b6ee-96d4007f49dc nodeName:}" failed. No retries permitted until 2026-08-04 10:37:27.915925032 +0000 UTC m=+37.013718429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls") pod "dns-default-vlqv6" (UID: "683c4f2b-d66c-4b5c-b6ee-96d4007f49dc") : secret "dns-default-metrics-tls" not found Aug 04 10:37:27.431800 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:27.431768 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:27.434442 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:27.434419 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 10:37:27.434599 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:27.434428 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t7blv\"" Aug 04 10:37:27.923617 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:27.923541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:27.923769 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:27.923683 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:37:27.923769 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:27.923748 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls podName:683c4f2b-d66c-4b5c-b6ee-96d4007f49dc nodeName:}" failed. No retries permitted until 2026-08-04 10:37:29.923731129 +0000 UTC m=+39.021524502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls") pod "dns-default-vlqv6" (UID: "683c4f2b-d66c-4b5c-b6ee-96d4007f49dc") : secret "dns-default-metrics-tls" not found Aug 04 10:37:28.011165 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:28.010978 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xwpcd"] Aug 04 10:37:28.013995 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:37:28.013961 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86857b2_2acf_4829_b4fb_2d5ac4bf5236.slice/crio-a448399d725ff62513dea2e8236fbf35d9e3c8c04e66345123aba45d8b2af15a WatchSource:0}: Error finding container a448399d725ff62513dea2e8236fbf35d9e3c8c04e66345123aba45d8b2af15a: Status 404 returned error can't find the container with id a448399d725ff62513dea2e8236fbf35d9e3c8c04e66345123aba45d8b2af15a Aug 04 10:37:28.600223 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:28.600190 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b065ca8-d420-4ffa-9da9-ab3e60c06322" containerID="9825d126f3cbee65776f3b4d24bf1ba221c5ceda906a7c739e8428fbb5a08b6f" exitCode=0 Aug 04 10:37:28.600889 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:28.600283 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f65wd" event={"ID":"3b065ca8-d420-4ffa-9da9-ab3e60c06322","Type":"ContainerDied","Data":"9825d126f3cbee65776f3b4d24bf1ba221c5ceda906a7c739e8428fbb5a08b6f"} Aug 04 10:37:28.601467 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:28.601444 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xwpcd" event={"ID":"b86857b2-2acf-4829-b4fb-2d5ac4bf5236","Type":"ContainerStarted","Data":"a448399d725ff62513dea2e8236fbf35d9e3c8c04e66345123aba45d8b2af15a"} Aug 04 10:37:29.388525 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:29.388473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:29.528950 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:29.528913 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:29.608697 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:29.608663 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b065ca8-d420-4ffa-9da9-ab3e60c06322" containerID="c675ccc445226c99b62f87f3f674f6195aa3b0d3ad17beac05a3aff04a986f6b" exitCode=0 Aug 04 10:37:29.609160 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:29.608703 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f65wd" event={"ID":"3b065ca8-d420-4ffa-9da9-ab3e60c06322","Type":"ContainerDied","Data":"c675ccc445226c99b62f87f3f674f6195aa3b0d3ad17beac05a3aff04a986f6b"} Aug 04 10:37:29.940257 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:29.940208 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:29.940777 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:29.940389 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:37:29.940777 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:29.940482 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls podName:683c4f2b-d66c-4b5c-b6ee-96d4007f49dc nodeName:}" failed. No retries permitted until 2026-08-04 10:37:33.940459172 +0000 UTC m=+43.038252557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls") pod "dns-default-vlqv6" (UID: "683c4f2b-d66c-4b5c-b6ee-96d4007f49dc") : secret "dns-default-metrics-tls" not found Aug 04 10:37:30.088992 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:30.088957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:30.181688 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:30.181659 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:30.336597 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:30.336564 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:30.415846 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:30.415811 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:30.525197 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:30.525172 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:30.614950 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:30.614915 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f65wd" event={"ID":"3b065ca8-d420-4ffa-9da9-ab3e60c06322","Type":"ContainerStarted","Data":"0e36f17a35cf3c8e74cfa8d81b8863d63ccb33b5a47fca42001824094784b358"} Aug 04 10:37:30.616366 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:30.616329 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xwpcd" event={"ID":"b86857b2-2acf-4829-b4fb-2d5ac4bf5236","Type":"ContainerStarted","Data":"edd79f7c901b43634e8e08469562536dc38e4a5325770b64c3ff1ff383d8af4b"} Aug 04 10:37:30.643931 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:30.643881 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f65wd" podStartSLOduration=8.646638038 podStartE2EDuration="38.643863469s" totalImagesPullingTime="29.997225431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:36:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:37:30.642695581 +0000 UTC m=+39.740488976" watchObservedRunningTime="2026-08-04 10:37:30.643863469 +0000 UTC m=+39.741656863" Aug 04 10:37:30.660026 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:30.659968 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xwpcd" podStartSLOduration=2.224350463 podStartE2EDuration="4.659949059s" totalImagesPullingTime="2.435598596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:37:30.659172936 +0000 UTC m=+39.756966331" watchObservedRunningTime="2026-08-04 10:37:30.659949059 +0000 UTC m=+39.757742435" Aug 04 10:37:30.995221 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:30.995040 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:31.169630 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:31.169597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:31.462562 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:31.462522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:31.563908 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:31.563879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:32.290475 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:32.290444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:32.409359 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:32.409325 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:32.652426 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:32.652383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:32.750686 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:32.750660 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:33.971307 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:33.971252 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:33.971731 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:33.971382 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:37:33.971731 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:33.971436 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls podName:683c4f2b-d66c-4b5c-b6ee-96d4007f49dc nodeName:}" failed. No retries permitted until 2026-08-04 10:37:41.971423504 +0000 UTC m=+51.069216877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls") pod "dns-default-vlqv6" (UID: "683c4f2b-d66c-4b5c-b6ee-96d4007f49dc") : secret "dns-default-metrics-tls" not found Aug 04 10:37:34.829298 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:34.829262 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:34.938791 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:34.938763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:39.174253 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:39.174209 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:39.256991 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:39.256955 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:42.030615 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:42.030574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:42.031113 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:42.030694 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:37:42.031113 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:42.030752 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls podName:683c4f2b-d66c-4b5c-b6ee-96d4007f49dc nodeName:}" failed. No retries permitted until 2026-08-04 10:37:58.0307365 +0000 UTC m=+67.128529875 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls") pod "dns-default-vlqv6" (UID: "683c4f2b-d66c-4b5c-b6ee-96d4007f49dc") : secret "dns-default-metrics-tls" not found Aug 04 10:37:51.587296 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:51.587264 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hmgnh" Aug 04 10:37:51.745098 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:51.745064 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:51.845270 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:51.845194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:52.185521 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:52.185431 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:52.279951 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:52.279907 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:52.698549 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:52.698494 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:52.764658 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:52.764604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:53.100533 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:53.100452 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:53.202242 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:53.202210 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:53.904823 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:53.904782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:54.296713 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:54.296604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:54.365606 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:54.365540 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:37:57.240295 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.240258 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:37:57.240772 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.240314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:57.242989 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.242970 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 10:37:57.243033 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.243011 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 10:37:57.250933 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:57.250902 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 04 10:37:57.251005 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:57.250975 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs podName:f67d2ed4-7812-45de-95c0-32cf86d9eef2 nodeName:}" failed. No retries permitted until 2026-08-04 10:39:01.25095771 +0000 UTC m=+130.348751088 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs") pod "network-metrics-daemon-ldmtm" (UID: "f67d2ed4-7812-45de-95c0-32cf86d9eef2") : secret "metrics-daemon-secret" not found Aug 04 10:37:57.254737 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.254719 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d71b33c-089f-4d23-be91-d48f28a9c333-original-pull-secret\") pod \"global-pull-secret-syncer-gr6kd\" (UID: \"0d71b33c-089f-4d23-be91-d48f28a9c333\") " pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:57.341483 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.341440 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzf2\" (UniqueName: \"kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2\") pod \"network-check-target-qxldr\" (UID: \"5b51f136-2d2f-4652-840c-54033e0df787\") " pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:57.343454 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.343433 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr6kd" Aug 04 10:37:57.344358 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.344340 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 10:37:57.354141 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.354121 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 10:37:57.364298 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.364277 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfzf2\" (UniqueName: \"kubernetes.io/projected/5b51f136-2d2f-4652-840c-54033e0df787-kube-api-access-cfzf2\") pod \"network-check-target-qxldr\" (UID: \"5b51f136-2d2f-4652-840c-54033e0df787\") " pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:57.469817 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.469782 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gr6kd"] Aug 04 10:37:57.473215 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:37:57.473186 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d71b33c_089f_4d23_be91_d48f28a9c333.slice/crio-a9e5f37c23266ae72ba5a8dfcec2ab65830388053d6595cb16e85acabffbc6eb WatchSource:0}: Error finding container a9e5f37c23266ae72ba5a8dfcec2ab65830388053d6595cb16e85acabffbc6eb: Status 404 returned error can't find the container with id a9e5f37c23266ae72ba5a8dfcec2ab65830388053d6595cb16e85acabffbc6eb Aug 04 10:37:57.652206 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.652179 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qf5ds\"" Aug 04 10:37:57.660570 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.660551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:37:57.680144 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.680110 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gr6kd" event={"ID":"0d71b33c-089f-4d23-be91-d48f28a9c333","Type":"ContainerStarted","Data":"a9e5f37c23266ae72ba5a8dfcec2ab65830388053d6595cb16e85acabffbc6eb"} Aug 04 10:37:57.783994 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:57.783963 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qxldr"] Aug 04 10:37:57.788657 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:37:57.788620 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b51f136_2d2f_4652_840c_54033e0df787.slice/crio-4cc0cc190bd03461b97d8522be2b170471c004d70e84f68c33c2aecb84120a6d WatchSource:0}: Error finding container 4cc0cc190bd03461b97d8522be2b170471c004d70e84f68c33c2aecb84120a6d: Status 404 returned error can't find the container with id 4cc0cc190bd03461b97d8522be2b170471c004d70e84f68c33c2aecb84120a6d Aug 04 10:37:58.045183 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:58.045098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:37:58.045340 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:58.045234 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:37:58.045340 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:37:58.045299 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls podName:683c4f2b-d66c-4b5c-b6ee-96d4007f49dc nodeName:}" failed. No retries permitted until 2026-08-04 10:38:30.045282156 +0000 UTC m=+99.143075530 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls") pod "dns-default-vlqv6" (UID: "683c4f2b-d66c-4b5c-b6ee-96d4007f49dc") : secret "dns-default-metrics-tls" not found Aug 04 10:37:58.685346 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:37:58.685295 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qxldr" event={"ID":"5b51f136-2d2f-4652-840c-54033e0df787","Type":"ContainerStarted","Data":"4cc0cc190bd03461b97d8522be2b170471c004d70e84f68c33c2aecb84120a6d"} Aug 04 10:38:02.697950 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:02.697912 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gr6kd" event={"ID":"0d71b33c-089f-4d23-be91-d48f28a9c333","Type":"ContainerStarted","Data":"e536ab5710fbbe2971f0c4b5c7b1310ef3843bc24551d07f57bff478ed159390"} Aug 04 10:38:02.699329 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:02.699297 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qxldr" event={"ID":"5b51f136-2d2f-4652-840c-54033e0df787","Type":"ContainerStarted","Data":"80610f934b7c612f2c9bd7d5731914346e5b9bc3d1101cc2f2b0a11387e95918"} Aug 04 10:38:02.699451 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:02.699353 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:38:02.715110 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:02.715046 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gr6kd" podStartSLOduration=65.884760031 podStartE2EDuration="1m10.715028977s" totalImagesPullingTime="4.830268946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:38:02.714840925 +0000 UTC m=+71.812634322" watchObservedRunningTime="2026-08-04 10:38:02.715028977 +0000 UTC m=+71.812822372" Aug 04 10:38:02.730867 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:02.730809 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qxldr" podStartSLOduration=66.219870634 podStartE2EDuration="1m10.730790431s" totalImagesPullingTime="4.510919797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:38:02.730103893 +0000 UTC m=+71.827897289" watchObservedRunningTime="2026-08-04 10:38:02.730790431 +0000 UTC m=+71.828583826" Aug 04 10:38:03.889931 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:03.889898 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:05.332482 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:05.332453 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:09.285666 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:09.285628 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:09.374119 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:09.374087 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:10.287248 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:10.287216 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:10.390787 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:10.390760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:17.854530 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:17.854397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:17.950739 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:17.950706 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:30.067397 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:30.067353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:38:30.067850 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:30.067534 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:38:30.067850 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:30.067603 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls podName:683c4f2b-d66c-4b5c-b6ee-96d4007f49dc nodeName:}" failed. No retries permitted until 2026-08-04 10:39:34.067586004 +0000 UTC m=+163.165379377 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls") pod "dns-default-vlqv6" (UID: "683c4f2b-d66c-4b5c-b6ee-96d4007f49dc") : secret "dns-default-metrics-tls" not found Aug 04 10:38:30.880661 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:30.880622 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:30.967520 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:30.967474 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:33.704669 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:33.704637 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qxldr" Aug 04 10:38:36.587679 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.587643 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd"] Aug 04 10:38:36.609807 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.609779 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd"] Aug 04 10:38:36.609956 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.609891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:38:36.614779 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.614731 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Aug 04 10:38:36.614779 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.614761 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Aug 04 10:38:36.615054 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.614855 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wshwv\"" Aug 04 10:38:36.615054 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.614890 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 10:38:36.615054 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.614951 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 10:38:36.694223 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.694192 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-798f494f76-94bvh"] Aug 04 10:38:36.710806 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.710766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4l2w\" (UniqueName: \"kubernetes.io/projected/97e701ea-3b55-4c01-8e14-695a759a7ad8-kube-api-access-m4l2w\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:38:36.710971 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.710825 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:38:36.710971 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.710867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/97e701ea-3b55-4c01-8e14-695a759a7ad8-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:38:36.717413 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.717393 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-798f494f76-94bvh"] Aug 04 10:38:36.717534 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.717491 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:36.720255 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.720236 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 10:38:36.720357 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.720237 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Aug 04 10:38:36.720491 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.720474 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fs7n7\"" Aug 04 10:38:36.720730 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.720716 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Aug 04 10:38:36.720820 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.720732 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Aug 04 10:38:36.720820 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.720783 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 10:38:36.720927 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.720801 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Aug 04 10:38:36.812070 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.812034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:38:36.812070 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.812071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/97e701ea-3b55-4c01-8e14-695a759a7ad8-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:38:36.812338 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.812096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-stats-auth\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:36.812338 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.812122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qd7c\" (UniqueName: \"kubernetes.io/projected/50e09e74-c462-4d60-a61a-b281bf126d83-kube-api-access-6qd7c\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:36.812338 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:36.812187 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 10:38:36.812338 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.812193 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-default-certificate\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:36.812338 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:36.812263 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls podName:97e701ea-3b55-4c01-8e14-695a759a7ad8 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:37.31224699 +0000 UTC m=+106.410040363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-bfjqd" (UID: "97e701ea-3b55-4c01-8e14-695a759a7ad8") : secret "cluster-monitoring-operator-tls" not found Aug 04 10:38:36.812338 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.812293 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:36.812338 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.812312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4l2w\" (UniqueName: \"kubernetes.io/projected/97e701ea-3b55-4c01-8e14-695a759a7ad8-kube-api-access-m4l2w\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:38:36.812338 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.812328 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:36.813026 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.813004 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/97e701ea-3b55-4c01-8e14-695a759a7ad8-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:38:36.826910 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.826888 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4l2w\" (UniqueName: \"kubernetes.io/projected/97e701ea-3b55-4c01-8e14-695a759a7ad8-kube-api-access-m4l2w\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:38:36.855105 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.855029 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:36.913294 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.913258 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qd7c\" (UniqueName: \"kubernetes.io/projected/50e09e74-c462-4d60-a61a-b281bf126d83-kube-api-access-6qd7c\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:36.913429 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.913315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-default-certificate\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:36.913429 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.913340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:36.913429 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.913359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:36.913429 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.913408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-stats-auth\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:36.913673 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:36.913519 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 10:38:36.913673 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:36.913564 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle podName:50e09e74-c462-4d60-a61a-b281bf126d83 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:37.413547468 +0000 UTC m=+106.511340860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle") pod "router-default-798f494f76-94bvh" (UID: "50e09e74-c462-4d60-a61a-b281bf126d83") : configmap references non-existent config key: service-ca.crt Aug 04 10:38:36.913673 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:36.913588 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs podName:50e09e74-c462-4d60-a61a-b281bf126d83 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:37.413580668 +0000 UTC m=+106.511374041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs") pod "router-default-798f494f76-94bvh" (UID: "50e09e74-c462-4d60-a61a-b281bf126d83") : secret "router-metrics-certs-default" not found Aug 04 10:38:36.915769 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.915745 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-default-certificate\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:36.915902 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.915882 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-stats-auth\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:36.924542 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.924523 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qd7c\" (UniqueName: \"kubernetes.io/projected/50e09e74-c462-4d60-a61a-b281bf126d83-kube-api-access-6qd7c\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:36.960002 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:36.959972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:37.316948 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:37.316848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:38:37.317100 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:37.317024 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 10:38:37.317100 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:37.317098 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls podName:97e701ea-3b55-4c01-8e14-695a759a7ad8 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:38.317081224 +0000 UTC m=+107.414874597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-bfjqd" (UID: "97e701ea-3b55-4c01-8e14-695a759a7ad8") : secret "cluster-monitoring-operator-tls" not found Aug 04 10:38:37.417439 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:37.417391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:37.417439 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:37.417448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:37.417619 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:37.417575 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 10:38:37.417664 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:37.417582 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle podName:50e09e74-c462-4d60-a61a-b281bf126d83 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:38.417569581 +0000 UTC m=+107.515362954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle") pod "router-default-798f494f76-94bvh" (UID: "50e09e74-c462-4d60-a61a-b281bf126d83") : configmap references non-existent config key: service-ca.crt Aug 04 10:38:37.417711 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:37.417669 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs podName:50e09e74-c462-4d60-a61a-b281bf126d83 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:38.417651092 +0000 UTC m=+107.515444468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs") pod "router-default-798f494f76-94bvh" (UID: "50e09e74-c462-4d60-a61a-b281bf126d83") : secret "router-metrics-certs-default" not found Aug 04 10:38:38.324151 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:38.324104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:38:38.324647 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:38.324259 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 10:38:38.324647 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:38.324336 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls podName:97e701ea-3b55-4c01-8e14-695a759a7ad8 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:40.324317532 +0000 UTC m=+109.422110907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-bfjqd" (UID: "97e701ea-3b55-4c01-8e14-695a759a7ad8") : secret "cluster-monitoring-operator-tls" not found Aug 04 10:38:38.424914 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:38.424873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:38.424914 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:38.424919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:38.425098 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:38.425035 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 10:38:38.425098 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:38.425086 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle podName:50e09e74-c462-4d60-a61a-b281bf126d83 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:40.425066627 +0000 UTC m=+109.522860020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle") pod "router-default-798f494f76-94bvh" (UID: "50e09e74-c462-4d60-a61a-b281bf126d83") : configmap references non-existent config key: service-ca.crt Aug 04 10:38:38.425163 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:38.425104 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs podName:50e09e74-c462-4d60-a61a-b281bf126d83 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:40.425096449 +0000 UTC m=+109.522889822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs") pod "router-default-798f494f76-94bvh" (UID: "50e09e74-c462-4d60-a61a-b281bf126d83") : secret "router-metrics-certs-default" not found Aug 04 10:38:40.337622 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:40.337578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:38:40.338042 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:40.337756 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 10:38:40.338042 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:40.337820 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls podName:97e701ea-3b55-4c01-8e14-695a759a7ad8 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:44.337805186 +0000 UTC m=+113.435598559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-bfjqd" (UID: "97e701ea-3b55-4c01-8e14-695a759a7ad8") : secret "cluster-monitoring-operator-tls" not found Aug 04 10:38:40.438723 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:40.438677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:40.438723 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:40.438727 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:40.438897 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:40.438818 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 10:38:40.438897 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:40.438848 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle podName:50e09e74-c462-4d60-a61a-b281bf126d83 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:44.438834965 +0000 UTC m=+113.536628338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle") pod "router-default-798f494f76-94bvh" (UID: "50e09e74-c462-4d60-a61a-b281bf126d83") : configmap references non-existent config key: service-ca.crt Aug 04 10:38:40.438897 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:40.438867 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs podName:50e09e74-c462-4d60-a61a-b281bf126d83 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:44.438855597 +0000 UTC m=+113.536648969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs") pod "router-default-798f494f76-94bvh" (UID: "50e09e74-c462-4d60-a61a-b281bf126d83") : secret "router-metrics-certs-default" not found Aug 04 10:38:44.364765 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.364707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:38:44.365180 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:44.364862 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 10:38:44.365180 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:44.364937 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls podName:97e701ea-3b55-4c01-8e14-695a759a7ad8 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:52.364921214 +0000 UTC m=+121.462714587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-bfjqd" (UID: "97e701ea-3b55-4c01-8e14-695a759a7ad8") : secret "cluster-monitoring-operator-tls" not found Aug 04 10:38:44.465301 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.465254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:44.465301 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.465306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:44.465450 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:44.465404 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 10:38:44.465450 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:44.465411 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle podName:50e09e74-c462-4d60-a61a-b281bf126d83 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:52.465398336 +0000 UTC m=+121.563191709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle") pod "router-default-798f494f76-94bvh" (UID: "50e09e74-c462-4d60-a61a-b281bf126d83") : configmap references non-existent config key: service-ca.crt Aug 04 10:38:44.465559 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:44.465477 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs podName:50e09e74-c462-4d60-a61a-b281bf126d83 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:52.465451981 +0000 UTC m=+121.563245354 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs") pod "router-default-798f494f76-94bvh" (UID: "50e09e74-c462-4d60-a61a-b281bf126d83") : secret "router-metrics-certs-default" not found Aug 04 10:38:44.652392 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.652311 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv"] Aug 04 10:38:44.654114 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.654098 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" Aug 04 10:38:44.656643 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.656619 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Aug 04 10:38:44.656990 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.656963 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Aug 04 10:38:44.657098 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.656968 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:38:44.657705 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.657690 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fjj5q\"" Aug 04 10:38:44.672823 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.672798 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv"] Aug 04 10:38:44.768427 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.768390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6grj\" (UniqueName: \"kubernetes.io/projected/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-kube-api-access-b6grj\") pod \"cluster-samples-operator-559cbf5745-qglcv\" (UID: \"1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" Aug 04 10:38:44.768427 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.768433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qglcv\" (UID: \"1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" Aug 04 10:38:44.869709 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.869675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6grj\" (UniqueName: \"kubernetes.io/projected/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-kube-api-access-b6grj\") pod \"cluster-samples-operator-559cbf5745-qglcv\" (UID: \"1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" Aug 04 10:38:44.869871 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.869716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qglcv\" (UID: \"1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" Aug 04 10:38:44.869871 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:44.869825 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 10:38:44.869939 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:44.869885 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls podName:1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a nodeName:}" failed. No retries permitted until 2026-08-04 10:38:45.369869601 +0000 UTC m=+114.467662974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qglcv" (UID: "1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a") : secret "samples-operator-tls" not found Aug 04 10:38:44.878409 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:44.878379 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6grj\" (UniqueName: \"kubernetes.io/projected/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-kube-api-access-b6grj\") pod \"cluster-samples-operator-559cbf5745-qglcv\" (UID: \"1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" Aug 04 10:38:45.373936 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:45.373899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qglcv\" (UID: \"1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" Aug 04 10:38:45.374305 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:45.374021 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 10:38:45.374305 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:45.374087 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls podName:1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a nodeName:}" failed. No retries permitted until 2026-08-04 10:38:46.374071911 +0000 UTC m=+115.471865285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qglcv" (UID: "1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a") : secret "samples-operator-tls" not found Aug 04 10:38:46.381410 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.381377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qglcv\" (UID: \"1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" Aug 04 10:38:46.381889 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:46.381583 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 10:38:46.381889 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:46.381665 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls podName:1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a nodeName:}" failed. No retries permitted until 2026-08-04 10:38:48.381644304 +0000 UTC m=+117.479437681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qglcv" (UID: "1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a") : secret "samples-operator-tls" not found Aug 04 10:38:46.636739 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.636657 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh"] Aug 04 10:38:46.638423 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.638408 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" Aug 04 10:38:46.640957 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.640935 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Aug 04 10:38:46.640957 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.640950 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Aug 04 10:38:46.641104 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.640938 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:38:46.641801 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.641784 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-w69gv\"" Aug 04 10:38:46.641869 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.641815 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Aug 04 10:38:46.647114 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.647095 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh"] Aug 04 10:38:46.742656 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.742626 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m"] Aug 04 10:38:46.744632 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.744615 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" Aug 04 10:38:46.747174 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.747142 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Aug 04 10:38:46.747174 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.747160 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Aug 04 10:38:46.747402 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.747383 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Aug 04 10:38:46.747540 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.747467 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:38:46.747540 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.747469 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-w7gc9\"" Aug 04 10:38:46.756082 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.756062 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m"] Aug 04 10:38:46.784623 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.784588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws4wg\" (UniqueName: \"kubernetes.io/projected/ee5a68b7-d654-4644-a54e-68191be69890-kube-api-access-ws4wg\") pod \"service-ca-operator-784f97bcf6-hc4bh\" (UID: \"ee5a68b7-d654-4644-a54e-68191be69890\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" Aug 04 10:38:46.784623 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.784629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5a68b7-d654-4644-a54e-68191be69890-serving-cert\") pod \"service-ca-operator-784f97bcf6-hc4bh\" (UID: \"ee5a68b7-d654-4644-a54e-68191be69890\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" Aug 04 10:38:46.784827 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.784700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee5a68b7-d654-4644-a54e-68191be69890-config\") pod \"service-ca-operator-784f97bcf6-hc4bh\" (UID: \"ee5a68b7-d654-4644-a54e-68191be69890\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" Aug 04 10:38:46.885836 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.885795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws4wg\" (UniqueName: \"kubernetes.io/projected/ee5a68b7-d654-4644-a54e-68191be69890-kube-api-access-ws4wg\") pod \"service-ca-operator-784f97bcf6-hc4bh\" (UID: \"ee5a68b7-d654-4644-a54e-68191be69890\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" Aug 04 10:38:46.885836 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.885842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5a68b7-d654-4644-a54e-68191be69890-serving-cert\") pod \"service-ca-operator-784f97bcf6-hc4bh\" (UID: \"ee5a68b7-d654-4644-a54e-68191be69890\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" Aug 04 10:38:46.886076 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.885872 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5gkx\" (UniqueName: \"kubernetes.io/projected/83c72ce4-7b81-4396-9be6-7b5e71d1a3ff-kube-api-access-z5gkx\") pod \"kube-storage-version-migrator-operator-7878f4b88c-h9x4m\" (UID: \"83c72ce4-7b81-4396-9be6-7b5e71d1a3ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" Aug 04 10:38:46.886076 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.885902 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83c72ce4-7b81-4396-9be6-7b5e71d1a3ff-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-h9x4m\" (UID: \"83c72ce4-7b81-4396-9be6-7b5e71d1a3ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" Aug 04 10:38:46.886076 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.885955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee5a68b7-d654-4644-a54e-68191be69890-config\") pod \"service-ca-operator-784f97bcf6-hc4bh\" (UID: \"ee5a68b7-d654-4644-a54e-68191be69890\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" Aug 04 10:38:46.886181 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.886065 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83c72ce4-7b81-4396-9be6-7b5e71d1a3ff-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-h9x4m\" (UID: \"83c72ce4-7b81-4396-9be6-7b5e71d1a3ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" Aug 04 10:38:46.886552 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.886529 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee5a68b7-d654-4644-a54e-68191be69890-config\") pod \"service-ca-operator-784f97bcf6-hc4bh\" (UID: \"ee5a68b7-d654-4644-a54e-68191be69890\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" Aug 04 10:38:46.888095 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.888037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee5a68b7-d654-4644-a54e-68191be69890-serving-cert\") pod \"service-ca-operator-784f97bcf6-hc4bh\" (UID: \"ee5a68b7-d654-4644-a54e-68191be69890\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" Aug 04 10:38:46.896069 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.896046 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws4wg\" (UniqueName: \"kubernetes.io/projected/ee5a68b7-d654-4644-a54e-68191be69890-kube-api-access-ws4wg\") pod \"service-ca-operator-784f97bcf6-hc4bh\" (UID: \"ee5a68b7-d654-4644-a54e-68191be69890\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" Aug 04 10:38:46.947235 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.947199 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" Aug 04 10:38:46.986710 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.986675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83c72ce4-7b81-4396-9be6-7b5e71d1a3ff-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-h9x4m\" (UID: \"83c72ce4-7b81-4396-9be6-7b5e71d1a3ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" Aug 04 10:38:46.986883 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.986753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5gkx\" (UniqueName: \"kubernetes.io/projected/83c72ce4-7b81-4396-9be6-7b5e71d1a3ff-kube-api-access-z5gkx\") pod \"kube-storage-version-migrator-operator-7878f4b88c-h9x4m\" (UID: \"83c72ce4-7b81-4396-9be6-7b5e71d1a3ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" Aug 04 10:38:46.986883 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.986776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83c72ce4-7b81-4396-9be6-7b5e71d1a3ff-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-h9x4m\" (UID: \"83c72ce4-7b81-4396-9be6-7b5e71d1a3ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" Aug 04 10:38:46.987328 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.987306 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83c72ce4-7b81-4396-9be6-7b5e71d1a3ff-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-h9x4m\" (UID: \"83c72ce4-7b81-4396-9be6-7b5e71d1a3ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" Aug 04 10:38:46.988829 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.988802 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83c72ce4-7b81-4396-9be6-7b5e71d1a3ff-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-h9x4m\" (UID: \"83c72ce4-7b81-4396-9be6-7b5e71d1a3ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" Aug 04 10:38:46.997150 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:46.997102 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5gkx\" (UniqueName: \"kubernetes.io/projected/83c72ce4-7b81-4396-9be6-7b5e71d1a3ff-kube-api-access-z5gkx\") pod \"kube-storage-version-migrator-operator-7878f4b88c-h9x4m\" (UID: \"83c72ce4-7b81-4396-9be6-7b5e71d1a3ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" Aug 04 10:38:47.052939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:47.052904 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" Aug 04 10:38:47.070104 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:47.070076 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh"] Aug 04 10:38:47.073088 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:38:47.073059 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5a68b7_d654_4644_a54e_68191be69890.slice/crio-41ced5517a5f9a484624d303fc5ef804fdf6376a6d859b2e0d91b4e246385514 WatchSource:0}: Error finding container 41ced5517a5f9a484624d303fc5ef804fdf6376a6d859b2e0d91b4e246385514: Status 404 returned error can't find the container with id 41ced5517a5f9a484624d303fc5ef804fdf6376a6d859b2e0d91b4e246385514 Aug 04 10:38:47.176071 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:47.175977 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m"] Aug 04 10:38:47.178279 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:38:47.178254 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c72ce4_7b81_4396_9be6_7b5e71d1a3ff.slice/crio-6b0661dccc5f2f89e6d7b41b4be9aa7349b5bd416d3e40b6947603cef45b8596 WatchSource:0}: Error finding container 6b0661dccc5f2f89e6d7b41b4be9aa7349b5bd416d3e40b6947603cef45b8596: Status 404 returned error can't find the container with id 6b0661dccc5f2f89e6d7b41b4be9aa7349b5bd416d3e40b6947603cef45b8596 Aug 04 10:38:47.342298 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:47.342267 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:47.810229 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:47.810192 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" event={"ID":"83c72ce4-7b81-4396-9be6-7b5e71d1a3ff","Type":"ContainerStarted","Data":"6b0661dccc5f2f89e6d7b41b4be9aa7349b5bd416d3e40b6947603cef45b8596"} Aug 04 10:38:47.811369 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:47.811337 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" event={"ID":"ee5a68b7-d654-4644-a54e-68191be69890","Type":"ContainerStarted","Data":"41ced5517a5f9a484624d303fc5ef804fdf6376a6d859b2e0d91b4e246385514"} Aug 04 10:38:48.368664 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.368622 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ntr5t"] Aug 04 10:38:48.371096 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.371069 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:48.373687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.373660 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 10:38:48.373819 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.373739 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v8c5x\"" Aug 04 10:38:48.373819 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.373752 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 10:38:48.373819 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.373776 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 10:38:48.374443 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.374422 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 10:38:48.383085 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.383043 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ntr5t"] Aug 04 10:38:48.396157 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.396121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:48.397329 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.397299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qglcv\" (UID: \"1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" Aug 04 10:38:48.397523 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:48.397484 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 10:38:48.397611 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:48.397580 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls podName:1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a nodeName:}" failed. No retries permitted until 2026-08-04 10:38:52.397559762 +0000 UTC m=+121.495353139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qglcv" (UID: "1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a") : secret "samples-operator-tls" not found Aug 04 10:38:48.498263 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.498218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:48.498438 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.498270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:48.498438 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.498333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04538f9b-1297-4c97-a89c-d68686229e69-crio-socket\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:48.498438 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.498400 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04538f9b-1297-4c97-a89c-d68686229e69-data-volume\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:48.498438 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.498434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr2sl\" (UniqueName: \"kubernetes.io/projected/04538f9b-1297-4c97-a89c-d68686229e69-kube-api-access-gr2sl\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:48.502824 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.502784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:48.599268 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.599230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04538f9b-1297-4c97-a89c-d68686229e69-crio-socket\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:48.599493 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.599284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04538f9b-1297-4c97-a89c-d68686229e69-data-volume\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:48.599493 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.599315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr2sl\" (UniqueName: \"kubernetes.io/projected/04538f9b-1297-4c97-a89c-d68686229e69-kube-api-access-gr2sl\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:48.599493 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.599367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/04538f9b-1297-4c97-a89c-d68686229e69-crio-socket\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:48.599493 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.599484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:48.599493 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.599535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:48.599862 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:48.599670 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:38:48.599862 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.599719 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/04538f9b-1297-4c97-a89c-d68686229e69-data-volume\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:48.599862 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:48.599727 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls podName:04538f9b-1297-4c97-a89c-d68686229e69 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:49.099708044 +0000 UTC m=+118.197501419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ntr5t" (UID: "04538f9b-1297-4c97-a89c-d68686229e69") : secret "insights-runtime-extractor-tls" not found Aug 04 10:38:48.599862 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:48.599768 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:38:48.599862 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:48.599818 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm podName:04538f9b-1297-4c97-a89c-d68686229e69 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:49.099800518 +0000 UTC m=+118.197593892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ntr5t" (UID: "04538f9b-1297-4c97-a89c-d68686229e69") : configmap "kube-rbac-proxy" not found Aug 04 10:38:48.610566 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:48.610540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr2sl\" (UniqueName: \"kubernetes.io/projected/04538f9b-1297-4c97-a89c-d68686229e69-kube-api-access-gr2sl\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:49.104017 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:49.103980 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:49.104017 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:49.104028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:49.104519 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:49.104117 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:38:49.104519 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:49.104206 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm podName:04538f9b-1297-4c97-a89c-d68686229e69 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:50.104181445 +0000 UTC m=+119.201974821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ntr5t" (UID: "04538f9b-1297-4c97-a89c-d68686229e69") : configmap "kube-rbac-proxy" not found Aug 04 10:38:49.104519 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:49.104226 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:38:49.104519 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:49.104285 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls podName:04538f9b-1297-4c97-a89c-d68686229e69 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:50.104272873 +0000 UTC m=+119.202066245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ntr5t" (UID: "04538f9b-1297-4c97-a89c-d68686229e69") : secret "insights-runtime-extractor-tls" not found Aug 04 10:38:49.819450 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:49.819349 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" event={"ID":"83c72ce4-7b81-4396-9be6-7b5e71d1a3ff","Type":"ContainerStarted","Data":"3b721f8127e6a989c14ef1ec3d3e784cfcf981cf1e11a43ef8f7ba031f072fb1"} Aug 04 10:38:49.820913 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:49.820890 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" event={"ID":"ee5a68b7-d654-4644-a54e-68191be69890","Type":"ContainerStarted","Data":"09fd4aefcf4aa30211dc58b62f2460275c98cda53c6b31eb24e26c570b3250c3"} Aug 04 10:38:49.838256 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:49.838203 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" podStartSLOduration=1.592268459 podStartE2EDuration="3.838186541s" totalImagesPullingTime="2.245918082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:38:49.837538888 +0000 UTC m=+118.935332284" watchObservedRunningTime="2026-08-04 10:38:49.838186541 +0000 UTC m=+118.935979975" Aug 04 10:38:49.855330 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:49.855273 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" podStartSLOduration=1.5012208089999999 podStartE2EDuration="3.855253965s" totalImagesPullingTime="2.354033156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:38:49.854327666 +0000 UTC m=+118.952121061" watchObservedRunningTime="2026-08-04 10:38:49.855253965 +0000 UTC m=+118.953047363" Aug 04 10:38:50.112000 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:50.111964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:50.112423 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:50.112009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:50.112423 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:50.112113 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:38:50.112423 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:50.112194 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:38:50.112423 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:50.112197 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm podName:04538f9b-1297-4c97-a89c-d68686229e69 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:52.112172015 +0000 UTC m=+121.209965392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ntr5t" (UID: "04538f9b-1297-4c97-a89c-d68686229e69") : configmap "kube-rbac-proxy" not found Aug 04 10:38:50.112423 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:50.112253 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls podName:04538f9b-1297-4c97-a89c-d68686229e69 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:52.112240023 +0000 UTC m=+121.210033396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ntr5t" (UID: "04538f9b-1297-4c97-a89c-d68686229e69") : secret "insights-runtime-extractor-tls" not found Aug 04 10:38:50.514765 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:50.514685 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r9kvv_7b19bc6d-13f7-4db1-bc59-34eba05608ff/dns-node-resolver/0.log" Aug 04 10:38:50.540097 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:50.540061 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:50.909816 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:50.909787 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c9kzp_fa44517a-0e03-4dad-b3b0-79ee2001d9cd/node-ca/0.log" Aug 04 10:38:51.429700 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:51.429669 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-7pnrz"] Aug 04 10:38:51.432947 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:51.432923 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7pnrz" Aug 04 10:38:51.435754 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:51.435731 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-schpk\"" Aug 04 10:38:51.447121 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:51.447095 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-7pnrz"] Aug 04 10:38:51.522582 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:51.522546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgv2z\" (UniqueName: \"kubernetes.io/projected/4923de7b-acc4-4bfa-b74a-0271c6321dff-kube-api-access-tgv2z\") pod \"network-check-source-65bdbbff45-7pnrz\" (UID: \"4923de7b-acc4-4bfa-b74a-0271c6321dff\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7pnrz" Aug 04 10:38:51.622992 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:51.622959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgv2z\" (UniqueName: \"kubernetes.io/projected/4923de7b-acc4-4bfa-b74a-0271c6321dff-kube-api-access-tgv2z\") pod \"network-check-source-65bdbbff45-7pnrz\" (UID: \"4923de7b-acc4-4bfa-b74a-0271c6321dff\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7pnrz" Aug 04 10:38:51.632342 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:51.632318 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgv2z\" (UniqueName: \"kubernetes.io/projected/4923de7b-acc4-4bfa-b74a-0271c6321dff-kube-api-access-tgv2z\") pod \"network-check-source-65bdbbff45-7pnrz\" (UID: \"4923de7b-acc4-4bfa-b74a-0271c6321dff\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7pnrz" Aug 04 10:38:51.741779 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:51.741695 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7pnrz" Aug 04 10:38:51.861768 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:51.861738 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-7pnrz"] Aug 04 10:38:51.864867 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:38:51.864842 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4923de7b_acc4_4bfa_b74a_0271c6321dff.slice/crio-38e43652074ecde0e59dadda53932e390ada8137be03e9f182aeb92ab4d1ad1a WatchSource:0}: Error finding container 38e43652074ecde0e59dadda53932e390ada8137be03e9f182aeb92ab4d1ad1a: Status 404 returned error can't find the container with id 38e43652074ecde0e59dadda53932e390ada8137be03e9f182aeb92ab4d1ad1a Aug 04 10:38:52.109556 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.109523 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xwpcd_b86857b2-2acf-4829-b4fb-2d5ac4bf5236/serve-healthcheck-canary/0.log" Aug 04 10:38:52.126854 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.126824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:52.127019 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.126865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:52.127019 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:52.126987 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:38:52.127122 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:52.127045 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls podName:04538f9b-1297-4c97-a89c-d68686229e69 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:56.127027833 +0000 UTC m=+125.224821211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ntr5t" (UID: "04538f9b-1297-4c97-a89c-d68686229e69") : secret "insights-runtime-extractor-tls" not found Aug 04 10:38:52.127122 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:52.127047 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:38:52.127122 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:52.127096 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm podName:04538f9b-1297-4c97-a89c-d68686229e69 nodeName:}" failed. No retries permitted until 2026-08-04 10:38:56.127082674 +0000 UTC m=+125.224876050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ntr5t" (UID: "04538f9b-1297-4c97-a89c-d68686229e69") : configmap "kube-rbac-proxy" not found Aug 04 10:38:52.196160 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.196125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:52.284612 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.284585 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:52.430247 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.430163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:38:52.430247 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.430214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qglcv\" (UID: \"1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" Aug 04 10:38:52.430662 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:52.430318 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 10:38:52.430662 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:52.430374 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls podName:97e701ea-3b55-4c01-8e14-695a759a7ad8 nodeName:}" failed. No retries permitted until 2026-08-04 10:39:08.430359604 +0000 UTC m=+137.528152977 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-bfjqd" (UID: "97e701ea-3b55-4c01-8e14-695a759a7ad8") : secret "cluster-monitoring-operator-tls" not found Aug 04 10:38:52.430662 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:52.430384 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 10:38:52.430662 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:52.430424 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls podName:1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a nodeName:}" failed. No retries permitted until 2026-08-04 10:39:00.430412776 +0000 UTC m=+129.528206150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qglcv" (UID: "1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a") : secret "samples-operator-tls" not found Aug 04 10:38:52.531090 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.531048 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:52.531279 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.531135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:38:52.531279 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:52.531216 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle podName:50e09e74-c462-4d60-a61a-b281bf126d83 nodeName:}" failed. No retries permitted until 2026-08-04 10:39:08.531198215 +0000 UTC m=+137.628991588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle") pod "router-default-798f494f76-94bvh" (UID: "50e09e74-c462-4d60-a61a-b281bf126d83") : configmap references non-existent config key: service-ca.crt Aug 04 10:38:52.531378 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:52.531269 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 10:38:52.531378 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:52.531329 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs podName:50e09e74-c462-4d60-a61a-b281bf126d83 nodeName:}" failed. No retries permitted until 2026-08-04 10:39:08.531317319 +0000 UTC m=+137.629110697 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs") pod "router-default-798f494f76-94bvh" (UID: "50e09e74-c462-4d60-a61a-b281bf126d83") : secret "router-metrics-certs-default" not found Aug 04 10:38:52.832108 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.832077 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7pnrz" event={"ID":"4923de7b-acc4-4bfa-b74a-0271c6321dff","Type":"ContainerStarted","Data":"2bfe6932cec07fa8e884e73b1e067a5bd8ca25dff6c79c575b21068dfdd59fb4"} Aug 04 10:38:52.832300 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.832122 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7pnrz" event={"ID":"4923de7b-acc4-4bfa-b74a-0271c6321dff","Type":"ContainerStarted","Data":"38e43652074ecde0e59dadda53932e390ada8137be03e9f182aeb92ab4d1ad1a"} Aug 04 10:38:52.852750 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.852693 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-7pnrz" podStartSLOduration=1.8526766700000001 podStartE2EDuration="1.85267667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:38:52.852113135 +0000 UTC m=+121.949906533" watchObservedRunningTime="2026-08-04 10:38:52.85267667 +0000 UTC m=+121.950470113" Aug 04 10:38:52.873246 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.873215 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-hjlch"] Aug 04 10:38:52.875339 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.875321 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-hjlch" Aug 04 10:38:52.879585 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.879567 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5s6z5\"" Aug 04 10:38:52.879700 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.879660 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Aug 04 10:38:52.879700 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.879686 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Aug 04 10:38:52.879838 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.879825 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Aug 04 10:38:52.879951 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.879937 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Aug 04 10:38:52.887319 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.887297 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-hjlch"] Aug 04 10:38:52.934967 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:52.934938 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:53.000859 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.000809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:53.036358 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.036323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfsql\" (UniqueName: \"kubernetes.io/projected/786f544a-99fa-4cc0-b77e-81e34ed19f68-kube-api-access-jfsql\") pod \"service-ca-98c88fbb5-hjlch\" (UID: \"786f544a-99fa-4cc0-b77e-81e34ed19f68\") " pod="openshift-service-ca/service-ca-98c88fbb5-hjlch" Aug 04 10:38:53.036358 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.036360 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/786f544a-99fa-4cc0-b77e-81e34ed19f68-signing-cabundle\") pod \"service-ca-98c88fbb5-hjlch\" (UID: \"786f544a-99fa-4cc0-b77e-81e34ed19f68\") " pod="openshift-service-ca/service-ca-98c88fbb5-hjlch" Aug 04 10:38:53.036563 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.036475 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/786f544a-99fa-4cc0-b77e-81e34ed19f68-signing-key\") pod \"service-ca-98c88fbb5-hjlch\" (UID: \"786f544a-99fa-4cc0-b77e-81e34ed19f68\") " pod="openshift-service-ca/service-ca-98c88fbb5-hjlch" Aug 04 10:38:53.137385 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.137300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfsql\" (UniqueName: \"kubernetes.io/projected/786f544a-99fa-4cc0-b77e-81e34ed19f68-kube-api-access-jfsql\") pod \"service-ca-98c88fbb5-hjlch\" (UID: \"786f544a-99fa-4cc0-b77e-81e34ed19f68\") " pod="openshift-service-ca/service-ca-98c88fbb5-hjlch" Aug 04 10:38:53.137385 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.137337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/786f544a-99fa-4cc0-b77e-81e34ed19f68-signing-cabundle\") pod \"service-ca-98c88fbb5-hjlch\" (UID: \"786f544a-99fa-4cc0-b77e-81e34ed19f68\") " pod="openshift-service-ca/service-ca-98c88fbb5-hjlch" Aug 04 10:38:53.137587 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.137401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/786f544a-99fa-4cc0-b77e-81e34ed19f68-signing-key\") pod \"service-ca-98c88fbb5-hjlch\" (UID: \"786f544a-99fa-4cc0-b77e-81e34ed19f68\") " pod="openshift-service-ca/service-ca-98c88fbb5-hjlch" Aug 04 10:38:53.138078 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.138052 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/786f544a-99fa-4cc0-b77e-81e34ed19f68-signing-cabundle\") pod \"service-ca-98c88fbb5-hjlch\" (UID: \"786f544a-99fa-4cc0-b77e-81e34ed19f68\") " pod="openshift-service-ca/service-ca-98c88fbb5-hjlch" Aug 04 10:38:53.139637 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.139620 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/786f544a-99fa-4cc0-b77e-81e34ed19f68-signing-key\") pod \"service-ca-98c88fbb5-hjlch\" (UID: \"786f544a-99fa-4cc0-b77e-81e34ed19f68\") " pod="openshift-service-ca/service-ca-98c88fbb5-hjlch" Aug 04 10:38:53.146320 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.146299 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfsql\" (UniqueName: \"kubernetes.io/projected/786f544a-99fa-4cc0-b77e-81e34ed19f68-kube-api-access-jfsql\") pod \"service-ca-98c88fbb5-hjlch\" (UID: \"786f544a-99fa-4cc0-b77e-81e34ed19f68\") " pod="openshift-service-ca/service-ca-98c88fbb5-hjlch" Aug 04 10:38:53.184361 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.184333 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-hjlch" Aug 04 10:38:53.328078 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.328055 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-hjlch"] Aug 04 10:38:53.330195 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:38:53.330169 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786f544a_99fa_4cc0_b77e_81e34ed19f68.slice/crio-a7be499c6921464a52593282db4756bf06951138e0ab8bf2eb8f42dbcb61dfb1 WatchSource:0}: Error finding container a7be499c6921464a52593282db4756bf06951138e0ab8bf2eb8f42dbcb61dfb1: Status 404 returned error can't find the container with id a7be499c6921464a52593282db4756bf06951138e0ab8bf2eb8f42dbcb61dfb1 Aug 04 10:38:53.353800 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.353780 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:53.836479 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.836439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-hjlch" event={"ID":"786f544a-99fa-4cc0-b77e-81e34ed19f68","Type":"ContainerStarted","Data":"bea7104ea00c43609172d5bc9bdc9a2cc027d89e94f2f31068706bcb0e34f91d"} Aug 04 10:38:53.836891 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.836495 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-hjlch" event={"ID":"786f544a-99fa-4cc0-b77e-81e34ed19f68","Type":"ContainerStarted","Data":"a7be499c6921464a52593282db4756bf06951138e0ab8bf2eb8f42dbcb61dfb1"} Aug 04 10:38:53.856280 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.856207 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-hjlch" podStartSLOduration=1.856186975 podStartE2EDuration="1.856186975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:38:53.855754282 +0000 UTC m=+122.953547686" watchObservedRunningTime="2026-08-04 10:38:53.856186975 +0000 UTC m=+122.953980371" Aug 04 10:38:53.991408 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:53.991375 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:54.764731 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:54.764661 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:54.986410 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:54.986362 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:55.088223 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:55.088194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:55.281381 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:55.281341 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:55.399838 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:55.399748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:55.512324 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:55.512294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:56.163799 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:56.163756 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:56.163799 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:56.163797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:38:56.164222 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:56.163873 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:38:56.164222 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:56.163890 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:38:56.164222 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:56.163939 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm podName:04538f9b-1297-4c97-a89c-d68686229e69 nodeName:}" failed. No retries permitted until 2026-08-04 10:39:04.163924507 +0000 UTC m=+133.261717883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ntr5t" (UID: "04538f9b-1297-4c97-a89c-d68686229e69") : configmap "kube-rbac-proxy" not found Aug 04 10:38:56.164222 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:38:56.164001 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls podName:04538f9b-1297-4c97-a89c-d68686229e69 nodeName:}" failed. No retries permitted until 2026-08-04 10:39:04.163987966 +0000 UTC m=+133.261781339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ntr5t" (UID: "04538f9b-1297-4c97-a89c-d68686229e69") : secret "insights-runtime-extractor-tls" not found Aug 04 10:38:56.261159 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:56.261123 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:56.870569 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:56.870535 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:57.066192 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:57.066160 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:57.269669 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:57.269574 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:57.678798 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:57.678765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:58.281141 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:58.281084 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:58.475793 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:58.475742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:58.665329 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:58.665277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:58.740757 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:58.740722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:59.068941 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:59.068837 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:59.144724 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:59.144687 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:59.271015 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:59.270986 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:38:59.681953 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:38:59.681905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:00.469536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:00.469467 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:00.498756 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:00.498722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qglcv\" (UID: \"1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" Aug 04 10:39:00.501418 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:00.501390 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qglcv\" (UID: \"1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" Aug 04 10:39:00.564811 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:00.564779 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fjj5q\"" Aug 04 10:39:00.572892 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:00.572871 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" Aug 04 10:39:00.675387 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:00.675355 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:00.703205 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:00.703165 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv"] Aug 04 10:39:00.866603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:00.866572 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" event={"ID":"1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a","Type":"ContainerStarted","Data":"5892c0e97735cf1656a2bd1cd8aa25695d43614d1a4fb7f6cc612b0e26289517"} Aug 04 10:39:01.305245 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:01.305194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:39:01.305414 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:39:01.305380 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 04 10:39:01.305490 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:39:01.305466 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs podName:f67d2ed4-7812-45de-95c0-32cf86d9eef2 nodeName:}" failed. No retries permitted until 2026-08-04 10:41:03.305444423 +0000 UTC m=+252.403237803 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs") pod "network-metrics-daemon-ldmtm" (UID: "f67d2ed4-7812-45de-95c0-32cf86d9eef2") : secret "metrics-daemon-secret" not found Aug 04 10:39:02.874923 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:02.874850 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" event={"ID":"1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a","Type":"ContainerStarted","Data":"0d0012bae3e121a4f478a83bde75b91977632fc72c0eea65def42fe2339ab3b0"} Aug 04 10:39:02.875275 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:02.874937 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" event={"ID":"1909dde2-4b5b-4f9f-ae1a-77bc71d5f12a","Type":"ContainerStarted","Data":"ff65daa4f1e26aa4da3a270273b2c5b8ade77e433cfcc011f2bea374ea979b94"} Aug 04 10:39:02.897986 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:02.897933 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qglcv" podStartSLOduration=17.0501673 podStartE2EDuration="18.897917287s" totalImagesPullingTime="1.847749987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:39:02.89757645 +0000 UTC m=+131.995369846" watchObservedRunningTime="2026-08-04 10:39:02.897917287 +0000 UTC m=+131.995710681" Aug 04 10:39:03.056465 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:03.056429 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:03.125308 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:03.125233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:03.259460 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:03.259427 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:03.481780 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:03.481692 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:03.875818 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:03.875779 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:04.082644 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:04.082607 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:04.234386 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:04.234293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:39:04.234386 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:04.234332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:39:04.234637 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:39:04.234455 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:39:04.234637 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:39:04.234533 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls podName:04538f9b-1297-4c97-a89c-d68686229e69 nodeName:}" failed. No retries permitted until 2026-08-04 10:39:20.234520496 +0000 UTC m=+149.332313883 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ntr5t" (UID: "04538f9b-1297-4c97-a89c-d68686229e69") : secret "insights-runtime-extractor-tls" not found Aug 04 10:39:04.234954 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:04.234931 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/04538f9b-1297-4c97-a89c-d68686229e69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:39:04.705190 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:04.705156 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:04.823612 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:04.823579 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:04.910335 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:04.910299 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:05.212511 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:05.212477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:05.312203 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:05.312170 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:05.482294 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:05.482219 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:08.469745 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:08.469699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:39:08.472041 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:08.472014 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97e701ea-3b55-4c01-8e14-695a759a7ad8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-bfjqd\" (UID: \"97e701ea-3b55-4c01-8e14-695a759a7ad8\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:39:08.570848 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:08.570797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:39:08.571053 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:08.570881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:39:08.571443 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:08.571421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e09e74-c462-4d60-a61a-b281bf126d83-service-ca-bundle\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:39:08.573092 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:08.573075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e09e74-c462-4d60-a61a-b281bf126d83-metrics-certs\") pod \"router-default-798f494f76-94bvh\" (UID: \"50e09e74-c462-4d60-a61a-b281bf126d83\") " pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:39:08.721728 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:08.721632 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wshwv\"" Aug 04 10:39:08.729768 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:08.729743 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" Aug 04 10:39:08.828332 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:08.828303 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fs7n7\"" Aug 04 10:39:08.836183 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:08.836157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:39:08.852199 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:08.852174 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd"] Aug 04 10:39:08.854284 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:39:08.854256 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e701ea_3b55_4c01_8e14_695a759a7ad8.slice/crio-2f918daec32c7092ef310c71647dad696dbf50946b6d59ad7f905e5a3884e1c4 WatchSource:0}: Error finding container 2f918daec32c7092ef310c71647dad696dbf50946b6d59ad7f905e5a3884e1c4: Status 404 returned error can't find the container with id 2f918daec32c7092ef310c71647dad696dbf50946b6d59ad7f905e5a3884e1c4 Aug 04 10:39:08.897028 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:08.896993 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" event={"ID":"97e701ea-3b55-4c01-8e14-695a759a7ad8","Type":"ContainerStarted","Data":"2f918daec32c7092ef310c71647dad696dbf50946b6d59ad7f905e5a3884e1c4"} Aug 04 10:39:08.963048 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:08.963015 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-798f494f76-94bvh"] Aug 04 10:39:08.965609 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:39:08.965586 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e09e74_c462_4d60_a61a_b281bf126d83.slice/crio-f0cfa36fe4bee27a8c01868e4727e3c5e7ffb1c5a904625732a5076b7dee8db8 WatchSource:0}: Error finding container f0cfa36fe4bee27a8c01868e4727e3c5e7ffb1c5a904625732a5076b7dee8db8: Status 404 returned error can't find the container with id f0cfa36fe4bee27a8c01868e4727e3c5e7ffb1c5a904625732a5076b7dee8db8 Aug 04 10:39:09.902137 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:09.902100 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-798f494f76-94bvh" event={"ID":"50e09e74-c462-4d60-a61a-b281bf126d83","Type":"ContainerStarted","Data":"fba7f1b81770f94ad31689283a31c0d8c5518d19f28ff2cf14fed75a92f7a65e"} Aug 04 10:39:09.902591 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:09.902157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-798f494f76-94bvh" event={"ID":"50e09e74-c462-4d60-a61a-b281bf126d83","Type":"ContainerStarted","Data":"f0cfa36fe4bee27a8c01868e4727e3c5e7ffb1c5a904625732a5076b7dee8db8"} Aug 04 10:39:09.929758 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:09.929699 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-798f494f76-94bvh" podStartSLOduration=33.929677846 podStartE2EDuration="33.929677846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:39:09.929020896 +0000 UTC m=+139.026814292" watchObservedRunningTime="2026-08-04 10:39:09.929677846 +0000 UTC m=+139.027471222" Aug 04 10:39:10.837178 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:10.837151 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:39:10.839921 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:10.839897 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:39:10.908065 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:10.908026 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" event={"ID":"97e701ea-3b55-4c01-8e14-695a759a7ad8","Type":"ContainerStarted","Data":"9d3a45c747864335fe06e53e9eb8a85deb0b4f0a308c0e1983e69a3f169e22c8"} Aug 04 10:39:10.908567 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:10.908540 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:39:10.909826 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:10.909802 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-798f494f76-94bvh" Aug 04 10:39:10.930968 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:10.930904 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-bfjqd" podStartSLOduration=33.023009722 podStartE2EDuration="34.930882621s" totalImagesPullingTime="1.907872899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:39:10.930577543 +0000 UTC m=+140.028370940" watchObservedRunningTime="2026-08-04 10:39:10.930882621 +0000 UTC m=+140.028676018" Aug 04 10:39:11.129759 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:11.129724 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:11.222325 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:11.222291 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:11.321026 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:11.320994 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:11.457117 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:11.457033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:11.856251 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:11.856223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:11.945897 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:11.945869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:12.078726 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:12.078697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:12.326358 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:12.326276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:12.401752 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:12.401720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:12.498800 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:12.498766 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:12.566296 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:12.566264 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:13.526558 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:13.526526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:13.589203 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:13.589165 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:13.717556 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:13.717521 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:13.816403 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:13.816333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:16.099483 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.099453 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-9l97v"] Aug 04 10:39:16.102647 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.102630 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-9l97v" Aug 04 10:39:16.106713 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.106692 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Aug 04 10:39:16.106713 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.106703 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8tf9g\"" Aug 04 10:39:16.106886 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.106709 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Aug 04 10:39:16.116376 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.116355 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-9l97v"] Aug 04 10:39:16.134591 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.134565 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6737f3c7-d910-4cb5-a73e-3eea0bab64e6-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9l97v\" (UID: \"6737f3c7-d910-4cb5-a73e-3eea0bab64e6\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9l97v" Aug 04 10:39:16.134727 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.134602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6737f3c7-d910-4cb5-a73e-3eea0bab64e6-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-9l97v\" (UID: \"6737f3c7-d910-4cb5-a73e-3eea0bab64e6\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9l97v" Aug 04 10:39:16.190763 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.190732 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7t7t5"] Aug 04 10:39:16.193816 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.193785 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7t7t5" Aug 04 10:39:16.198788 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.198765 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Aug 04 10:39:16.198920 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.198804 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nh8t4\"" Aug 04 10:39:16.216417 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.216381 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7t7t5"] Aug 04 10:39:16.229137 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.229110 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84bbdfb5b5-grjg4"] Aug 04 10:39:16.232019 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.232003 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.235650 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.235624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6737f3c7-d910-4cb5-a73e-3eea0bab64e6-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9l97v\" (UID: \"6737f3c7-d910-4cb5-a73e-3eea0bab64e6\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9l97v" Aug 04 10:39:16.235756 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.235672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6737f3c7-d910-4cb5-a73e-3eea0bab64e6-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-9l97v\" (UID: \"6737f3c7-d910-4cb5-a73e-3eea0bab64e6\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9l97v" Aug 04 10:39:16.235833 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.235755 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/763d8cd2-4479-4fb4-8107-af81b4bdc56b-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-7t7t5\" (UID: \"763d8cd2-4479-4fb4-8107-af81b4bdc56b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7t7t5" Aug 04 10:39:16.236338 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.236321 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6737f3c7-d910-4cb5-a73e-3eea0bab64e6-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-9l97v\" (UID: \"6737f3c7-d910-4cb5-a73e-3eea0bab64e6\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9l97v" Aug 04 10:39:16.238121 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.238102 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6737f3c7-d910-4cb5-a73e-3eea0bab64e6-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-9l97v\" (UID: \"6737f3c7-d910-4cb5-a73e-3eea0bab64e6\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-9l97v" Aug 04 10:39:16.240453 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.240435 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 10:39:16.241389 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.241359 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 10:39:16.242041 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.242017 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xjf6f\"" Aug 04 10:39:16.243530 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.243484 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 10:39:16.249432 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.249412 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 10:39:16.252809 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.252784 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84bbdfb5b5-grjg4"] Aug 04 10:39:16.299888 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.299858 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:16.336800 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.336774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853f18a6-1400-45ea-964a-fbb87c0a67bc-registry-tls\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.336948 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.336813 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/853f18a6-1400-45ea-964a-fbb87c0a67bc-bound-sa-token\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.336948 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.336835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5df\" (UniqueName: \"kubernetes.io/projected/853f18a6-1400-45ea-964a-fbb87c0a67bc-kube-api-access-xg5df\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.336948 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.336895 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/853f18a6-1400-45ea-964a-fbb87c0a67bc-image-registry-private-configuration\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.336948 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.336917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/853f18a6-1400-45ea-964a-fbb87c0a67bc-ca-trust-extracted\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.336948 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.336935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/853f18a6-1400-45ea-964a-fbb87c0a67bc-registry-certificates\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.337104 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.336992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/763d8cd2-4479-4fb4-8107-af81b4bdc56b-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-7t7t5\" (UID: \"763d8cd2-4479-4fb4-8107-af81b4bdc56b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7t7t5" Aug 04 10:39:16.337104 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.337036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/853f18a6-1400-45ea-964a-fbb87c0a67bc-trusted-ca\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.337104 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.337053 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/853f18a6-1400-45ea-964a-fbb87c0a67bc-installation-pull-secrets\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.339470 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.339436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/763d8cd2-4479-4fb4-8107-af81b4bdc56b-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-7t7t5\" (UID: \"763d8cd2-4479-4fb4-8107-af81b4bdc56b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7t7t5" Aug 04 10:39:16.399946 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.399917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:16.411804 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.411777 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-9l97v" Aug 04 10:39:16.437643 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.437610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/853f18a6-1400-45ea-964a-fbb87c0a67bc-trusted-ca\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.437643 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.437640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/853f18a6-1400-45ea-964a-fbb87c0a67bc-installation-pull-secrets\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.437878 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.437659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853f18a6-1400-45ea-964a-fbb87c0a67bc-registry-tls\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.437878 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.437685 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/853f18a6-1400-45ea-964a-fbb87c0a67bc-bound-sa-token\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.437878 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.437835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5df\" (UniqueName: \"kubernetes.io/projected/853f18a6-1400-45ea-964a-fbb87c0a67bc-kube-api-access-xg5df\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.438033 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.437885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/853f18a6-1400-45ea-964a-fbb87c0a67bc-image-registry-private-configuration\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.438033 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.437915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/853f18a6-1400-45ea-964a-fbb87c0a67bc-ca-trust-extracted\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.438033 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.437943 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/853f18a6-1400-45ea-964a-fbb87c0a67bc-registry-certificates\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.438529 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.438479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/853f18a6-1400-45ea-964a-fbb87c0a67bc-ca-trust-extracted\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.438778 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.438752 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/853f18a6-1400-45ea-964a-fbb87c0a67bc-trusted-ca\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.438890 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.438824 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/853f18a6-1400-45ea-964a-fbb87c0a67bc-registry-certificates\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.440322 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.440296 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/853f18a6-1400-45ea-964a-fbb87c0a67bc-registry-tls\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.440445 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.440425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/853f18a6-1400-45ea-964a-fbb87c0a67bc-image-registry-private-configuration\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.441010 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.440986 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/853f18a6-1400-45ea-964a-fbb87c0a67bc-installation-pull-secrets\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.447464 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.447442 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/853f18a6-1400-45ea-964a-fbb87c0a67bc-bound-sa-token\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.448234 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.448211 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5df\" (UniqueName: \"kubernetes.io/projected/853f18a6-1400-45ea-964a-fbb87c0a67bc-kube-api-access-xg5df\") pod \"image-registry-84bbdfb5b5-grjg4\" (UID: \"853f18a6-1400-45ea-964a-fbb87c0a67bc\") " pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.502851 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.502814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7t7t5" Aug 04 10:39:16.544728 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.544699 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-9l97v"] Aug 04 10:39:16.548442 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:39:16.548416 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6737f3c7_d910_4cb5_a73e_3eea0bab64e6.slice/crio-e307dcea8fead6015f91a5eac8221eca6db948991e085a1dd81a62daef372bd5 WatchSource:0}: Error finding container e307dcea8fead6015f91a5eac8221eca6db948991e085a1dd81a62daef372bd5: Status 404 returned error can't find the container with id e307dcea8fead6015f91a5eac8221eca6db948991e085a1dd81a62daef372bd5 Aug 04 10:39:16.549762 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.549729 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.661630 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.661588 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7t7t5"] Aug 04 10:39:16.665708 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:39:16.665679 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763d8cd2_4479_4fb4_8107_af81b4bdc56b.slice/crio-c456f923320801e94e42cb5d001705d5647224e306dfe989d9ba1879c02b4669 WatchSource:0}: Error finding container c456f923320801e94e42cb5d001705d5647224e306dfe989d9ba1879c02b4669: Status 404 returned error can't find the container with id c456f923320801e94e42cb5d001705d5647224e306dfe989d9ba1879c02b4669 Aug 04 10:39:16.710634 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.710602 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84bbdfb5b5-grjg4"] Aug 04 10:39:16.714277 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:39:16.714249 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853f18a6_1400_45ea_964a_fbb87c0a67bc.slice/crio-9ecbf557cd9a1bc45a14388347ad45247a216ef8502ac0eacd80279bb9279140 WatchSource:0}: Error finding container 9ecbf557cd9a1bc45a14388347ad45247a216ef8502ac0eacd80279bb9279140: Status 404 returned error can't find the container with id 9ecbf557cd9a1bc45a14388347ad45247a216ef8502ac0eacd80279bb9279140 Aug 04 10:39:16.928438 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.928341 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-9l97v" event={"ID":"6737f3c7-d910-4cb5-a73e-3eea0bab64e6","Type":"ContainerStarted","Data":"e307dcea8fead6015f91a5eac8221eca6db948991e085a1dd81a62daef372bd5"} Aug 04 10:39:16.929423 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.929397 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7t7t5" event={"ID":"763d8cd2-4479-4fb4-8107-af81b4bdc56b","Type":"ContainerStarted","Data":"c456f923320801e94e42cb5d001705d5647224e306dfe989d9ba1879c02b4669"} Aug 04 10:39:16.930834 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.930814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" event={"ID":"853f18a6-1400-45ea-964a-fbb87c0a67bc","Type":"ContainerStarted","Data":"a4fe8d4b0a3dbf73e890dda691b2d931ef38736b8d68cfbe66a9ca215da36c8f"} Aug 04 10:39:16.930938 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.930849 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" event={"ID":"853f18a6-1400-45ea-964a-fbb87c0a67bc","Type":"ContainerStarted","Data":"9ecbf557cd9a1bc45a14388347ad45247a216ef8502ac0eacd80279bb9279140"} Aug 04 10:39:16.930979 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.930952 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:16.956641 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:16.956588 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" podStartSLOduration=0.956571688 podStartE2EDuration="956.571688ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:39:16.956074851 +0000 UTC m=+146.053868237" watchObservedRunningTime="2026-08-04 10:39:16.956571688 +0000 UTC m=+146.054365083" Aug 04 10:39:18.938314 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:18.938278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7t7t5" event={"ID":"763d8cd2-4479-4fb4-8107-af81b4bdc56b","Type":"ContainerStarted","Data":"c21de75c42090b1c96ec884770e1f576e995b633b442166d2d61f53091f49b26"} Aug 04 10:39:18.938780 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:18.938429 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7t7t5" Aug 04 10:39:18.939843 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:18.939818 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-9l97v" event={"ID":"6737f3c7-d910-4cb5-a73e-3eea0bab64e6","Type":"ContainerStarted","Data":"5d6f917fc0ce6bd8f8a0ba3823e83adbd00cad27084a8041cd936c3e45ec9b7a"} Aug 04 10:39:18.943333 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:18.943313 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7t7t5" Aug 04 10:39:18.956910 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:18.956863 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-7t7t5" podStartSLOduration=1.694300028 podStartE2EDuration="2.956846943s" totalImagesPullingTime="1.262546915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:39:18.95525682 +0000 UTC m=+148.053050211" watchObservedRunningTime="2026-08-04 10:39:18.956846943 +0000 UTC m=+148.054640368" Aug 04 10:39:18.972933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:18.972878 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-9l97v" podStartSLOduration=1.591050941 podStartE2EDuration="2.972864879s" totalImagesPullingTime="1.381813938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:39:18.971535332 +0000 UTC m=+148.069328727" watchObservedRunningTime="2026-08-04 10:39:18.972864879 +0000 UTC m=+148.070658274" Aug 04 10:39:19.088686 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.088650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:19.174207 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.174176 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:19.322942 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.322867 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:19.371853 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.371820 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-hm4tg"] Aug 04 10:39:19.375240 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.375225 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:19.377682 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.377659 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Aug 04 10:39:19.377809 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.377773 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l9h2m\"" Aug 04 10:39:19.378654 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.378633 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 10:39:19.378794 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.378673 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Aug 04 10:39:19.387802 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.387779 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-hm4tg"] Aug 04 10:39:19.419189 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.419167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:19.467226 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.467188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5be3683e-4567-4763-9475-960ebec0e2ec-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-hm4tg\" (UID: \"5be3683e-4567-4763-9475-960ebec0e2ec\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:19.467226 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.467234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5be3683e-4567-4763-9475-960ebec0e2ec-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-hm4tg\" (UID: \"5be3683e-4567-4763-9475-960ebec0e2ec\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:19.467484 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.467389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5be3683e-4567-4763-9475-960ebec0e2ec-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-hm4tg\" (UID: \"5be3683e-4567-4763-9475-960ebec0e2ec\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:19.467484 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.467446 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8xnn\" (UniqueName: \"kubernetes.io/projected/5be3683e-4567-4763-9475-960ebec0e2ec-kube-api-access-l8xnn\") pod \"prometheus-operator-758d7cb874-hm4tg\" (UID: \"5be3683e-4567-4763-9475-960ebec0e2ec\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:19.568248 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.568207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5be3683e-4567-4763-9475-960ebec0e2ec-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-hm4tg\" (UID: \"5be3683e-4567-4763-9475-960ebec0e2ec\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:19.568424 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.568261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8xnn\" (UniqueName: \"kubernetes.io/projected/5be3683e-4567-4763-9475-960ebec0e2ec-kube-api-access-l8xnn\") pod \"prometheus-operator-758d7cb874-hm4tg\" (UID: \"5be3683e-4567-4763-9475-960ebec0e2ec\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:19.568424 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.568294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5be3683e-4567-4763-9475-960ebec0e2ec-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-hm4tg\" (UID: \"5be3683e-4567-4763-9475-960ebec0e2ec\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:19.568424 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.568328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5be3683e-4567-4763-9475-960ebec0e2ec-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-hm4tg\" (UID: \"5be3683e-4567-4763-9475-960ebec0e2ec\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:19.568424 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:39:19.568367 2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Aug 04 10:39:19.568689 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:39:19.568435 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5be3683e-4567-4763-9475-960ebec0e2ec-prometheus-operator-tls podName:5be3683e-4567-4763-9475-960ebec0e2ec nodeName:}" failed. No retries permitted until 2026-08-04 10:39:20.068416327 +0000 UTC m=+149.166209719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/5be3683e-4567-4763-9475-960ebec0e2ec-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-hm4tg" (UID: "5be3683e-4567-4763-9475-960ebec0e2ec") : secret "prometheus-operator-tls" not found Aug 04 10:39:19.569118 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.569097 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5be3683e-4567-4763-9475-960ebec0e2ec-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-hm4tg\" (UID: \"5be3683e-4567-4763-9475-960ebec0e2ec\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:19.571052 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.571025 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5be3683e-4567-4763-9475-960ebec0e2ec-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-hm4tg\" (UID: \"5be3683e-4567-4763-9475-960ebec0e2ec\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:19.578015 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.577956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8xnn\" (UniqueName: \"kubernetes.io/projected/5be3683e-4567-4763-9475-960ebec0e2ec-kube-api-access-l8xnn\") pod \"prometheus-operator-758d7cb874-hm4tg\" (UID: \"5be3683e-4567-4763-9475-960ebec0e2ec\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:19.733729 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:19.733690 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:20.071470 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:20.071413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5be3683e-4567-4763-9475-960ebec0e2ec-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-hm4tg\" (UID: \"5be3683e-4567-4763-9475-960ebec0e2ec\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:20.073951 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:20.073921 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5be3683e-4567-4763-9475-960ebec0e2ec-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-hm4tg\" (UID: \"5be3683e-4567-4763-9475-960ebec0e2ec\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:20.146761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:20.146723 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:20.221539 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:20.221485 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:20.273232 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:20.273194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:39:20.275627 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:20.275601 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/04538f9b-1297-4c97-a89c-d68686229e69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntr5t\" (UID: \"04538f9b-1297-4c97-a89c-d68686229e69\") " pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:39:20.284561 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:20.284533 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" Aug 04 10:39:20.415726 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:20.415534 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-hm4tg"] Aug 04 10:39:20.418223 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:39:20.418193 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be3683e_4567_4763_9475_960ebec0e2ec.slice/crio-c8d2679c72ace166fe670eba1d3f5f6ef7acd64ef48c3d551497c39d3e11d2f5 WatchSource:0}: Error finding container c8d2679c72ace166fe670eba1d3f5f6ef7acd64ef48c3d551497c39d3e11d2f5: Status 404 returned error can't find the container with id c8d2679c72ace166fe670eba1d3f5f6ef7acd64ef48c3d551497c39d3e11d2f5 Aug 04 10:39:20.484808 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:20.484782 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v8c5x\"" Aug 04 10:39:20.492830 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:20.492809 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ntr5t" Aug 04 10:39:20.637375 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:20.637348 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ntr5t"] Aug 04 10:39:20.640305 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:39:20.640276 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04538f9b_1297_4c97_a89c_d68686229e69.slice/crio-3a664e165221a93a0d357a74df4d5813eb7ddde047cbc53a23d8cbbf2ae69bfe WatchSource:0}: Error finding container 3a664e165221a93a0d357a74df4d5813eb7ddde047cbc53a23d8cbbf2ae69bfe: Status 404 returned error can't find the container with id 3a664e165221a93a0d357a74df4d5813eb7ddde047cbc53a23d8cbbf2ae69bfe Aug 04 10:39:20.947495 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:20.947387 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntr5t" event={"ID":"04538f9b-1297-4c97-a89c-d68686229e69","Type":"ContainerStarted","Data":"bc2251c51c15c38aa5c3e5724104754810712bce1a35e922f531a9772afb4e6a"} Aug 04 10:39:20.947495 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:20.947457 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntr5t" event={"ID":"04538f9b-1297-4c97-a89c-d68686229e69","Type":"ContainerStarted","Data":"3a664e165221a93a0d357a74df4d5813eb7ddde047cbc53a23d8cbbf2ae69bfe"} Aug 04 10:39:20.948590 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:20.948566 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" event={"ID":"5be3683e-4567-4763-9475-960ebec0e2ec","Type":"ContainerStarted","Data":"c8d2679c72ace166fe670eba1d3f5f6ef7acd64ef48c3d551497c39d3e11d2f5"} Aug 04 10:39:21.953078 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:21.953043 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntr5t" event={"ID":"04538f9b-1297-4c97-a89c-d68686229e69","Type":"ContainerStarted","Data":"1688a382603153d8f30dec869dbb4786868f5d4b81d3904e546e49b23064135f"} Aug 04 10:39:22.959184 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:22.959148 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntr5t" event={"ID":"04538f9b-1297-4c97-a89c-d68686229e69","Type":"ContainerStarted","Data":"393ee965c102bed02eda40389d705749453f0bab02c0bbb503a7ccc859fcd575"} Aug 04 10:39:22.976855 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:22.976787 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ntr5t" podStartSLOduration=32.880631542 podStartE2EDuration="34.976767978s" totalImagesPullingTime="2.096136436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:38:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:39:22.976708138 +0000 UTC m=+152.074501575" watchObservedRunningTime="2026-08-04 10:39:22.976767978 +0000 UTC m=+152.074561373" Aug 04 10:39:24.342881 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:24.342847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:24.542426 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:24.542392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:24.610729 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:24.610648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:24.967746 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:24.967660 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" event={"ID":"5be3683e-4567-4763-9475-960ebec0e2ec","Type":"ContainerStarted","Data":"ca871c8256092065d8a8c4c4a0de110a54f2543ab984698b052052de1deff246"} Aug 04 10:39:24.967746 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:24.967713 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" event={"ID":"5be3683e-4567-4763-9475-960ebec0e2ec","Type":"ContainerStarted","Data":"bdeef56c3d8b32a97746f1816d4e86a91fa2152c1840b0b631fa8b210b997d12"} Aug 04 10:39:24.985867 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:24.985811 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-hm4tg" podStartSLOduration=2.336782613 podStartE2EDuration="5.985793028s" totalImagesPullingTime="3.649010415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:39:24.984611099 +0000 UTC m=+154.082404485" watchObservedRunningTime="2026-08-04 10:39:24.985793028 +0000 UTC m=+154.083586421" Aug 04 10:39:26.491202 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.491169 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:26.591132 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.591096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:26.707824 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.707739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:26.818805 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.818716 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-blsdj"] Aug 04 10:39:26.820267 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.820207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:26.822750 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.822731 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:26.825448 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.825427 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 10:39:26.825874 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.825854 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 10:39:26.826015 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.825922 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 10:39:26.827849 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.827831 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8vcbw\"" Aug 04 10:39:26.933672 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.933633 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-tls\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:26.933672 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.933673 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-textfile\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:26.933940 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.933700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:26.933940 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.933726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/380264cd-6db8-4b2a-bd74-a78aed669c40-metrics-client-ca\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:26.933940 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.933755 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzqw8\" (UniqueName: \"kubernetes.io/projected/380264cd-6db8-4b2a-bd74-a78aed669c40-kube-api-access-gzqw8\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:26.933940 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.933827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-accelerators-collector-config\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:26.933940 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.933878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/380264cd-6db8-4b2a-bd74-a78aed669c40-sys\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:26.933940 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.933917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-wtmp\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:26.934153 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:26.933951 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/380264cd-6db8-4b2a-bd74-a78aed669c40-root\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.035118 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.035071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzqw8\" (UniqueName: \"kubernetes.io/projected/380264cd-6db8-4b2a-bd74-a78aed669c40-kube-api-access-gzqw8\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.035318 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.035135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-accelerators-collector-config\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.035318 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.035184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/380264cd-6db8-4b2a-bd74-a78aed669c40-sys\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.035318 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.035222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-wtmp\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.035318 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.035255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/380264cd-6db8-4b2a-bd74-a78aed669c40-root\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.035318 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.035304 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/380264cd-6db8-4b2a-bd74-a78aed669c40-sys\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.035611 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.035358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-tls\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.035611 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.035381 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-wtmp\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.035611 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.035392 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/380264cd-6db8-4b2a-bd74-a78aed669c40-root\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.035611 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.035431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-textfile\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.035611 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.035471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.035611 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.035517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/380264cd-6db8-4b2a-bd74-a78aed669c40-metrics-client-ca\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.035930 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.035908 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-textfile\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.036185 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.036159 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-accelerators-collector-config\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.036459 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.036442 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/380264cd-6db8-4b2a-bd74-a78aed669c40-metrics-client-ca\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.037936 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.037911 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.038158 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.038141 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/380264cd-6db8-4b2a-bd74-a78aed669c40-node-exporter-tls\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.043810 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.043778 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzqw8\" (UniqueName: \"kubernetes.io/projected/380264cd-6db8-4b2a-bd74-a78aed669c40-kube-api-access-gzqw8\") pod \"node-exporter-blsdj\" (UID: \"380264cd-6db8-4b2a-bd74-a78aed669c40\") " pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.133690 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.133658 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-blsdj" Aug 04 10:39:27.145394 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:39:27.145351 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380264cd_6db8_4b2a_bd74_a78aed669c40.slice/crio-4630cd84903fa977cab8fc10bfe27c0739f47529f082b2490cc63053d3473d40 WatchSource:0}: Error finding container 4630cd84903fa977cab8fc10bfe27c0739f47529f082b2490cc63053d3473d40: Status 404 returned error can't find the container with id 4630cd84903fa977cab8fc10bfe27c0739f47529f082b2490cc63053d3473d40 Aug 04 10:39:27.853886 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.853845 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 10:39:27.857755 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.857728 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:27.862218 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.862177 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 04 10:39:27.862427 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.862406 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 04 10:39:27.862528 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.862465 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 04 10:39:27.862641 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.862625 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 04 10:39:27.863294 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.862719 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 04 10:39:27.863294 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.862826 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 04 10:39:27.863294 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.862932 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 04 10:39:27.863294 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.862974 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 04 10:39:27.863294 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.863185 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 04 10:39:27.863294 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.863194 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bhc8s\"" Aug 04 10:39:27.873564 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.873538 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 10:39:27.943196 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.943154 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:27.943404 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.943209 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-web-config\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:27.943404 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.943247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:27.943404 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.943294 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:27.943404 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.943340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:27.943404 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.943369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:27.943404 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.943396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:27.943983 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.943428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-config-out\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:27.943983 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.943451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp7h5\" (UniqueName: \"kubernetes.io/projected/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-kube-api-access-mp7h5\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:27.943983 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.943486 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:27.943983 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.943525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:27.943983 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.943607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:27.943983 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.943678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:27.983158 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:27.983117 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blsdj" event={"ID":"380264cd-6db8-4b2a-bd74-a78aed669c40","Type":"ContainerStarted","Data":"4630cd84903fa977cab8fc10bfe27c0739f47529f082b2490cc63053d3473d40"} Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.044486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.044583 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.044620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.044651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.044692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-config-out\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.044718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp7h5\" (UniqueName: \"kubernetes.io/projected/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-kube-api-access-mp7h5\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.044754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.044782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.044815 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.044840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.044886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.044908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-web-config\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.044933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:39:28.045273 2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:39:28.045410 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-main-tls podName:5792800b-2d2d-4416-8a49-4a46fb4dd1c5 nodeName:}" failed. No retries permitted until 2026-08-04 10:39:28.545389431 +0000 UTC m=+157.643182820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5") : secret "alertmanager-main-tls" not found Aug 04 10:39:28.045939 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.045618 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.046864 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.045630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.047517 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.047441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.049683 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.049642 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.050015 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.049993 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-web-config\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.050085 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.049997 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.050361 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.050322 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-config-out\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.051571 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.051537 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.052258 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.052196 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.052405 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.052383 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.053411 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.053372 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.061496 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.061469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp7h5\" (UniqueName: \"kubernetes.io/projected/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-kube-api-access-mp7h5\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.549052 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.548956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:28.549198 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:39:28.549091 2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Aug 04 10:39:28.549198 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:39:28.549165 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-main-tls podName:5792800b-2d2d-4416-8a49-4a46fb4dd1c5 nodeName:}" failed. No retries permitted until 2026-08-04 10:39:29.54914709 +0000 UTC m=+158.646940463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5") : secret "alertmanager-main-tls" not found Aug 04 10:39:28.836305 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.836264 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-78df79b496-ssxqq"] Aug 04 10:39:28.840277 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.840251 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:28.842806 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.842780 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Aug 04 10:39:28.842806 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.842802 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Aug 04 10:39:28.843028 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.842826 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-78vokh3r68k2i\"" Aug 04 10:39:28.843028 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.842941 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ntlrm\"" Aug 04 10:39:28.843131 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.843101 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Aug 04 10:39:28.843131 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.843108 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Aug 04 10:39:28.843238 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.843127 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Aug 04 10:39:28.852699 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.852676 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78df79b496-ssxqq"] Aug 04 10:39:28.910646 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.910611 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:28.952475 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.952443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-grpc-tls\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:28.952656 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.952512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:28.952656 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.952590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0183d10-099c-4b35-bfd5-4729f172fffd-metrics-client-ca\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:28.952656 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.952620 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8czl\" (UniqueName: \"kubernetes.io/projected/a0183d10-099c-4b35-bfd5-4729f172fffd-kube-api-access-h8czl\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:28.952754 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.952655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:28.952754 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.952674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-tls\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:28.952754 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.952703 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:28.952754 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.952724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:28.984129 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.984086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:28.988246 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.988212 2576 generic.go:356] "Generic (PLEG): container finished" podID="380264cd-6db8-4b2a-bd74-a78aed669c40" containerID="499a462316a0db55b1659c5e81e59921dab2661b1313065eac30848474695209" exitCode=0 Aug 04 10:39:28.988418 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:28.988255 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blsdj" event={"ID":"380264cd-6db8-4b2a-bd74-a78aed669c40","Type":"ContainerDied","Data":"499a462316a0db55b1659c5e81e59921dab2661b1313065eac30848474695209"} Aug 04 10:39:29.053410 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.053380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-grpc-tls\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.053593 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.053444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.053593 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.053524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0183d10-099c-4b35-bfd5-4729f172fffd-metrics-client-ca\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.053593 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.053576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8czl\" (UniqueName: \"kubernetes.io/projected/a0183d10-099c-4b35-bfd5-4729f172fffd-kube-api-access-h8czl\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.053756 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.053618 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.053756 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.053653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-tls\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.054012 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.053985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.054137 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.054022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.054656 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.054626 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0183d10-099c-4b35-bfd5-4729f172fffd-metrics-client-ca\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.056659 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.056623 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-tls\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.056796 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.056715 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.056865 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.056843 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.056976 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.056948 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-grpc-tls\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.057086 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.057038 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.057549 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.057524 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0183d10-099c-4b35-bfd5-4729f172fffd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.061987 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.061960 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8czl\" (UniqueName: \"kubernetes.io/projected/a0183d10-099c-4b35-bfd5-4729f172fffd-kube-api-access-h8czl\") pod \"thanos-querier-78df79b496-ssxqq\" (UID: \"a0183d10-099c-4b35-bfd5-4729f172fffd\") " pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.149762 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.149727 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:29.184208 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.184167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:29.251425 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:39:29.251384 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vlqv6" podUID="683c4f2b-d66c-4b5c-b6ee-96d4007f49dc" Aug 04 10:39:29.287325 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.287292 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78df79b496-ssxqq"] Aug 04 10:39:29.292745 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:39:29.292711 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0183d10_099c_4b35_bfd5_4729f172fffd.slice/crio-550c765fc423fbe9512445ddb2aeef63c952b1df3a1bbeb88ae33b99cadc77c6 WatchSource:0}: Error finding container 550c765fc423fbe9512445ddb2aeef63c952b1df3a1bbeb88ae33b99cadc77c6: Status 404 returned error can't find the container with id 550c765fc423fbe9512445ddb2aeef63c952b1df3a1bbeb88ae33b99cadc77c6 Aug 04 10:39:29.558786 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.558707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:29.561163 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.561144 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:29.672488 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.672451 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:39:29.765020 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.764983 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:29.830850 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.830820 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 10:39:29.832785 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:39:29.832760 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5792800b_2d2d_4416_8a49_4a46fb4dd1c5.slice/crio-a816912af4abf66575783ce92c44b42ee90d7039b86d8c1092de97c0f8bde7aa WatchSource:0}: Error finding container a816912af4abf66575783ce92c44b42ee90d7039b86d8c1092de97c0f8bde7aa: Status 404 returned error can't find the container with id a816912af4abf66575783ce92c44b42ee90d7039b86d8c1092de97c0f8bde7aa Aug 04 10:39:29.867115 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.867070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:29.993158 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.993121 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" event={"ID":"a0183d10-099c-4b35-bfd5-4729f172fffd","Type":"ContainerStarted","Data":"550c765fc423fbe9512445ddb2aeef63c952b1df3a1bbeb88ae33b99cadc77c6"} Aug 04 10:39:29.994339 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.994314 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerStarted","Data":"a816912af4abf66575783ce92c44b42ee90d7039b86d8c1092de97c0f8bde7aa"} Aug 04 10:39:29.996874 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.996853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vlqv6" Aug 04 10:39:29.997008 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.996889 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blsdj" event={"ID":"380264cd-6db8-4b2a-bd74-a78aed669c40","Type":"ContainerStarted","Data":"2b0cbb4d7b4168e8b16651252fd180ba8b3b80bbfce922e23d920f5d90237044"} Aug 04 10:39:29.997008 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:29.996953 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blsdj" event={"ID":"380264cd-6db8-4b2a-bd74-a78aed669c40","Type":"ContainerStarted","Data":"f73a9ca908fee31c72a0225e4b7b9b7e2a417f747445882982beac422ec5f871"} Aug 04 10:39:30.019183 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:30.019119 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-blsdj" podStartSLOduration=2.924191265 podStartE2EDuration="4.019096845s" totalImagesPullingTime="1.027714584s" totalInitContainerRuntime="67.190996ms" isStatefulPod=true podCreationTimestamp="2026-08-04 10:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:39:30.017830976 +0000 UTC m=+159.115624372" watchObservedRunningTime="2026-08-04 10:39:30.019096845 +0000 UTC m=+159.116890240" Aug 04 10:39:30.055464 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:30.055430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:30.124697 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:30.124656 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:30.442824 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:39:30.442729 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ldmtm" podUID="f67d2ed4-7812-45de-95c0-32cf86d9eef2" Aug 04 10:39:31.533311 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.533269 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:31.542546 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.542495 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-762hd"] Aug 04 10:39:31.546216 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.546185 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-762hd" Aug 04 10:39:31.548662 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.548632 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-9sccc\"" Aug 04 10:39:31.548820 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.548681 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Aug 04 10:39:31.560947 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.560912 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-762hd"] Aug 04 10:39:31.619865 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.619826 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:31.681969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.681935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e4af0c2f-759e-47b4-a24e-a2771c9434d9-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-762hd\" (UID: \"e4af0c2f-759e-47b4-a24e-a2771c9434d9\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-762hd" Aug 04 10:39:31.682132 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.681979 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch87f\" (UniqueName: \"kubernetes.io/projected/e4af0c2f-759e-47b4-a24e-a2771c9434d9-kube-api-access-ch87f\") pod \"monitoring-plugin-6dffbbd67-762hd\" (UID: \"e4af0c2f-759e-47b4-a24e-a2771c9434d9\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-762hd" Aug 04 10:39:31.687752 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.687707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:31.783079 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.783045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e4af0c2f-759e-47b4-a24e-a2771c9434d9-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-762hd\" (UID: \"e4af0c2f-759e-47b4-a24e-a2771c9434d9\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-762hd" Aug 04 10:39:31.783233 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.783105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch87f\" (UniqueName: \"kubernetes.io/projected/e4af0c2f-759e-47b4-a24e-a2771c9434d9-kube-api-access-ch87f\") pod \"monitoring-plugin-6dffbbd67-762hd\" (UID: \"e4af0c2f-759e-47b4-a24e-a2771c9434d9\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-762hd" Aug 04 10:39:31.785923 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.785890 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e4af0c2f-759e-47b4-a24e-a2771c9434d9-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-762hd\" (UID: \"e4af0c2f-759e-47b4-a24e-a2771c9434d9\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-762hd" Aug 04 10:39:31.792006 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.791977 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch87f\" (UniqueName: \"kubernetes.io/projected/e4af0c2f-759e-47b4-a24e-a2771c9434d9-kube-api-access-ch87f\") pod \"monitoring-plugin-6dffbbd67-762hd\" (UID: \"e4af0c2f-759e-47b4-a24e-a2771c9434d9\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-762hd" Aug 04 10:39:31.860176 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:31.860134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-762hd" Aug 04 10:39:32.002702 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:32.002678 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-762hd"] Aug 04 10:39:32.007823 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:39:32.007791 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4af0c2f_759e_47b4_a24e_a2771c9434d9.slice/crio-8ad337fc22a2fa468308611b0dbcd686d522d38d37b8f3a0b35093393290c847 WatchSource:0}: Error finding container 8ad337fc22a2fa468308611b0dbcd686d522d38d37b8f3a0b35093393290c847: Status 404 returned error can't find the container with id 8ad337fc22a2fa468308611b0dbcd686d522d38d37b8f3a0b35093393290c847 Aug 04 10:39:32.010534 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:32.010492 2576 generic.go:356] "Generic (PLEG): container finished" podID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerID="a3a7ed4fd06db2fe060aabf94b656d7969dd480a352657885018b54b48ee881e" exitCode=0 Aug 04 10:39:32.010658 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:32.010538 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerDied","Data":"a3a7ed4fd06db2fe060aabf94b656d7969dd480a352657885018b54b48ee881e"} Aug 04 10:39:32.013422 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:32.013389 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" event={"ID":"a0183d10-099c-4b35-bfd5-4729f172fffd","Type":"ContainerStarted","Data":"3447b91c4e4d2ea236fc44a97a36d8c5b21d80987a4d30a7e2c2c3345b6625e8"} Aug 04 10:39:32.013600 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:32.013465 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" event={"ID":"a0183d10-099c-4b35-bfd5-4729f172fffd","Type":"ContainerStarted","Data":"7cf7fd23a897c690361bd21c2ecad700d28f23b5b54b057544a0cd10f6e289de"} Aug 04 10:39:32.013600 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:32.013559 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" event={"ID":"a0183d10-099c-4b35-bfd5-4729f172fffd","Type":"ContainerStarted","Data":"a7eaa4cf936a814ac05dfade9b83d63fb8128945d82db1518896c524771bd206"} Aug 04 10:39:32.465904 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:32.465872 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:33.024669 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:33.023644 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-762hd" event={"ID":"e4af0c2f-759e-47b4-a24e-a2771c9434d9","Type":"ContainerStarted","Data":"8ad337fc22a2fa468308611b0dbcd686d522d38d37b8f3a0b35093393290c847"} Aug 04 10:39:34.031430 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.031396 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" event={"ID":"a0183d10-099c-4b35-bfd5-4729f172fffd","Type":"ContainerStarted","Data":"9e4396c645ad713996f7b3049d4fbff0532fe1cd61efba7925dc6298c4859d59"} Aug 04 10:39:34.031948 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.031917 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" event={"ID":"a0183d10-099c-4b35-bfd5-4729f172fffd","Type":"ContainerStarted","Data":"3d2cca70c3c7ffdf550ccb522a0c18c9d79507b90454205431abee44878f311b"} Aug 04 10:39:34.034554 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.034528 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerStarted","Data":"d5c3432903c425560e80763dbbef5b39bd17de5166c200542a3bf9a5615da17d"} Aug 04 10:39:34.034658 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.034592 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerStarted","Data":"2895e3321f1f3b8e0786272abb6f8c45a49d7a8d20964d4ee3b0e6a0b633cc93"} Aug 04 10:39:34.036578 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.036552 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-762hd" event={"ID":"e4af0c2f-759e-47b4-a24e-a2771c9434d9","Type":"ContainerStarted","Data":"b77c7516965fbbbe690b869c81ffacefea9cbfd68f9fb494755aee350ec7ca47"} Aug 04 10:39:34.036787 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.036766 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-762hd" Aug 04 10:39:34.043260 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.043212 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-762hd" Aug 04 10:39:34.054528 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.054416 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-762hd" podStartSLOduration=1.260098228 podStartE2EDuration="3.054392496s" totalImagesPullingTime="1.794294268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:39:34.053084531 +0000 UTC m=+163.150877960" watchObservedRunningTime="2026-08-04 10:39:34.054392496 +0000 UTC m=+163.152185895" Aug 04 10:39:34.106077 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.105840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:39:34.108452 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.108374 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/683c4f2b-d66c-4b5c-b6ee-96d4007f49dc-metrics-tls\") pod \"dns-default-vlqv6\" (UID: \"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc\") " pod="openshift-dns/dns-default-vlqv6" Aug 04 10:39:34.199701 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.199673 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f4ggs\"" Aug 04 10:39:34.207457 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.207436 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vlqv6" Aug 04 10:39:34.377880 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:39:34.377855 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683c4f2b_d66c_4b5c_b6ee_96d4007f49dc.slice/crio-bb7dcf1ee01a1fe03fd973b8ce4943e0543c91522b82e846dca887efde1a65de WatchSource:0}: Error finding container bb7dcf1ee01a1fe03fd973b8ce4943e0543c91522b82e846dca887efde1a65de: Status 404 returned error can't find the container with id bb7dcf1ee01a1fe03fd973b8ce4943e0543c91522b82e846dca887efde1a65de Aug 04 10:39:34.381972 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.381948 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vlqv6"] Aug 04 10:39:34.511669 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.511635 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:34.607130 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:34.607090 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:35.045172 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:35.045036 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" event={"ID":"a0183d10-099c-4b35-bfd5-4729f172fffd","Type":"ContainerStarted","Data":"855ab3787e073f17a04e76a4f0c82861ee38875740a116ad0d5c28e637bb83c7"} Aug 04 10:39:35.045652 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:35.045245 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:35.049033 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:35.048997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerStarted","Data":"d2036562ad2ff96c9cd3737bfc74e65fe9038867a67f2bd6b5c017de2c9da42b"} Aug 04 10:39:35.049177 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:35.049070 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerStarted","Data":"63ed8916af8b2304aeabf5a731ff192bc38044028f7b5c97ae5e0f024efe8c88"} Aug 04 10:39:35.049177 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:35.049106 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerStarted","Data":"cc9eee3aa0d39bcc40387b26a15a7da397b1f8d1d8723391f2929db7f078bdcc"} Aug 04 10:39:35.049177 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:35.049132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerStarted","Data":"9ca940ac056e4fb096aeae07b72fe0a904c10eb8a1569534f2e05e820272e7dd"} Aug 04 10:39:35.050765 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:35.050714 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vlqv6" event={"ID":"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc","Type":"ContainerStarted","Data":"bb7dcf1ee01a1fe03fd973b8ce4943e0543c91522b82e846dca887efde1a65de"} Aug 04 10:39:35.072618 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:35.072549 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" podStartSLOduration=2.831761927 podStartE2EDuration="7.072523503s" totalImagesPullingTime="4.240761576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:39:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:39:35.069868409 +0000 UTC m=+164.167661839" watchObservedRunningTime="2026-08-04 10:39:35.072523503 +0000 UTC m=+164.170316896" Aug 04 10:39:35.099314 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:35.099248 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.392109886 podStartE2EDuration="8.099222219s" totalImagesPullingTime="3.686153622s" totalInitContainerRuntime="20.958711ms" isStatefulPod=false podCreationTimestamp="2026-08-04 10:39:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:39:35.097938167 +0000 UTC m=+164.195731562" watchObservedRunningTime="2026-08-04 10:39:35.099222219 +0000 UTC m=+164.197015615" Aug 04 10:39:35.355622 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:35.355577 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:35.455851 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:35.455812 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:36.062328 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:36.062299 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-78df79b496-ssxqq" Aug 04 10:39:36.345745 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:36.345713 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:36.405345 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:36.405310 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:37.062247 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:37.062205 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vlqv6" event={"ID":"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc","Type":"ContainerStarted","Data":"96a3ef6bf86ce2a22c74230ede4c41597595bd0e8d8cb36455e728f230d3472a"} Aug 04 10:39:37.062484 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:37.062271 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vlqv6" event={"ID":"683c4f2b-d66c-4b5c-b6ee-96d4007f49dc","Type":"ContainerStarted","Data":"2119c8d326631747e9b3d110c949519a634589479a9503b0e95256989dbf1728"} Aug 04 10:39:37.062484 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:37.062304 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vlqv6" Aug 04 10:39:37.083404 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:37.083332 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vlqv6" podStartSLOduration=129.42764912 podStartE2EDuration="2m11.083308127s" totalImagesPullingTime="1.655659007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:39:37.08149153 +0000 UTC m=+166.179284925" watchObservedRunningTime="2026-08-04 10:39:37.083308127 +0000 UTC m=+166.181101523" Aug 04 10:39:37.939702 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:37.939675 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84bbdfb5b5-grjg4" Aug 04 10:39:38.079259 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:38.079223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:38.185794 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:38.185714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:38.281131 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:38.281043 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:38.381626 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:38.381594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:40.221750 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:40.221712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:40.321806 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:40.321773 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:42.672218 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:42.672184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:42.737517 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:42.737470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:44.640167 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:44.640080 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:44.712550 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:44.712519 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:45.436326 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:45.436293 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:39:46.002567 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:46.002511 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:46.077732 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:46.077686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:47.069632 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:47.069604 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vlqv6" Aug 04 10:39:49.760194 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:49.760163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:49.860170 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:49.860106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:54.544546 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:54.544514 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:54.644718 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:54.644684 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:55.075386 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:55.075352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:55.137473 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:55.137443 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:55.139645 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:55.139615 2576 generic.go:356] "Generic (PLEG): container finished" podID="ee5a68b7-d654-4644-a54e-68191be69890" containerID="09fd4aefcf4aa30211dc58b62f2460275c98cda53c6b31eb24e26c570b3250c3" exitCode=0 Aug 04 10:39:55.139774 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:55.139660 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" event={"ID":"ee5a68b7-d654-4644-a54e-68191be69890","Type":"ContainerDied","Data":"09fd4aefcf4aa30211dc58b62f2460275c98cda53c6b31eb24e26c570b3250c3"} Aug 04 10:39:55.140049 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:55.140028 2576 scope.go:117] "RemoveContainer" containerID="09fd4aefcf4aa30211dc58b62f2460275c98cda53c6b31eb24e26c570b3250c3" Aug 04 10:39:56.145848 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:56.145810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-hc4bh" event={"ID":"ee5a68b7-d654-4644-a54e-68191be69890","Type":"ContainerStarted","Data":"ef8caff028f0e1476989cd29692a17ca4b486f242e8e4052a1b91c68503e19a3"} Aug 04 10:39:57.171180 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:57.171146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:57.282135 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:57.282100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:57.384308 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:57.384275 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:39:57.712363 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:39:57.712330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:40:03.517176 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:03.517144 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:40:18.229396 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:18.229357 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:40:18.306016 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:18.305981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:40:20.244621 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:20.243118 2576 generic.go:356] "Generic (PLEG): container finished" podID="83c72ce4-7b81-4396-9be6-7b5e71d1a3ff" containerID="3b721f8127e6a989c14ef1ec3d3e784cfcf981cf1e11a43ef8f7ba031f072fb1" exitCode=0 Aug 04 10:40:20.244621 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:20.243204 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" event={"ID":"83c72ce4-7b81-4396-9be6-7b5e71d1a3ff","Type":"ContainerDied","Data":"3b721f8127e6a989c14ef1ec3d3e784cfcf981cf1e11a43ef8f7ba031f072fb1"} Aug 04 10:40:20.244621 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:20.243622 2576 scope.go:117] "RemoveContainer" containerID="3b721f8127e6a989c14ef1ec3d3e784cfcf981cf1e11a43ef8f7ba031f072fb1" Aug 04 10:40:21.249211 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:21.249176 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-h9x4m" event={"ID":"83c72ce4-7b81-4396-9be6-7b5e71d1a3ff","Type":"ContainerStarted","Data":"efeaca81af6076917adabafe51316c0b0577523d4a359252cda0d4aa10aed46c"} Aug 04 10:40:46.048156 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:46.048120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:40:46.377625 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:46.377586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:40:47.130145 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.130112 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 10:40:47.130590 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.130459 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="alertmanager" containerID="cri-o://2895e3321f1f3b8e0786272abb6f8c45a49d7a8d20964d4ee3b0e6a0b633cc93" gracePeriod=120 Aug 04 10:40:47.130590 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.130572 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="kube-rbac-proxy-web" containerID="cri-o://d2036562ad2ff96c9cd3737bfc74e65fe9038867a67f2bd6b5c017de2c9da42b" gracePeriod=120 Aug 04 10:40:47.130731 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.130561 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="kube-rbac-proxy-metric" containerID="cri-o://cc9eee3aa0d39bcc40387b26a15a7da397b1f8d1d8723391f2929db7f078bdcc" gracePeriod=120 Aug 04 10:40:47.130731 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.130614 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="config-reloader" containerID="cri-o://d5c3432903c425560e80763dbbef5b39bd17de5166c200542a3bf9a5615da17d" gracePeriod=120 Aug 04 10:40:47.130731 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.130634 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="kube-rbac-proxy" containerID="cri-o://9ca940ac056e4fb096aeae07b72fe0a904c10eb8a1569534f2e05e820272e7dd" gracePeriod=120 Aug 04 10:40:47.130731 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.130696 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="prom-label-proxy" containerID="cri-o://63ed8916af8b2304aeabf5a731ff192bc38044028f7b5c97ae5e0f024efe8c88" gracePeriod=120 Aug 04 10:40:47.353731 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.353690 2576 generic.go:356] "Generic (PLEG): container finished" podID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerID="63ed8916af8b2304aeabf5a731ff192bc38044028f7b5c97ae5e0f024efe8c88" exitCode=0 Aug 04 10:40:47.353731 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.353726 2576 generic.go:356] "Generic (PLEG): container finished" podID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerID="9ca940ac056e4fb096aeae07b72fe0a904c10eb8a1569534f2e05e820272e7dd" exitCode=0 Aug 04 10:40:47.353731 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.353736 2576 generic.go:356] "Generic (PLEG): container finished" podID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerID="d5c3432903c425560e80763dbbef5b39bd17de5166c200542a3bf9a5615da17d" exitCode=0 Aug 04 10:40:47.353962 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.353744 2576 generic.go:356] "Generic (PLEG): container finished" podID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerID="2895e3321f1f3b8e0786272abb6f8c45a49d7a8d20964d4ee3b0e6a0b633cc93" exitCode=0 Aug 04 10:40:47.353962 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.353754 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerDied","Data":"63ed8916af8b2304aeabf5a731ff192bc38044028f7b5c97ae5e0f024efe8c88"} Aug 04 10:40:47.353962 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.353844 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerDied","Data":"9ca940ac056e4fb096aeae07b72fe0a904c10eb8a1569534f2e05e820272e7dd"} Aug 04 10:40:47.353962 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.353871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerDied","Data":"d5c3432903c425560e80763dbbef5b39bd17de5166c200542a3bf9a5615da17d"} Aug 04 10:40:47.353962 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:47.353898 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerDied","Data":"2895e3321f1f3b8e0786272abb6f8c45a49d7a8d20964d4ee3b0e6a0b633cc93"} Aug 04 10:40:48.365844 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.365809 2576 generic.go:356] "Generic (PLEG): container finished" podID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerID="cc9eee3aa0d39bcc40387b26a15a7da397b1f8d1d8723391f2929db7f078bdcc" exitCode=0 Aug 04 10:40:48.365844 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.365840 2576 generic.go:356] "Generic (PLEG): container finished" podID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerID="d2036562ad2ff96c9cd3737bfc74e65fe9038867a67f2bd6b5c017de2c9da42b" exitCode=0 Aug 04 10:40:48.366190 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.365874 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerDied","Data":"cc9eee3aa0d39bcc40387b26a15a7da397b1f8d1d8723391f2929db7f078bdcc"} Aug 04 10:40:48.366190 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.365940 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerDied","Data":"d2036562ad2ff96c9cd3737bfc74e65fe9038867a67f2bd6b5c017de2c9da42b"} Aug 04 10:40:48.382269 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.382249 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:48.457449 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.457418 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-tls-assets\") pod \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " Aug 04 10:40:48.457659 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.457465 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-alertmanager-trusted-ca-bundle\") pod \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " Aug 04 10:40:48.457659 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.457496 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-config-out\") pod \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " Aug 04 10:40:48.457659 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.457543 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-web-config\") pod \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " Aug 04 10:40:48.457659 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.457572 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-alertmanager-main-db\") pod \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " Aug 04 10:40:48.457659 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.457607 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy\") pod \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " Aug 04 10:40:48.457930 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.457666 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " Aug 04 10:40:48.457930 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.457696 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-config-volume\") pod \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " Aug 04 10:40:48.457930 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.457720 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-cluster-tls-config\") pod \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " Aug 04 10:40:48.457930 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.457763 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-main-tls\") pod \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " Aug 04 10:40:48.457930 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.457798 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp7h5\" (UniqueName: \"kubernetes.io/projected/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-kube-api-access-mp7h5\") pod \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " Aug 04 10:40:48.457930 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.457865 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " Aug 04 10:40:48.457930 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.457887 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-metrics-client-ca\") pod \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\" (UID: \"5792800b-2d2d-4416-8a49-4a46fb4dd1c5\") " Aug 04 10:40:48.458289 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.457930 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5792800b-2d2d-4416-8a49-4a46fb4dd1c5" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:40:48.458289 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.458222 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:40:48.458625 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.458587 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5792800b-2d2d-4416-8a49-4a46fb4dd1c5" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:40:48.458787 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.458596 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5792800b-2d2d-4416-8a49-4a46fb4dd1c5" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:40:48.461127 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.461081 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5792800b-2d2d-4416-8a49-4a46fb4dd1c5" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:40:48.461267 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.461190 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-config-out" (OuterVolumeSpecName: "config-out") pod "5792800b-2d2d-4416-8a49-4a46fb4dd1c5" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:40:48.461747 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.461704 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-kube-api-access-mp7h5" (OuterVolumeSpecName: "kube-api-access-mp7h5") pod "5792800b-2d2d-4416-8a49-4a46fb4dd1c5" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5"). InnerVolumeSpecName "kube-api-access-mp7h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:40:48.461854 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.461762 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5792800b-2d2d-4416-8a49-4a46fb4dd1c5" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:40:48.462049 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.462008 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5792800b-2d2d-4416-8a49-4a46fb4dd1c5" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:40:48.462168 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.462068 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-config-volume" (OuterVolumeSpecName: "config-volume") pod "5792800b-2d2d-4416-8a49-4a46fb4dd1c5" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:40:48.462488 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.462452 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5792800b-2d2d-4416-8a49-4a46fb4dd1c5" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:40:48.462709 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.462683 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5792800b-2d2d-4416-8a49-4a46fb4dd1c5" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:40:48.466450 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.466422 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5792800b-2d2d-4416-8a49-4a46fb4dd1c5" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:40:48.473248 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.473195 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-web-config" (OuterVolumeSpecName: "web-config") pod "5792800b-2d2d-4416-8a49-4a46fb4dd1c5" (UID: "5792800b-2d2d-4416-8a49-4a46fb4dd1c5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:40:48.559059 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.559008 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-alertmanager-main-db\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:40:48.559059 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.559053 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:40:48.559059 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.559065 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:40:48.559059 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.559076 2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-config-volume\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:40:48.559320 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.559085 2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-cluster-tls-config\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:40:48.559320 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.559093 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-main-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:40:48.559320 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.559101 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp7h5\" (UniqueName: \"kubernetes.io/projected/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-kube-api-access-mp7h5\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:40:48.559320 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.559110 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:40:48.559320 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.559119 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-metrics-client-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:40:48.559320 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.559127 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-tls-assets\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:40:48.559320 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.559135 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-config-out\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:40:48.559320 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:48.559145 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5792800b-2d2d-4416-8a49-4a46fb4dd1c5-web-config\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:40:49.373048 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.373013 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5792800b-2d2d-4416-8a49-4a46fb4dd1c5","Type":"ContainerDied","Data":"a816912af4abf66575783ce92c44b42ee90d7039b86d8c1092de97c0f8bde7aa"} Aug 04 10:40:49.373543 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.373083 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.373543 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.373089 2576 scope.go:117] "RemoveContainer" containerID="63ed8916af8b2304aeabf5a731ff192bc38044028f7b5c97ae5e0f024efe8c88" Aug 04 10:40:49.381782 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.381758 2576 scope.go:117] "RemoveContainer" containerID="cc9eee3aa0d39bcc40387b26a15a7da397b1f8d1d8723391f2929db7f078bdcc" Aug 04 10:40:49.390194 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.390174 2576 scope.go:117] "RemoveContainer" containerID="9ca940ac056e4fb096aeae07b72fe0a904c10eb8a1569534f2e05e820272e7dd" Aug 04 10:40:49.397564 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.397513 2576 scope.go:117] "RemoveContainer" containerID="d2036562ad2ff96c9cd3737bfc74e65fe9038867a67f2bd6b5c017de2c9da42b" Aug 04 10:40:49.399155 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.399137 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 10:40:49.403102 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.403082 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 10:40:49.405735 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.405720 2576 scope.go:117] "RemoveContainer" containerID="d5c3432903c425560e80763dbbef5b39bd17de5166c200542a3bf9a5615da17d" Aug 04 10:40:49.412925 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.412899 2576 scope.go:117] "RemoveContainer" containerID="2895e3321f1f3b8e0786272abb6f8c45a49d7a8d20964d4ee3b0e6a0b633cc93" Aug 04 10:40:49.420089 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.420072 2576 scope.go:117] "RemoveContainer" containerID="a3a7ed4fd06db2fe060aabf94b656d7969dd480a352657885018b54b48ee881e" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.438355 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" path="/var/lib/kubelet/pods/5792800b-2d2d-4416-8a49-4a46fb4dd1c5/volumes" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439069 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439671 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="kube-rbac-proxy-metric" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439688 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="kube-rbac-proxy-metric" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439707 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="config-reloader" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439716 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="config-reloader" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439727 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="alertmanager" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439735 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="alertmanager" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439752 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="init-config-reloader" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439760 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="init-config-reloader" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439771 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="kube-rbac-proxy-web" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439779 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="kube-rbac-proxy-web" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439800 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="prom-label-proxy" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439808 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="prom-label-proxy" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439822 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="kube-rbac-proxy" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439830 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="kube-rbac-proxy" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439955 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="prom-label-proxy" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439970 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="config-reloader" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.439983 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="kube-rbac-proxy-web" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.440001 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="kube-rbac-proxy" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.440010 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="alertmanager" Aug 04 10:40:49.442969 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.440026 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5792800b-2d2d-4416-8a49-4a46fb4dd1c5" containerName="kube-rbac-proxy-metric" Aug 04 10:40:49.447618 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.447595 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.451534 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.451490 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 04 10:40:49.451860 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.451802 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 04 10:40:49.451860 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.451837 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 04 10:40:49.451860 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.451852 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 04 10:40:49.452053 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.451860 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bhc8s\"" Aug 04 10:40:49.452219 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.452199 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 04 10:40:49.452282 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.452203 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 04 10:40:49.452363 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.452280 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 04 10:40:49.452363 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.452228 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 04 10:40:49.453267 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.453248 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 10:40:49.457469 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.457450 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 04 10:40:49.464982 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.464962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.465072 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.464990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.465072 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.465015 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bedc61f-96d3-4f61-b87c-5db98c787dea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.465072 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.465057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bedc61f-96d3-4f61-b87c-5db98c787dea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.465166 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.465088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.465166 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.465134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-config-volume\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.465166 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.465149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.465258 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.465167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.465258 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.465185 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5bedc61f-96d3-4f61-b87c-5db98c787dea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.465258 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.465242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bedc61f-96d3-4f61-b87c-5db98c787dea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.465358 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.465282 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-web-config\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.465358 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.465310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5d4z\" (UniqueName: \"kubernetes.io/projected/5bedc61f-96d3-4f61-b87c-5db98c787dea-kube-api-access-w5d4z\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.465358 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.465345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bedc61f-96d3-4f61-b87c-5db98c787dea-config-out\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.566545 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.566434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bedc61f-96d3-4f61-b87c-5db98c787dea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.566545 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.566477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.566545 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.566527 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-config-volume\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.566545 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.566545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.566868 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.566564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.566868 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.566583 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5bedc61f-96d3-4f61-b87c-5db98c787dea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.566868 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.566628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bedc61f-96d3-4f61-b87c-5db98c787dea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.566868 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.566668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-web-config\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.567061 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.566983 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5bedc61f-96d3-4f61-b87c-5db98c787dea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.567877 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.567285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5d4z\" (UniqueName: \"kubernetes.io/projected/5bedc61f-96d3-4f61-b87c-5db98c787dea-kube-api-access-w5d4z\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.567877 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.567296 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bedc61f-96d3-4f61-b87c-5db98c787dea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.567877 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.567332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bedc61f-96d3-4f61-b87c-5db98c787dea-config-out\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.567877 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.567392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.567877 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.567420 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.567877 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.567461 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bedc61f-96d3-4f61-b87c-5db98c787dea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.568397 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.567995 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bedc61f-96d3-4f61-b87c-5db98c787dea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.570719 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.569817 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-config-volume\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.570719 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.570253 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-web-config\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.570883 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.570721 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.570946 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.570925 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.571000 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.570962 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.571253 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.571221 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bedc61f-96d3-4f61-b87c-5db98c787dea-config-out\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.571342 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.571266 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bedc61f-96d3-4f61-b87c-5db98c787dea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.571977 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.571938 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.572235 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.572214 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bedc61f-96d3-4f61-b87c-5db98c787dea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.576929 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.576903 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5d4z\" (UniqueName: \"kubernetes.io/projected/5bedc61f-96d3-4f61-b87c-5db98c787dea-kube-api-access-w5d4z\") pod \"alertmanager-main-0\" (UID: \"5bedc61f-96d3-4f61-b87c-5db98c787dea\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.759077 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.759038 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:40:49.906923 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:49.906894 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 10:40:49.910221 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:40:49.910199 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bedc61f_96d3_4f61_b87c_5db98c787dea.slice/crio-885cb5b349a351cab905f6de053164ebc76f3ca571271bdf72db9a52e79df0a8 WatchSource:0}: Error finding container 885cb5b349a351cab905f6de053164ebc76f3ca571271bdf72db9a52e79df0a8: Status 404 returned error can't find the container with id 885cb5b349a351cab905f6de053164ebc76f3ca571271bdf72db9a52e79df0a8 Aug 04 10:40:50.379606 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:50.379575 2576 generic.go:356] "Generic (PLEG): container finished" podID="5bedc61f-96d3-4f61-b87c-5db98c787dea" containerID="4c54153a451dd3c6501ea188af6432ec21233dd2549ac85426afbd6875c707bb" exitCode=0 Aug 04 10:40:50.380002 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:50.379659 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bedc61f-96d3-4f61-b87c-5db98c787dea","Type":"ContainerDied","Data":"4c54153a451dd3c6501ea188af6432ec21233dd2549ac85426afbd6875c707bb"} Aug 04 10:40:50.380002 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:50.379731 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bedc61f-96d3-4f61-b87c-5db98c787dea","Type":"ContainerStarted","Data":"885cb5b349a351cab905f6de053164ebc76f3ca571271bdf72db9a52e79df0a8"} Aug 04 10:40:51.154205 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.154167 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6576c8665-c8j66"] Aug 04 10:40:51.157749 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.157726 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.160636 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.160608 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Aug 04 10:40:51.160870 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.160852 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Aug 04 10:40:51.161129 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.161110 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-hlkkv\"" Aug 04 10:40:51.161378 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.161359 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Aug 04 10:40:51.161919 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.161901 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Aug 04 10:40:51.162006 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.161908 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Aug 04 10:40:51.168425 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.168405 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Aug 04 10:40:51.179888 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.179861 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6576c8665-c8j66"] Aug 04 10:40:51.183810 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.183789 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a3898d6-385b-4066-b9fb-3054f2d4c29d-metrics-client-ca\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.183920 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.183819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4a3898d6-385b-4066-b9fb-3054f2d4c29d-federate-client-tls\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.183920 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.183845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h46xc\" (UniqueName: \"kubernetes.io/projected/4a3898d6-385b-4066-b9fb-3054f2d4c29d-kube-api-access-h46xc\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.183920 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.183868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4a3898d6-385b-4066-b9fb-3054f2d4c29d-telemeter-client-tls\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.183920 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.183893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4a3898d6-385b-4066-b9fb-3054f2d4c29d-secret-telemeter-client\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.183920 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.183915 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a3898d6-385b-4066-b9fb-3054f2d4c29d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.184154 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.183984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3898d6-385b-4066-b9fb-3054f2d4c29d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.184154 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.184016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3898d6-385b-4066-b9fb-3054f2d4c29d-serving-certs-ca-bundle\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.285464 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.285411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3898d6-385b-4066-b9fb-3054f2d4c29d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.285646 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.285541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3898d6-385b-4066-b9fb-3054f2d4c29d-serving-certs-ca-bundle\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.285646 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.285594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a3898d6-385b-4066-b9fb-3054f2d4c29d-metrics-client-ca\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.285802 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.285780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4a3898d6-385b-4066-b9fb-3054f2d4c29d-federate-client-tls\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.285857 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.285845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h46xc\" (UniqueName: \"kubernetes.io/projected/4a3898d6-385b-4066-b9fb-3054f2d4c29d-kube-api-access-h46xc\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.285910 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.285879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4a3898d6-385b-4066-b9fb-3054f2d4c29d-telemeter-client-tls\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.285965 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.285911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4a3898d6-385b-4066-b9fb-3054f2d4c29d-secret-telemeter-client\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.285965 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.285939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a3898d6-385b-4066-b9fb-3054f2d4c29d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.286463 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.286432 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a3898d6-385b-4066-b9fb-3054f2d4c29d-metrics-client-ca\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.286463 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.286457 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3898d6-385b-4066-b9fb-3054f2d4c29d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.286754 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.286730 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3898d6-385b-4066-b9fb-3054f2d4c29d-serving-certs-ca-bundle\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.288901 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.288875 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4a3898d6-385b-4066-b9fb-3054f2d4c29d-secret-telemeter-client\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.288994 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.288961 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a3898d6-385b-4066-b9fb-3054f2d4c29d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.288994 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.288970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4a3898d6-385b-4066-b9fb-3054f2d4c29d-federate-client-tls\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.289062 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.289007 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4a3898d6-385b-4066-b9fb-3054f2d4c29d-telemeter-client-tls\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.295955 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.295913 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h46xc\" (UniqueName: \"kubernetes.io/projected/4a3898d6-385b-4066-b9fb-3054f2d4c29d-kube-api-access-h46xc\") pod \"telemeter-client-6576c8665-c8j66\" (UID: \"4a3898d6-385b-4066-b9fb-3054f2d4c29d\") " pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.386786 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.386749 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bedc61f-96d3-4f61-b87c-5db98c787dea","Type":"ContainerStarted","Data":"249002c04e0584ae5c7ed64419bfad9e1b2bba269252a7d13c178d62bde62d40"} Aug 04 10:40:51.387211 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.386848 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bedc61f-96d3-4f61-b87c-5db98c787dea","Type":"ContainerStarted","Data":"4d7b95aef401db9cee002a29bef8843802ad5c4a301999670d6ba11f76f1e67f"} Aug 04 10:40:51.387211 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.386875 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bedc61f-96d3-4f61-b87c-5db98c787dea","Type":"ContainerStarted","Data":"687a5c5a1e68f5fff5207c8bc2d4bf88a0e8fe1fe69cbe80471ada29af61dbf1"} Aug 04 10:40:51.387211 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.386899 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bedc61f-96d3-4f61-b87c-5db98c787dea","Type":"ContainerStarted","Data":"adf71d299be31a6647866d8737018a334519f74b1dae62fe48119055267f025c"} Aug 04 10:40:51.387211 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.386924 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bedc61f-96d3-4f61-b87c-5db98c787dea","Type":"ContainerStarted","Data":"cb6a66d53fb25fa3a93930848b4352caf95702b9d8db65c1ea5e21bf72bfd9d4"} Aug 04 10:40:51.387211 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.386950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bedc61f-96d3-4f61-b87c-5db98c787dea","Type":"ContainerStarted","Data":"42308a7da873066884749149cac6e0cc56cea55093e23e28ab3da0de0a7d3896"} Aug 04 10:40:51.416712 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.416594 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.401253647 podStartE2EDuration="2.41657383s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.320183ms" isStatefulPod=false podCreationTimestamp="2026-08-04 10:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:40:51.414667431 +0000 UTC m=+240.512460825" watchObservedRunningTime="2026-08-04 10:40:51.41657383 +0000 UTC m=+240.514367224" Aug 04 10:40:51.470788 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.470756 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-hlkkv\"" Aug 04 10:40:51.479261 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.479229 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" Aug 04 10:40:51.638478 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:51.638452 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6576c8665-c8j66"] Aug 04 10:40:51.640873 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:40:51.640848 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3898d6_385b_4066_b9fb_3054f2d4c29d.slice/crio-a9e58a8d34a751bc63b085d2e7d4f0b771252391bdb6a2ee919f951233da10a5 WatchSource:0}: Error finding container a9e58a8d34a751bc63b085d2e7d4f0b771252391bdb6a2ee919f951233da10a5: Status 404 returned error can't find the container with id a9e58a8d34a751bc63b085d2e7d4f0b771252391bdb6a2ee919f951233da10a5 Aug 04 10:40:52.393605 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:52.393563 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" event={"ID":"4a3898d6-385b-4066-b9fb-3054f2d4c29d","Type":"ContainerStarted","Data":"a9e58a8d34a751bc63b085d2e7d4f0b771252391bdb6a2ee919f951233da10a5"} Aug 04 10:40:54.404893 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:54.404853 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" event={"ID":"4a3898d6-385b-4066-b9fb-3054f2d4c29d","Type":"ContainerStarted","Data":"44d735d2b82cc123a9bccf8032e40bd84a45df449892776691331495bc9c217f"} Aug 04 10:40:54.405342 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:54.404943 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" event={"ID":"4a3898d6-385b-4066-b9fb-3054f2d4c29d","Type":"ContainerStarted","Data":"90859fd397b168392c8f0c1e469725df6448b70d67219b0d1753c6fd71897d08"} Aug 04 10:40:54.405342 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:54.404964 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" event={"ID":"4a3898d6-385b-4066-b9fb-3054f2d4c29d","Type":"ContainerStarted","Data":"206a1e7ba40f47e979572dd344112435c227306bfb3601c824661ee4ddb8b618"} Aug 04 10:40:54.430740 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:54.430533 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6576c8665-c8j66" podStartSLOduration=1.699939063 podStartE2EDuration="3.430495693s" totalImagesPullingTime="1.73055663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:40:54.4291198 +0000 UTC m=+243.526913196" watchObservedRunningTime="2026-08-04 10:40:54.430495693 +0000 UTC m=+243.528289089" Aug 04 10:40:54.638854 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:54.638817 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:40:54.756415 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:54.756325 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:40:54.838200 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:54.838171 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:40:55.338449 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:40:55.338421 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:41:03.398436 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:03.398398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:41:03.400701 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:03.400681 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67d2ed4-7812-45de-95c0-32cf86d9eef2-metrics-certs\") pod \"network-metrics-daemon-ldmtm\" (UID: \"f67d2ed4-7812-45de-95c0-32cf86d9eef2\") " pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:41:03.439330 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:03.439306 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t7blv\"" Aug 04 10:41:03.447635 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:03.447616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldmtm" Aug 04 10:41:03.579841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:03.579811 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldmtm"] Aug 04 10:41:03.582733 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:41:03.582690 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67d2ed4_7812_45de_95c0_32cf86d9eef2.slice/crio-d0cd98e65913dc81155d76581c7ebf5dbb83923ac3c6036dd20a9cfec7a18185 WatchSource:0}: Error finding container d0cd98e65913dc81155d76581c7ebf5dbb83923ac3c6036dd20a9cfec7a18185: Status 404 returned error can't find the container with id d0cd98e65913dc81155d76581c7ebf5dbb83923ac3c6036dd20a9cfec7a18185 Aug 04 10:41:04.456295 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:04.456253 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldmtm" event={"ID":"f67d2ed4-7812-45de-95c0-32cf86d9eef2","Type":"ContainerStarted","Data":"d0cd98e65913dc81155d76581c7ebf5dbb83923ac3c6036dd20a9cfec7a18185"} Aug 04 10:41:05.373592 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:05.373554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:41:05.462241 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:05.462203 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldmtm" event={"ID":"f67d2ed4-7812-45de-95c0-32cf86d9eef2","Type":"ContainerStarted","Data":"6cd5991ca57b6742571927920b61ab63af0979c5dc8059eef9a97c8088e74ca7"} Aug 04 10:41:05.462757 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:05.462265 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldmtm" event={"ID":"f67d2ed4-7812-45de-95c0-32cf86d9eef2","Type":"ContainerStarted","Data":"62d393790ed8b298c449977ed5b08789558fdd72284903b39a02d2f05f8d38c3"} Aug 04 10:41:05.474074 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:05.474040 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:41:05.479625 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:05.479580 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ldmtm" podStartSLOduration=252.518106744 podStartE2EDuration="4m13.479563987s" totalImagesPullingTime="961.457243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:41:05.478184862 +0000 UTC m=+254.575978258" watchObservedRunningTime="2026-08-04 10:41:05.479563987 +0000 UTC m=+254.577357389" Aug 04 10:41:05.641146 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:05.641065 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:41:05.871363 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:05.871329 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:41:05.986352 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:05.986273 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:41:06.092859 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:06.092823 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:41:24.132032 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:24.131953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:41:51.341810 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:41:51.341764 2576 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 10:42:01.634746 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:01.634705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:42:01.698973 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:01.698943 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:42:06.993665 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:06.993624 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:42:07.095291 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:07.095259 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:42:07.194024 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:07.193992 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:42:07.745382 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:07.745350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:42:07.875200 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:07.875165 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:42:13.348204 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:13.348170 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:42:13.410599 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:13.410564 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:42:18.302340 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:18.302300 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:42:18.403004 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:18.402971 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:42:18.471614 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:18.471580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:42:19.770044 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:19.770010 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:42:19.870612 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:19.870579 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:42:20.042158 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:42:20.042080 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:43:34.117560 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:43:34.117496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:43:37.552091 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:43:37.552057 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:43:58.141337 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:43:58.141299 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:44:24.896832 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:24.896752 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:44:41.952791 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:41.952756 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg"] Aug 04 10:44:41.956311 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:41.956293 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" Aug 04 10:44:41.958633 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:41.958610 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 10:44:41.958764 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:41.958615 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n59pt\"" Aug 04 10:44:41.959472 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:41.959455 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 10:44:41.969236 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:41.969211 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg"] Aug 04 10:44:42.034731 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:42.034702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg\" (UID: \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" Aug 04 10:44:42.034918 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:42.034744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg\" (UID: \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" Aug 04 10:44:42.034918 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:42.034773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgfd9\" (UniqueName: \"kubernetes.io/projected/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-kube-api-access-sgfd9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg\" (UID: \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" Aug 04 10:44:42.136000 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:42.135958 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgfd9\" (UniqueName: \"kubernetes.io/projected/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-kube-api-access-sgfd9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg\" (UID: \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" Aug 04 10:44:42.136200 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:42.136047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg\" (UID: \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" Aug 04 10:44:42.136200 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:42.136079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg\" (UID: \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" Aug 04 10:44:42.136404 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:42.136387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg\" (UID: \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" Aug 04 10:44:42.136475 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:42.136447 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg\" (UID: \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" Aug 04 10:44:42.144460 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:42.144434 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgfd9\" (UniqueName: \"kubernetes.io/projected/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-kube-api-access-sgfd9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg\" (UID: \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" Aug 04 10:44:42.265404 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:42.265294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" Aug 04 10:44:42.404039 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:42.403877 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg"] Aug 04 10:44:42.406959 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:44:42.406936 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e23f1e9_3baf_4f05_a9e5_2faee00e7797.slice/crio-c37cc32706c722de3d8ab8f02d5b8779887eb81e3fab34a4ca1d8db798232c3e WatchSource:0}: Error finding container c37cc32706c722de3d8ab8f02d5b8779887eb81e3fab34a4ca1d8db798232c3e: Status 404 returned error can't find the container with id c37cc32706c722de3d8ab8f02d5b8779887eb81e3fab34a4ca1d8db798232c3e Aug 04 10:44:42.408887 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:42.408867 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 10:44:43.358897 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:43.358855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" event={"ID":"5e23f1e9-3baf-4f05-a9e5-2faee00e7797","Type":"ContainerStarted","Data":"c37cc32706c722de3d8ab8f02d5b8779887eb81e3fab34a4ca1d8db798232c3e"} Aug 04 10:44:48.387616 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:48.387579 2576 generic.go:356] "Generic (PLEG): container finished" podID="5e23f1e9-3baf-4f05-a9e5-2faee00e7797" containerID="6cbc03f2b52b76b75f93a1ae83ec2330715909d40e07b7c0e6f07940e6df72c4" exitCode=0 Aug 04 10:44:48.387999 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:48.387669 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" event={"ID":"5e23f1e9-3baf-4f05-a9e5-2faee00e7797","Type":"ContainerDied","Data":"6cbc03f2b52b76b75f93a1ae83ec2330715909d40e07b7c0e6f07940e6df72c4"} Aug 04 10:44:51.401822 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:51.401781 2576 generic.go:356] "Generic (PLEG): container finished" podID="5e23f1e9-3baf-4f05-a9e5-2faee00e7797" containerID="a091c1c5d020f51b74e8c8bc36bbb1f77f9c0e8f1763e2d69cf710764b9117d5" exitCode=0 Aug 04 10:44:51.402272 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:51.401860 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" event={"ID":"5e23f1e9-3baf-4f05-a9e5-2faee00e7797","Type":"ContainerDied","Data":"a091c1c5d020f51b74e8c8bc36bbb1f77f9c0e8f1763e2d69cf710764b9117d5"} Aug 04 10:44:53.231366 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:53.231333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:44:53.304512 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:53.304461 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:44:59.448186 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:59.448145 2576 generic.go:356] "Generic (PLEG): container finished" podID="5e23f1e9-3baf-4f05-a9e5-2faee00e7797" containerID="12a2a5078544b8f3b3366cd2f7e3fcc9c65c8bbb4c1245016426af6fe31998f5" exitCode=0 Aug 04 10:44:59.448580 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:44:59.448222 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" event={"ID":"5e23f1e9-3baf-4f05-a9e5-2faee00e7797","Type":"ContainerDied","Data":"12a2a5078544b8f3b3366cd2f7e3fcc9c65c8bbb4c1245016426af6fe31998f5"} Aug 04 10:45:00.578925 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:00.578899 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" Aug 04 10:45:00.707398 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:00.707311 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgfd9\" (UniqueName: \"kubernetes.io/projected/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-kube-api-access-sgfd9\") pod \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\" (UID: \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\") " Aug 04 10:45:00.707398 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:00.707390 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-bundle\") pod \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\" (UID: \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\") " Aug 04 10:45:00.707621 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:00.707555 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-util\") pod \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\" (UID: \"5e23f1e9-3baf-4f05-a9e5-2faee00e7797\") " Aug 04 10:45:00.708060 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:00.708039 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-bundle" (OuterVolumeSpecName: "bundle") pod "5e23f1e9-3baf-4f05-a9e5-2faee00e7797" (UID: "5e23f1e9-3baf-4f05-a9e5-2faee00e7797"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:45:00.709578 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:00.709558 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-kube-api-access-sgfd9" (OuterVolumeSpecName: "kube-api-access-sgfd9") pod "5e23f1e9-3baf-4f05-a9e5-2faee00e7797" (UID: "5e23f1e9-3baf-4f05-a9e5-2faee00e7797"). InnerVolumeSpecName "kube-api-access-sgfd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:45:00.712249 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:00.712224 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-util" (OuterVolumeSpecName: "util") pod "5e23f1e9-3baf-4f05-a9e5-2faee00e7797" (UID: "5e23f1e9-3baf-4f05-a9e5-2faee00e7797"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:45:00.809250 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:00.809201 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgfd9\" (UniqueName: \"kubernetes.io/projected/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-kube-api-access-sgfd9\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:45:00.809250 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:00.809233 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-bundle\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:45:00.809250 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:00.809242 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e23f1e9-3baf-4f05-a9e5-2faee00e7797-util\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:45:01.457182 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:01.457156 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" Aug 04 10:45:01.457182 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:01.457171 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xkktg" event={"ID":"5e23f1e9-3baf-4f05-a9e5-2faee00e7797","Type":"ContainerDied","Data":"c37cc32706c722de3d8ab8f02d5b8779887eb81e3fab34a4ca1d8db798232c3e"} Aug 04 10:45:01.457403 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:01.457214 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c37cc32706c722de3d8ab8f02d5b8779887eb81e3fab34a4ca1d8db798232c3e" Aug 04 10:45:03.923489 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:03.923455 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:04.025849 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:04.025818 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:04.584917 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:04.584881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:06.375239 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:06.375208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:08.229375 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.229344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:08.361925 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.361895 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8mg8d"] Aug 04 10:45:08.362220 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.362209 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e23f1e9-3baf-4f05-a9e5-2faee00e7797" containerName="util" Aug 04 10:45:08.362264 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.362221 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e23f1e9-3baf-4f05-a9e5-2faee00e7797" containerName="util" Aug 04 10:45:08.362264 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.362231 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e23f1e9-3baf-4f05-a9e5-2faee00e7797" containerName="extract" Aug 04 10:45:08.362264 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.362237 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e23f1e9-3baf-4f05-a9e5-2faee00e7797" containerName="extract" Aug 04 10:45:08.362264 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.362247 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e23f1e9-3baf-4f05-a9e5-2faee00e7797" containerName="pull" Aug 04 10:45:08.362264 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.362254 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e23f1e9-3baf-4f05-a9e5-2faee00e7797" containerName="pull" Aug 04 10:45:08.362409 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.362330 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e23f1e9-3baf-4f05-a9e5-2faee00e7797" containerName="extract" Aug 04 10:45:08.389269 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.389240 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8mg8d" Aug 04 10:45:08.392988 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.392956 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9mmlf\"" Aug 04 10:45:08.393139 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.393061 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:45:08.395359 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.395342 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Aug 04 10:45:08.402337 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.402311 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8mg8d"] Aug 04 10:45:08.476868 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.476827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq7fl\" (UniqueName: \"kubernetes.io/projected/456deba5-f86b-4656-be13-c9ab228b9613-kube-api-access-dq7fl\") pod \"cert-manager-operator-controller-manager-55c444bf86-8mg8d\" (UID: \"456deba5-f86b-4656-be13-c9ab228b9613\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8mg8d" Aug 04 10:45:08.477066 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.476903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/456deba5-f86b-4656-be13-c9ab228b9613-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8mg8d\" (UID: \"456deba5-f86b-4656-be13-c9ab228b9613\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8mg8d" Aug 04 10:45:08.577568 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.577473 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/456deba5-f86b-4656-be13-c9ab228b9613-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8mg8d\" (UID: \"456deba5-f86b-4656-be13-c9ab228b9613\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8mg8d" Aug 04 10:45:08.577568 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.577567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq7fl\" (UniqueName: \"kubernetes.io/projected/456deba5-f86b-4656-be13-c9ab228b9613-kube-api-access-dq7fl\") pod \"cert-manager-operator-controller-manager-55c444bf86-8mg8d\" (UID: \"456deba5-f86b-4656-be13-c9ab228b9613\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8mg8d" Aug 04 10:45:08.577866 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.577845 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/456deba5-f86b-4656-be13-c9ab228b9613-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8mg8d\" (UID: \"456deba5-f86b-4656-be13-c9ab228b9613\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8mg8d" Aug 04 10:45:08.589004 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.588977 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq7fl\" (UniqueName: \"kubernetes.io/projected/456deba5-f86b-4656-be13-c9ab228b9613-kube-api-access-dq7fl\") pod \"cert-manager-operator-controller-manager-55c444bf86-8mg8d\" (UID: \"456deba5-f86b-4656-be13-c9ab228b9613\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8mg8d" Aug 04 10:45:08.698886 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.698851 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8mg8d" Aug 04 10:45:08.851847 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:08.851814 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8mg8d"] Aug 04 10:45:08.855346 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:45:08.855314 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456deba5_f86b_4656_be13_c9ab228b9613.slice/crio-82502cbef5fe1698d2cba1aafe38aaa79c277c5d128445acd4374063356989bb WatchSource:0}: Error finding container 82502cbef5fe1698d2cba1aafe38aaa79c277c5d128445acd4374063356989bb: Status 404 returned error can't find the container with id 82502cbef5fe1698d2cba1aafe38aaa79c277c5d128445acd4374063356989bb Aug 04 10:45:09.494439 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:09.494407 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8mg8d" event={"ID":"456deba5-f86b-4656-be13-c9ab228b9613","Type":"ContainerStarted","Data":"82502cbef5fe1698d2cba1aafe38aaa79c277c5d128445acd4374063356989bb"} Aug 04 10:45:10.529061 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:10.529026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:10.637788 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:10.637756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:11.429776 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:11.429744 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:11.506227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:11.506193 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8mg8d" event={"ID":"456deba5-f86b-4656-be13-c9ab228b9613","Type":"ContainerStarted","Data":"dffb14440c97ded03603d68738878b9b89c6029fdb4f2ef3d664ec1f42cc1c3c"} Aug 04 10:45:11.529162 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:11.529107 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8mg8d" podStartSLOduration=1.729118992 podStartE2EDuration="3.529087285s" totalImagesPullingTime="1.799968293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:45:11.527806814 +0000 UTC m=+500.625600209" watchObservedRunningTime="2026-08-04 10:45:11.529087285 +0000 UTC m=+500.626880679" Aug 04 10:45:11.532625 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:11.532589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:12.159229 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:12.159193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:12.247678 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:12.247645 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:13.177260 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.177225 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4"] Aug 04 10:45:13.179946 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.179930 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" Aug 04 10:45:13.182629 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.182608 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n59pt\"" Aug 04 10:45:13.182885 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.182870 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 10:45:13.183403 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.183389 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 10:45:13.190447 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.190427 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4"] Aug 04 10:45:13.322472 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.322434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e555d92-ca59-4033-a383-649f7031b888-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4\" (UID: \"7e555d92-ca59-4033-a383-649f7031b888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" Aug 04 10:45:13.322664 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.322494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e555d92-ca59-4033-a383-649f7031b888-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4\" (UID: \"7e555d92-ca59-4033-a383-649f7031b888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" Aug 04 10:45:13.322664 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.322549 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhhdb\" (UniqueName: \"kubernetes.io/projected/7e555d92-ca59-4033-a383-649f7031b888-kube-api-access-xhhdb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4\" (UID: \"7e555d92-ca59-4033-a383-649f7031b888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" Aug 04 10:45:13.421959 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.421929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:13.423059 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.423039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e555d92-ca59-4033-a383-649f7031b888-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4\" (UID: \"7e555d92-ca59-4033-a383-649f7031b888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" Aug 04 10:45:13.423131 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.423070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhhdb\" (UniqueName: \"kubernetes.io/projected/7e555d92-ca59-4033-a383-649f7031b888-kube-api-access-xhhdb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4\" (UID: \"7e555d92-ca59-4033-a383-649f7031b888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" Aug 04 10:45:13.423131 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.423125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e555d92-ca59-4033-a383-649f7031b888-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4\" (UID: \"7e555d92-ca59-4033-a383-649f7031b888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" Aug 04 10:45:13.423453 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.423436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e555d92-ca59-4033-a383-649f7031b888-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4\" (UID: \"7e555d92-ca59-4033-a383-649f7031b888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" Aug 04 10:45:13.423496 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.423444 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e555d92-ca59-4033-a383-649f7031b888-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4\" (UID: \"7e555d92-ca59-4033-a383-649f7031b888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" Aug 04 10:45:13.432234 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.432176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhhdb\" (UniqueName: \"kubernetes.io/projected/7e555d92-ca59-4033-a383-649f7031b888-kube-api-access-xhhdb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4\" (UID: \"7e555d92-ca59-4033-a383-649f7031b888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" Aug 04 10:45:13.489001 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.488970 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" Aug 04 10:45:13.622194 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.622169 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4"] Aug 04 10:45:13.624243 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:45:13.624214 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e555d92_ca59_4033_a383_649f7031b888.slice/crio-1eebe7b0cc85420a433395e5a37168dcfd5e3067c529fb23733677b616750eda WatchSource:0}: Error finding container 1eebe7b0cc85420a433395e5a37168dcfd5e3067c529fb23733677b616750eda: Status 404 returned error can't find the container with id 1eebe7b0cc85420a433395e5a37168dcfd5e3067c529fb23733677b616750eda Aug 04 10:45:13.821206 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:13.821175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:14.520655 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:14.520616 2576 generic.go:356] "Generic (PLEG): container finished" podID="7e555d92-ca59-4033-a383-649f7031b888" containerID="8a418e00c19270bed7c4a479c00d37007242bb87792b5d132b2f2de21b578f65" exitCode=0 Aug 04 10:45:14.521157 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:14.520670 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" event={"ID":"7e555d92-ca59-4033-a383-649f7031b888","Type":"ContainerDied","Data":"8a418e00c19270bed7c4a479c00d37007242bb87792b5d132b2f2de21b578f65"} Aug 04 10:45:14.521157 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:14.520713 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" event={"ID":"7e555d92-ca59-4033-a383-649f7031b888","Type":"ContainerStarted","Data":"1eebe7b0cc85420a433395e5a37168dcfd5e3067c529fb23733677b616750eda"} Aug 04 10:45:16.726284 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:16.726248 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:16.804465 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:16.804429 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:20.551442 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:20.551352 2576 generic.go:356] "Generic (PLEG): container finished" podID="7e555d92-ca59-4033-a383-649f7031b888" containerID="af78bd8a4c9d946244fd51d4fd4a294604d2b8e72443603a9c408c5dc22c97a7" exitCode=0 Aug 04 10:45:20.551875 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:20.551435 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" event={"ID":"7e555d92-ca59-4033-a383-649f7031b888","Type":"ContainerDied","Data":"af78bd8a4c9d946244fd51d4fd4a294604d2b8e72443603a9c408c5dc22c97a7"} Aug 04 10:45:21.557393 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:21.557354 2576 generic.go:356] "Generic (PLEG): container finished" podID="7e555d92-ca59-4033-a383-649f7031b888" containerID="c9614644703976acc15bc8de00165f2c712561b00ef91971b4a2af267129be15" exitCode=0 Aug 04 10:45:21.557838 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:21.557418 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" event={"ID":"7e555d92-ca59-4033-a383-649f7031b888","Type":"ContainerDied","Data":"c9614644703976acc15bc8de00165f2c712561b00ef91971b4a2af267129be15"} Aug 04 10:45:22.248419 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:22.248387 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:22.562033 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:22.559345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:22.656836 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:22.656798 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:22.691498 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:22.691467 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" Aug 04 10:45:22.806805 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:22.806773 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e555d92-ca59-4033-a383-649f7031b888-bundle\") pod \"7e555d92-ca59-4033-a383-649f7031b888\" (UID: \"7e555d92-ca59-4033-a383-649f7031b888\") " Aug 04 10:45:22.806805 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:22.806809 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhhdb\" (UniqueName: \"kubernetes.io/projected/7e555d92-ca59-4033-a383-649f7031b888-kube-api-access-xhhdb\") pod \"7e555d92-ca59-4033-a383-649f7031b888\" (UID: \"7e555d92-ca59-4033-a383-649f7031b888\") " Aug 04 10:45:22.807052 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:22.806833 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e555d92-ca59-4033-a383-649f7031b888-util\") pod \"7e555d92-ca59-4033-a383-649f7031b888\" (UID: \"7e555d92-ca59-4033-a383-649f7031b888\") " Aug 04 10:45:22.807198 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:22.807175 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e555d92-ca59-4033-a383-649f7031b888-bundle" (OuterVolumeSpecName: "bundle") pod "7e555d92-ca59-4033-a383-649f7031b888" (UID: "7e555d92-ca59-4033-a383-649f7031b888"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:45:22.809098 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:22.809066 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e555d92-ca59-4033-a383-649f7031b888-kube-api-access-xhhdb" (OuterVolumeSpecName: "kube-api-access-xhhdb") pod "7e555d92-ca59-4033-a383-649f7031b888" (UID: "7e555d92-ca59-4033-a383-649f7031b888"). InnerVolumeSpecName "kube-api-access-xhhdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:45:22.811442 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:22.811422 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e555d92-ca59-4033-a383-649f7031b888-util" (OuterVolumeSpecName: "util") pod "7e555d92-ca59-4033-a383-649f7031b888" (UID: "7e555d92-ca59-4033-a383-649f7031b888"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:45:22.908373 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:22.908337 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e555d92-ca59-4033-a383-649f7031b888-bundle\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:45:22.908373 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:22.908366 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhhdb\" (UniqueName: \"kubernetes.io/projected/7e555d92-ca59-4033-a383-649f7031b888-kube-api-access-xhhdb\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:45:22.908373 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:22.908378 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e555d92-ca59-4033-a383-649f7031b888-util\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:45:23.569962 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:23.569930 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" event={"ID":"7e555d92-ca59-4033-a383-649f7031b888","Type":"ContainerDied","Data":"1eebe7b0cc85420a433395e5a37168dcfd5e3067c529fb23733677b616750eda"} Aug 04 10:45:23.569962 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:23.569958 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6h2h4" Aug 04 10:45:23.570427 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:23.569974 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eebe7b0cc85420a433395e5a37168dcfd5e3067c529fb23733677b616750eda" Aug 04 10:45:25.343416 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:25.343386 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:25.444372 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:25.444341 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:27.057180 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.057147 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-stpb8"] Aug 04 10:45:27.057573 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.057518 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e555d92-ca59-4033-a383-649f7031b888" containerName="util" Aug 04 10:45:27.057573 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.057532 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e555d92-ca59-4033-a383-649f7031b888" containerName="util" Aug 04 10:45:27.057573 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.057545 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e555d92-ca59-4033-a383-649f7031b888" containerName="extract" Aug 04 10:45:27.057573 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.057550 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e555d92-ca59-4033-a383-649f7031b888" containerName="extract" Aug 04 10:45:27.057573 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.057571 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e555d92-ca59-4033-a383-649f7031b888" containerName="pull" Aug 04 10:45:27.057573 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.057576 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e555d92-ca59-4033-a383-649f7031b888" containerName="pull" Aug 04 10:45:27.057754 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.057637 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e555d92-ca59-4033-a383-649f7031b888" containerName="extract" Aug 04 10:45:27.060433 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.060416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-stpb8" Aug 04 10:45:27.065286 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.065260 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-cq5g6\"" Aug 04 10:45:27.066140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.066118 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:45:27.067015 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.066931 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 10:45:27.073872 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.073846 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-stpb8"] Aug 04 10:45:27.139234 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.139198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phhdk\" (UniqueName: \"kubernetes.io/projected/911f00c5-a0a1-4b20-9d65-146bddbf69f3-kube-api-access-phhdk\") pod \"openshift-lws-operator-fd8ccff4c-stpb8\" (UID: \"911f00c5-a0a1-4b20-9d65-146bddbf69f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-stpb8" Aug 04 10:45:27.139426 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.139262 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911f00c5-a0a1-4b20-9d65-146bddbf69f3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-stpb8\" (UID: \"911f00c5-a0a1-4b20-9d65-146bddbf69f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-stpb8" Aug 04 10:45:27.240221 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.240185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phhdk\" (UniqueName: \"kubernetes.io/projected/911f00c5-a0a1-4b20-9d65-146bddbf69f3-kube-api-access-phhdk\") pod \"openshift-lws-operator-fd8ccff4c-stpb8\" (UID: \"911f00c5-a0a1-4b20-9d65-146bddbf69f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-stpb8" Aug 04 10:45:27.240398 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.240255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911f00c5-a0a1-4b20-9d65-146bddbf69f3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-stpb8\" (UID: \"911f00c5-a0a1-4b20-9d65-146bddbf69f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-stpb8" Aug 04 10:45:27.240619 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.240602 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911f00c5-a0a1-4b20-9d65-146bddbf69f3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-stpb8\" (UID: \"911f00c5-a0a1-4b20-9d65-146bddbf69f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-stpb8" Aug 04 10:45:27.250563 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.250527 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phhdk\" (UniqueName: \"kubernetes.io/projected/911f00c5-a0a1-4b20-9d65-146bddbf69f3-kube-api-access-phhdk\") pod \"openshift-lws-operator-fd8ccff4c-stpb8\" (UID: \"911f00c5-a0a1-4b20-9d65-146bddbf69f3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-stpb8" Aug 04 10:45:27.382852 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.382820 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-stpb8" Aug 04 10:45:27.515597 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.515478 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-stpb8"] Aug 04 10:45:27.518412 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:45:27.518382 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod911f00c5_a0a1_4b20_9d65_146bddbf69f3.slice/crio-432f2ea5d56009ab48a839477a4ed96b1b279f2bf1e9f1465c49b38031b97f1e WatchSource:0}: Error finding container 432f2ea5d56009ab48a839477a4ed96b1b279f2bf1e9f1465c49b38031b97f1e: Status 404 returned error can't find the container with id 432f2ea5d56009ab48a839477a4ed96b1b279f2bf1e9f1465c49b38031b97f1e Aug 04 10:45:27.588936 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:27.588898 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-stpb8" event={"ID":"911f00c5-a0a1-4b20-9d65-146bddbf69f3","Type":"ContainerStarted","Data":"432f2ea5d56009ab48a839477a4ed96b1b279f2bf1e9f1465c49b38031b97f1e"} Aug 04 10:45:29.607577 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:29.607539 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-stpb8" event={"ID":"911f00c5-a0a1-4b20-9d65-146bddbf69f3","Type":"ContainerStarted","Data":"11d10f9ffb661659a4cc7ebada1d09beb75f7d4bb278d1c297eceba401eaa435"} Aug 04 10:45:29.625992 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:29.625941 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-stpb8" podStartSLOduration=0.959366533 podStartE2EDuration="2.625924245s" totalImagesPullingTime="1.666557712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:45:29.623931328 +0000 UTC m=+518.721724724" watchObservedRunningTime="2026-08-04 10:45:29.625924245 +0000 UTC m=+518.723717639" Aug 04 10:45:30.360803 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:30.360765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:30.474137 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:30.474105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:31.868479 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:31.868437 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p"] Aug 04 10:45:31.871577 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:31.871554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" Aug 04 10:45:31.874199 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:31.874173 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 10:45:31.875047 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:31.875033 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 10:45:31.875112 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:31.875067 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n59pt\"" Aug 04 10:45:31.880131 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:31.880104 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p"] Aug 04 10:45:31.977863 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:31.977830 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b47ea31-2115-4b13-b5ec-32087a7c52ee-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p\" (UID: \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" Aug 04 10:45:31.977863 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:31.977867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7cpv\" (UniqueName: \"kubernetes.io/projected/8b47ea31-2115-4b13-b5ec-32087a7c52ee-kube-api-access-g7cpv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p\" (UID: \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" Aug 04 10:45:31.978083 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:31.977895 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b47ea31-2115-4b13-b5ec-32087a7c52ee-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p\" (UID: \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" Aug 04 10:45:32.078710 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:32.078671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b47ea31-2115-4b13-b5ec-32087a7c52ee-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p\" (UID: \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" Aug 04 10:45:32.078710 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:32.078709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7cpv\" (UniqueName: \"kubernetes.io/projected/8b47ea31-2115-4b13-b5ec-32087a7c52ee-kube-api-access-g7cpv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p\" (UID: \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" Aug 04 10:45:32.078977 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:32.078746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b47ea31-2115-4b13-b5ec-32087a7c52ee-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p\" (UID: \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" Aug 04 10:45:32.079198 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:32.079175 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b47ea31-2115-4b13-b5ec-32087a7c52ee-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p\" (UID: \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" Aug 04 10:45:32.079272 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:32.079209 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b47ea31-2115-4b13-b5ec-32087a7c52ee-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p\" (UID: \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" Aug 04 10:45:32.087272 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:32.087236 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7cpv\" (UniqueName: \"kubernetes.io/projected/8b47ea31-2115-4b13-b5ec-32087a7c52ee-kube-api-access-g7cpv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p\" (UID: \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" Aug 04 10:45:32.182136 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:32.182052 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" Aug 04 10:45:32.320415 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:32.320385 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p"] Aug 04 10:45:32.323137 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:45:32.323102 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b47ea31_2115_4b13_b5ec_32087a7c52ee.slice/crio-3b6384cf44e47813e4b7347be9acade44dba69ac2d4ecf02a19830b7c481927f WatchSource:0}: Error finding container 3b6384cf44e47813e4b7347be9acade44dba69ac2d4ecf02a19830b7c481927f: Status 404 returned error can't find the container with id 3b6384cf44e47813e4b7347be9acade44dba69ac2d4ecf02a19830b7c481927f Aug 04 10:45:32.469804 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:32.469681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:32.623246 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:32.623213 2576 generic.go:356] "Generic (PLEG): container finished" podID="8b47ea31-2115-4b13-b5ec-32087a7c52ee" containerID="337a60318895af99e575a01a651534f7069c184c774b6ca70e37e28b6f82c51f" exitCode=0 Aug 04 10:45:32.623423 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:32.623264 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" event={"ID":"8b47ea31-2115-4b13-b5ec-32087a7c52ee","Type":"ContainerDied","Data":"337a60318895af99e575a01a651534f7069c184c774b6ca70e37e28b6f82c51f"} Aug 04 10:45:32.623423 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:32.623300 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" event={"ID":"8b47ea31-2115-4b13-b5ec-32087a7c52ee","Type":"ContainerStarted","Data":"3b6384cf44e47813e4b7347be9acade44dba69ac2d4ecf02a19830b7c481927f"} Aug 04 10:45:33.630820 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:33.630787 2576 generic.go:356] "Generic (PLEG): container finished" podID="8b47ea31-2115-4b13-b5ec-32087a7c52ee" containerID="bb893b6f9ed7510cdb1a34c448590949e4f372e28bb8a14de26d36ca9d09d02d" exitCode=0 Aug 04 10:45:33.631246 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:33.630865 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" event={"ID":"8b47ea31-2115-4b13-b5ec-32087a7c52ee","Type":"ContainerDied","Data":"bb893b6f9ed7510cdb1a34c448590949e4f372e28bb8a14de26d36ca9d09d02d"} Aug 04 10:45:34.638392 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:34.638356 2576 generic.go:356] "Generic (PLEG): container finished" podID="8b47ea31-2115-4b13-b5ec-32087a7c52ee" containerID="4287a6a39d4f7f8b5356ac7b1a89cac5aeb202d3c7c634a832161597a9738b59" exitCode=0 Aug 04 10:45:34.638855 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:34.638403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" event={"ID":"8b47ea31-2115-4b13-b5ec-32087a7c52ee","Type":"ContainerDied","Data":"4287a6a39d4f7f8b5356ac7b1a89cac5aeb202d3c7c634a832161597a9738b59"} Aug 04 10:45:35.774735 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:35.774709 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" Aug 04 10:45:35.809563 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:35.809532 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b47ea31-2115-4b13-b5ec-32087a7c52ee-util\") pod \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\" (UID: \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\") " Aug 04 10:45:35.809763 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:35.809616 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7cpv\" (UniqueName: \"kubernetes.io/projected/8b47ea31-2115-4b13-b5ec-32087a7c52ee-kube-api-access-g7cpv\") pod \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\" (UID: \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\") " Aug 04 10:45:35.809763 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:35.809672 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b47ea31-2115-4b13-b5ec-32087a7c52ee-bundle\") pod \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\" (UID: \"8b47ea31-2115-4b13-b5ec-32087a7c52ee\") " Aug 04 10:45:35.810440 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:35.810407 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b47ea31-2115-4b13-b5ec-32087a7c52ee-bundle" (OuterVolumeSpecName: "bundle") pod "8b47ea31-2115-4b13-b5ec-32087a7c52ee" (UID: "8b47ea31-2115-4b13-b5ec-32087a7c52ee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:45:35.811768 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:35.811742 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b47ea31-2115-4b13-b5ec-32087a7c52ee-kube-api-access-g7cpv" (OuterVolumeSpecName: "kube-api-access-g7cpv") pod "8b47ea31-2115-4b13-b5ec-32087a7c52ee" (UID: "8b47ea31-2115-4b13-b5ec-32087a7c52ee"). InnerVolumeSpecName "kube-api-access-g7cpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:45:35.814982 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:35.814959 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b47ea31-2115-4b13-b5ec-32087a7c52ee-util" (OuterVolumeSpecName: "util") pod "8b47ea31-2115-4b13-b5ec-32087a7c52ee" (UID: "8b47ea31-2115-4b13-b5ec-32087a7c52ee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:45:35.911185 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:35.911093 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b47ea31-2115-4b13-b5ec-32087a7c52ee-util\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:45:35.911185 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:35.911127 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7cpv\" (UniqueName: \"kubernetes.io/projected/8b47ea31-2115-4b13-b5ec-32087a7c52ee-kube-api-access-g7cpv\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:45:35.911185 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:35.911137 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b47ea31-2115-4b13-b5ec-32087a7c52ee-bundle\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:45:36.649989 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:36.649950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" event={"ID":"8b47ea31-2115-4b13-b5ec-32087a7c52ee","Type":"ContainerDied","Data":"3b6384cf44e47813e4b7347be9acade44dba69ac2d4ecf02a19830b7c481927f"} Aug 04 10:45:36.649989 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:36.649969 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgsd2p" Aug 04 10:45:36.650204 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:36.650002 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b6384cf44e47813e4b7347be9acade44dba69ac2d4ecf02a19830b7c481927f" Aug 04 10:45:45.940031 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:45.939935 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:46.027148 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.027108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:46.562631 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.562592 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:46.782794 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.782759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:46.883806 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.883776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:46.899752 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.899723 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g"] Aug 04 10:45:46.900079 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.900067 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b47ea31-2115-4b13-b5ec-32087a7c52ee" containerName="extract" Aug 04 10:45:46.900122 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.900080 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b47ea31-2115-4b13-b5ec-32087a7c52ee" containerName="extract" Aug 04 10:45:46.900122 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.900098 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b47ea31-2115-4b13-b5ec-32087a7c52ee" containerName="pull" Aug 04 10:45:46.900122 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.900104 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b47ea31-2115-4b13-b5ec-32087a7c52ee" containerName="pull" Aug 04 10:45:46.900122 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.900120 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b47ea31-2115-4b13-b5ec-32087a7c52ee" containerName="util" Aug 04 10:45:46.900241 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.900126 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b47ea31-2115-4b13-b5ec-32087a7c52ee" containerName="util" Aug 04 10:45:46.900241 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.900175 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b47ea31-2115-4b13-b5ec-32087a7c52ee" containerName="extract" Aug 04 10:45:46.903393 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.903369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:46.905743 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.905716 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 10:45:46.905861 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.905756 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 10:45:46.906000 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.905985 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hzkt2\"" Aug 04 10:45:46.906000 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.905993 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 10:45:46.906128 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.906114 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 10:45:46.929200 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:46.929162 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g"] Aug 04 10:45:47.010920 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.010889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/554f951a-7a34-4831-b714-3e4a5d9f2528-tmp\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.011317 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.010952 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/554f951a-7a34-4831-b714-3e4a5d9f2528-manifests\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.011317 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.010974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/554f951a-7a34-4831-b714-3e4a5d9f2528-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.011317 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.011054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/554f951a-7a34-4831-b714-3e4a5d9f2528-webhook-cert\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.011317 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.011095 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blxz4\" (UniqueName: \"kubernetes.io/projected/554f951a-7a34-4831-b714-3e4a5d9f2528-kube-api-access-blxz4\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.111996 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.111892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/554f951a-7a34-4831-b714-3e4a5d9f2528-webhook-cert\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.112161 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.112036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blxz4\" (UniqueName: \"kubernetes.io/projected/554f951a-7a34-4831-b714-3e4a5d9f2528-kube-api-access-blxz4\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.112161 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.112071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/554f951a-7a34-4831-b714-3e4a5d9f2528-tmp\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.112161 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.112152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/554f951a-7a34-4831-b714-3e4a5d9f2528-manifests\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.112287 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.112178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/554f951a-7a34-4831-b714-3e4a5d9f2528-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.112616 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.112590 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/554f951a-7a34-4831-b714-3e4a5d9f2528-manifests\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.114382 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.114348 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/554f951a-7a34-4831-b714-3e4a5d9f2528-webhook-cert\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.114535 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.114362 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/554f951a-7a34-4831-b714-3e4a5d9f2528-tmp\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.114608 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.114573 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/554f951a-7a34-4831-b714-3e4a5d9f2528-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.120591 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.120570 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blxz4\" (UniqueName: \"kubernetes.io/projected/554f951a-7a34-4831-b714-3e4a5d9f2528-kube-api-access-blxz4\") pod \"opendatahub-operator-controller-manager-79895c7c95-vk94g\" (UID: \"554f951a-7a34-4831-b714-3e4a5d9f2528\") " pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.136130 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.136066 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:47.213324 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.213289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:47.228998 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.228954 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:47.330104 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.330069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:47.369305 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.369280 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g"] Aug 04 10:45:47.371898 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:45:47.371865 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554f951a_7a34_4831_b714_3e4a5d9f2528.slice/crio-3d35de27cdf79030bf0eed508122b863c8fa67b61ee5c8143c5af215d58a3ade WatchSource:0}: Error finding container 3d35de27cdf79030bf0eed508122b863c8fa67b61ee5c8143c5af215d58a3ade: Status 404 returned error can't find the container with id 3d35de27cdf79030bf0eed508122b863c8fa67b61ee5c8143c5af215d58a3ade Aug 04 10:45:47.710784 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:47.710749 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" event={"ID":"554f951a-7a34-4831-b714-3e4a5d9f2528","Type":"ContainerStarted","Data":"3d35de27cdf79030bf0eed508122b863c8fa67b61ee5c8143c5af215d58a3ade"} Aug 04 10:45:48.138449 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:48.138412 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:48.246528 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:48.246478 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:51.733741 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:51.733701 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" event={"ID":"554f951a-7a34-4831-b714-3e4a5d9f2528","Type":"ContainerStarted","Data":"c4925d03091f521cb8f839221a32dda67083c6f0bb2a02d86b4d747ee769d29d"} Aug 04 10:45:52.740014 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:52.739974 2576 generic.go:356] "Generic (PLEG): container finished" podID="554f951a-7a34-4831-b714-3e4a5d9f2528" containerID="c4925d03091f521cb8f839221a32dda67083c6f0bb2a02d86b4d747ee769d29d" exitCode=0 Aug 04 10:45:52.740412 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:52.740064 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" event={"ID":"554f951a-7a34-4831-b714-3e4a5d9f2528","Type":"ContainerDied","Data":"c4925d03091f521cb8f839221a32dda67083c6f0bb2a02d86b4d747ee769d29d"} Aug 04 10:45:54.751702 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:54.751665 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" event={"ID":"554f951a-7a34-4831-b714-3e4a5d9f2528","Type":"ContainerStarted","Data":"3a65ef55b4ff81a69cf21c2d47a03f3aab0ac12adafc61c31f01f8c0881980a1"} Aug 04 10:45:54.752151 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:54.751777 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:45:54.775482 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:54.775431 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" podStartSLOduration=3.267310515 podStartE2EDuration="8.775414659s" totalImagesPullingTime="4.377004735s" totalInitContainerRuntime="1.131099409s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:45:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:45:54.77290369 +0000 UTC m=+543.870697085" watchObservedRunningTime="2026-08-04 10:45:54.775414659 +0000 UTC m=+543.873208054" Aug 04 10:45:58.968731 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:58.968693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:45:59.072133 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:45:59.072099 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:02.688878 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:02.688847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:02.822291 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:02.822261 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:05.758933 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:05.758902 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-79895c7c95-vk94g" Aug 04 10:46:07.405165 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:07.405132 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:07.560060 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:07.560022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:07.955703 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:07.955673 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:08.022454 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.022420 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks"] Aug 04 10:46:08.031103 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.031077 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks" Aug 04 10:46:08.033388 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.033363 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 04 10:46:08.033388 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.033381 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 10:46:08.033732 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.033715 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8s9wk\"" Aug 04 10:46:08.038267 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.038246 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks"] Aug 04 10:46:08.182788 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.182750 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hvdmm"] Aug 04 10:46:08.186131 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.186112 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hvdmm" Aug 04 10:46:08.188713 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.188692 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Aug 04 10:46:08.188852 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.188752 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-q89qd\"" Aug 04 10:46:08.203232 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.203198 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hvdmm"] Aug 04 10:46:08.210463 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.210381 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks92s\" (UniqueName: \"kubernetes.io/projected/0267bd4e-d9ea-4efb-b847-5b0c9d8ae507-kube-api-access-ks92s\") pod \"kube-auth-proxy-6bc4f665b9-dmdks\" (UID: \"0267bd4e-d9ea-4efb-b847-5b0c9d8ae507\") " pod="openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks" Aug 04 10:46:08.210463 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.210452 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0267bd4e-d9ea-4efb-b847-5b0c9d8ae507-tmp\") pod \"kube-auth-proxy-6bc4f665b9-dmdks\" (UID: \"0267bd4e-d9ea-4efb-b847-5b0c9d8ae507\") " pod="openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks" Aug 04 10:46:08.210682 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.210568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0267bd4e-d9ea-4efb-b847-5b0c9d8ae507-tls-certs\") pod \"kube-auth-proxy-6bc4f665b9-dmdks\" (UID: \"0267bd4e-d9ea-4efb-b847-5b0c9d8ae507\") " pod="openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks" Aug 04 10:46:08.225825 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.225757 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:08.311544 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.311481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0267bd4e-d9ea-4efb-b847-5b0c9d8ae507-tmp\") pod \"kube-auth-proxy-6bc4f665b9-dmdks\" (UID: \"0267bd4e-d9ea-4efb-b847-5b0c9d8ae507\") " pod="openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks" Aug 04 10:46:08.311745 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.311583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8xm\" (UniqueName: \"kubernetes.io/projected/e7aef772-361d-453a-af5c-03e4b3a87619-kube-api-access-cb8xm\") pod \"odh-model-controller-7998c67dc4-hvdmm\" (UID: \"e7aef772-361d-453a-af5c-03e4b3a87619\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvdmm" Aug 04 10:46:08.311745 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.311613 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7aef772-361d-453a-af5c-03e4b3a87619-cert\") pod \"odh-model-controller-7998c67dc4-hvdmm\" (UID: \"e7aef772-361d-453a-af5c-03e4b3a87619\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvdmm" Aug 04 10:46:08.311745 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.311637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0267bd4e-d9ea-4efb-b847-5b0c9d8ae507-tls-certs\") pod \"kube-auth-proxy-6bc4f665b9-dmdks\" (UID: \"0267bd4e-d9ea-4efb-b847-5b0c9d8ae507\") " pod="openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks" Aug 04 10:46:08.311745 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.311722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks92s\" (UniqueName: \"kubernetes.io/projected/0267bd4e-d9ea-4efb-b847-5b0c9d8ae507-kube-api-access-ks92s\") pod \"kube-auth-proxy-6bc4f665b9-dmdks\" (UID: \"0267bd4e-d9ea-4efb-b847-5b0c9d8ae507\") " pod="openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks" Aug 04 10:46:08.313833 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.313806 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0267bd4e-d9ea-4efb-b847-5b0c9d8ae507-tmp\") pod \"kube-auth-proxy-6bc4f665b9-dmdks\" (UID: \"0267bd4e-d9ea-4efb-b847-5b0c9d8ae507\") " pod="openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks" Aug 04 10:46:08.314077 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.314058 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0267bd4e-d9ea-4efb-b847-5b0c9d8ae507-tls-certs\") pod \"kube-auth-proxy-6bc4f665b9-dmdks\" (UID: \"0267bd4e-d9ea-4efb-b847-5b0c9d8ae507\") " pod="openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks" Aug 04 10:46:08.320613 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.320586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:08.321397 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.321375 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks92s\" (UniqueName: \"kubernetes.io/projected/0267bd4e-d9ea-4efb-b847-5b0c9d8ae507-kube-api-access-ks92s\") pod \"kube-auth-proxy-6bc4f665b9-dmdks\" (UID: \"0267bd4e-d9ea-4efb-b847-5b0c9d8ae507\") " pod="openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks" Aug 04 10:46:08.344103 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.344061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks" Aug 04 10:46:08.412481 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.412446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7aef772-361d-453a-af5c-03e4b3a87619-cert\") pod \"odh-model-controller-7998c67dc4-hvdmm\" (UID: \"e7aef772-361d-453a-af5c-03e4b3a87619\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvdmm" Aug 04 10:46:08.412963 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:46:08.412642 2576 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Aug 04 10:46:08.412963 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:46:08.412737 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7aef772-361d-453a-af5c-03e4b3a87619-cert podName:e7aef772-361d-453a-af5c-03e4b3a87619 nodeName:}" failed. No retries permitted until 2026-08-04 10:46:08.912714832 +0000 UTC m=+558.010508219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e7aef772-361d-453a-af5c-03e4b3a87619-cert") pod "odh-model-controller-7998c67dc4-hvdmm" (UID: "e7aef772-361d-453a-af5c-03e4b3a87619") : secret "odh-model-controller-webhook-cert" not found Aug 04 10:46:08.412963 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.412648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8xm\" (UniqueName: \"kubernetes.io/projected/e7aef772-361d-453a-af5c-03e4b3a87619-kube-api-access-cb8xm\") pod \"odh-model-controller-7998c67dc4-hvdmm\" (UID: \"e7aef772-361d-453a-af5c-03e4b3a87619\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvdmm" Aug 04 10:46:08.421336 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.421304 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8xm\" (UniqueName: \"kubernetes.io/projected/e7aef772-361d-453a-af5c-03e4b3a87619-kube-api-access-cb8xm\") pod \"odh-model-controller-7998c67dc4-hvdmm\" (UID: \"e7aef772-361d-453a-af5c-03e4b3a87619\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvdmm" Aug 04 10:46:08.424004 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.423977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:08.505013 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.504986 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks"] Aug 04 10:46:08.506933 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:46:08.506909 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0267bd4e_d9ea_4efb_b847_5b0c9d8ae507.slice/crio-dc0559806887bab801b80434e0acdf9106938f6f6550bced505ad5d5fc271d17 WatchSource:0}: Error finding container dc0559806887bab801b80434e0acdf9106938f6f6550bced505ad5d5fc271d17: Status 404 returned error can't find the container with id dc0559806887bab801b80434e0acdf9106938f6f6550bced505ad5d5fc271d17 Aug 04 10:46:08.744124 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.744087 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:08.828298 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.828215 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks" event={"ID":"0267bd4e-d9ea-4efb-b847-5b0c9d8ae507","Type":"ContainerStarted","Data":"dc0559806887bab801b80434e0acdf9106938f6f6550bced505ad5d5fc271d17"} Aug 04 10:46:08.917875 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.917835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7aef772-361d-453a-af5c-03e4b3a87619-cert\") pod \"odh-model-controller-7998c67dc4-hvdmm\" (UID: \"e7aef772-361d-453a-af5c-03e4b3a87619\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvdmm" Aug 04 10:46:08.920279 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:08.920248 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7aef772-361d-453a-af5c-03e4b3a87619-cert\") pod \"odh-model-controller-7998c67dc4-hvdmm\" (UID: \"e7aef772-361d-453a-af5c-03e4b3a87619\") " pod="opendatahub/odh-model-controller-7998c67dc4-hvdmm" Aug 04 10:46:09.098481 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.098447 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hvdmm" Aug 04 10:46:09.109696 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.109664 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:09.218991 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.218954 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:09.225649 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.225617 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q"] Aug 04 10:46:09.230885 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.230860 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:09.233598 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.233574 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 04 10:46:09.233598 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.233591 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zlsvd\"" Aug 04 10:46:09.233773 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.233585 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 04 10:46:09.236444 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.236420 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 04 10:46:09.242223 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.242198 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q"] Aug 04 10:46:09.252303 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.252263 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hvdmm"] Aug 04 10:46:09.255242 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:46:09.255213 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7aef772_361d_453a_af5c_03e4b3a87619.slice/crio-00f806c676fbb769caac986c9fe9444dd2219c0b19bc642b68ea0649097d635a WatchSource:0}: Error finding container 00f806c676fbb769caac986c9fe9444dd2219c0b19bc642b68ea0649097d635a: Status 404 returned error can't find the container with id 00f806c676fbb769caac986c9fe9444dd2219c0b19bc642b68ea0649097d635a Aug 04 10:46:09.317236 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.317199 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:09.423411 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.423327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2178a1c3-6a02-4bb9-9a67-c59e239888c8-cert\") pod \"lws-controller-manager-5b459fcc7c-k9r7q\" (UID: \"2178a1c3-6a02-4bb9-9a67-c59e239888c8\") " pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:09.423792 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.423387 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2178a1c3-6a02-4bb9-9a67-c59e239888c8-metrics-cert\") pod \"lws-controller-manager-5b459fcc7c-k9r7q\" (UID: \"2178a1c3-6a02-4bb9-9a67-c59e239888c8\") " pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:09.423792 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.423439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2178a1c3-6a02-4bb9-9a67-c59e239888c8-manager-config\") pod \"lws-controller-manager-5b459fcc7c-k9r7q\" (UID: \"2178a1c3-6a02-4bb9-9a67-c59e239888c8\") " pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:09.423792 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.423552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm92t\" (UniqueName: \"kubernetes.io/projected/2178a1c3-6a02-4bb9-9a67-c59e239888c8-kube-api-access-jm92t\") pod \"lws-controller-manager-5b459fcc7c-k9r7q\" (UID: \"2178a1c3-6a02-4bb9-9a67-c59e239888c8\") " pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:09.458644 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.458605 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:09.524379 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.524337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2178a1c3-6a02-4bb9-9a67-c59e239888c8-cert\") pod \"lws-controller-manager-5b459fcc7c-k9r7q\" (UID: \"2178a1c3-6a02-4bb9-9a67-c59e239888c8\") " pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:09.524585 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.524404 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2178a1c3-6a02-4bb9-9a67-c59e239888c8-metrics-cert\") pod \"lws-controller-manager-5b459fcc7c-k9r7q\" (UID: \"2178a1c3-6a02-4bb9-9a67-c59e239888c8\") " pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:09.524585 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.524431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2178a1c3-6a02-4bb9-9a67-c59e239888c8-manager-config\") pod \"lws-controller-manager-5b459fcc7c-k9r7q\" (UID: \"2178a1c3-6a02-4bb9-9a67-c59e239888c8\") " pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:09.524585 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.524478 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm92t\" (UniqueName: \"kubernetes.io/projected/2178a1c3-6a02-4bb9-9a67-c59e239888c8-kube-api-access-jm92t\") pod \"lws-controller-manager-5b459fcc7c-k9r7q\" (UID: \"2178a1c3-6a02-4bb9-9a67-c59e239888c8\") " pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:09.525410 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.525381 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2178a1c3-6a02-4bb9-9a67-c59e239888c8-manager-config\") pod \"lws-controller-manager-5b459fcc7c-k9r7q\" (UID: \"2178a1c3-6a02-4bb9-9a67-c59e239888c8\") " pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:09.527373 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.527350 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2178a1c3-6a02-4bb9-9a67-c59e239888c8-metrics-cert\") pod \"lws-controller-manager-5b459fcc7c-k9r7q\" (UID: \"2178a1c3-6a02-4bb9-9a67-c59e239888c8\") " pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:09.527682 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.527662 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2178a1c3-6a02-4bb9-9a67-c59e239888c8-cert\") pod \"lws-controller-manager-5b459fcc7c-k9r7q\" (UID: \"2178a1c3-6a02-4bb9-9a67-c59e239888c8\") " pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:09.549389 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.549349 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm92t\" (UniqueName: \"kubernetes.io/projected/2178a1c3-6a02-4bb9-9a67-c59e239888c8-kube-api-access-jm92t\") pod \"lws-controller-manager-5b459fcc7c-k9r7q\" (UID: \"2178a1c3-6a02-4bb9-9a67-c59e239888c8\") " pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:09.582150 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.582114 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:09.753022 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.752939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:09.834602 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.834565 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hvdmm" event={"ID":"e7aef772-361d-453a-af5c-03e4b3a87619","Type":"ContainerStarted","Data":"00f806c676fbb769caac986c9fe9444dd2219c0b19bc642b68ea0649097d635a"} Aug 04 10:46:09.842954 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:09.842934 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:10.023079 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:10.023038 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q"] Aug 04 10:46:10.027489 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:46:10.026933 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2178a1c3_6a02_4bb9_9a67_c59e239888c8.slice/crio-d4ddf7651b1b69688528f975f9120171c449389acc3e9483937eba078c5ebcc2 WatchSource:0}: Error finding container d4ddf7651b1b69688528f975f9120171c449389acc3e9483937eba078c5ebcc2: Status 404 returned error can't find the container with id d4ddf7651b1b69688528f975f9120171c449389acc3e9483937eba078c5ebcc2 Aug 04 10:46:10.825382 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:10.825298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:10.844875 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:10.844834 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" event={"ID":"2178a1c3-6a02-4bb9-9a67-c59e239888c8","Type":"ContainerStarted","Data":"d4ddf7651b1b69688528f975f9120171c449389acc3e9483937eba078c5ebcc2"} Aug 04 10:46:13.866297 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:13.866258 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks" event={"ID":"0267bd4e-d9ea-4efb-b847-5b0c9d8ae507","Type":"ContainerStarted","Data":"758417d0ff1f2e32cd1888048c87f0868b45f02bd9b226f55858c1b7a4b17696"} Aug 04 10:46:13.867974 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:13.867938 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hvdmm" event={"ID":"e7aef772-361d-453a-af5c-03e4b3a87619","Type":"ContainerStarted","Data":"74898293b14bd0ad6e603b0961651342ad4b91db180ff28d041be5e4f8a9130b"} Aug 04 10:46:13.868146 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:13.867998 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-hvdmm" Aug 04 10:46:13.869622 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:13.869586 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" event={"ID":"2178a1c3-6a02-4bb9-9a67-c59e239888c8","Type":"ContainerStarted","Data":"093a76a1138f7e3beb5d43bc04373c46034b1990f7e7cf43c4866071fa8e982f"} Aug 04 10:46:13.869744 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:13.869664 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:13.883031 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:13.882977 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6bc4f665b9-dmdks" podStartSLOduration=0.695743919 podStartE2EDuration="5.88295899s" totalImagesPullingTime="5.187215071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:46:13.881224597 +0000 UTC m=+562.979017994" watchObservedRunningTime="2026-08-04 10:46:13.88295899 +0000 UTC m=+562.980752385" Aug 04 10:46:13.896662 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:13.896601 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-hvdmm" podStartSLOduration=1.452420631 podStartE2EDuration="5.896580347s" totalImagesPullingTime="4.444159716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:46:13.895369697 +0000 UTC m=+562.993163093" watchObservedRunningTime="2026-08-04 10:46:13.896580347 +0000 UTC m=+562.994373744" Aug 04 10:46:13.915724 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:13.915625 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" podStartSLOduration=1.246106588 podStartE2EDuration="4.915609966s" totalImagesPullingTime="3.669503378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:46:13.914519046 +0000 UTC m=+563.012312442" watchObservedRunningTime="2026-08-04 10:46:13.915609966 +0000 UTC m=+563.013403361" Aug 04 10:46:15.904310 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:15.904281 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:15.995252 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:15.995215 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:16.776462 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:16.776425 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:17.073172 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:17.073094 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:17.168652 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:17.168618 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:19.165140 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:19.165106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:19.272181 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:19.272146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:19.523740 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:19.523658 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:19.628112 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:19.628081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:20.664948 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:20.664896 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:20.785766 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:20.785737 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:46:24.876934 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:24.876901 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b459fcc7c-k9r7q" Aug 04 10:46:24.877377 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:24.877074 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-hvdmm" Aug 04 10:46:34.918984 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:46:34.918942 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:09.184937 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:09.184902 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:09.285414 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:09.285368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:13.182121 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.182086 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv"] Aug 04 10:47:13.184967 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.184944 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" Aug 04 10:47:13.187398 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.187377 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 10:47:13.187531 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.187497 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n59pt\"" Aug 04 10:47:13.188185 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.188170 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 10:47:13.195063 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.195043 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv"] Aug 04 10:47:13.280678 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.280605 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7"] Aug 04 10:47:13.281190 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.281168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv\" (UID: \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" Aug 04 10:47:13.281233 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.281202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lwsr\" (UniqueName: \"kubernetes.io/projected/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-kube-api-access-4lwsr\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv\" (UID: \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" Aug 04 10:47:13.281268 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.281239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv\" (UID: \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" Aug 04 10:47:13.283225 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.283209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" Aug 04 10:47:13.293353 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.293328 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7"] Aug 04 10:47:13.381709 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.381673 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv\" (UID: \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" Aug 04 10:47:13.381925 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.381726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lwsr\" (UniqueName: \"kubernetes.io/projected/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-kube-api-access-4lwsr\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv\" (UID: \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" Aug 04 10:47:13.381925 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.381781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv\" (UID: \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" Aug 04 10:47:13.381925 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.381871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94bd4f50-2139-4e73-9fbc-0034074dba89-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7\" (UID: \"94bd4f50-2139-4e73-9fbc-0034074dba89\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" Aug 04 10:47:13.381925 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.381900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94bd4f50-2139-4e73-9fbc-0034074dba89-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7\" (UID: \"94bd4f50-2139-4e73-9fbc-0034074dba89\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" Aug 04 10:47:13.382112 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.381929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kks8\" (UniqueName: \"kubernetes.io/projected/94bd4f50-2139-4e73-9fbc-0034074dba89-kube-api-access-2kks8\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7\" (UID: \"94bd4f50-2139-4e73-9fbc-0034074dba89\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" Aug 04 10:47:13.382170 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.382147 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv\" (UID: \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" Aug 04 10:47:13.382204 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.382147 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv\" (UID: \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" Aug 04 10:47:13.388800 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.388769 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb"] Aug 04 10:47:13.391414 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.391387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lwsr\" (UniqueName: \"kubernetes.io/projected/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-kube-api-access-4lwsr\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv\" (UID: \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" Aug 04 10:47:13.391553 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.391452 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" Aug 04 10:47:13.400807 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.400781 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb"] Aug 04 10:47:13.483321 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.483233 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64vkp\" (UniqueName: \"kubernetes.io/projected/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-kube-api-access-64vkp\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb\" (UID: \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" Aug 04 10:47:13.483321 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.483298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94bd4f50-2139-4e73-9fbc-0034074dba89-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7\" (UID: \"94bd4f50-2139-4e73-9fbc-0034074dba89\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" Aug 04 10:47:13.483592 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.483366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94bd4f50-2139-4e73-9fbc-0034074dba89-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7\" (UID: \"94bd4f50-2139-4e73-9fbc-0034074dba89\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" Aug 04 10:47:13.483592 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.483395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kks8\" (UniqueName: \"kubernetes.io/projected/94bd4f50-2139-4e73-9fbc-0034074dba89-kube-api-access-2kks8\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7\" (UID: \"94bd4f50-2139-4e73-9fbc-0034074dba89\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" Aug 04 10:47:13.483592 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.483437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb\" (UID: \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" Aug 04 10:47:13.483592 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.483466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb\" (UID: \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" Aug 04 10:47:13.483806 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.483697 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94bd4f50-2139-4e73-9fbc-0034074dba89-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7\" (UID: \"94bd4f50-2139-4e73-9fbc-0034074dba89\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" Aug 04 10:47:13.483806 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.483758 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94bd4f50-2139-4e73-9fbc-0034074dba89-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7\" (UID: \"94bd4f50-2139-4e73-9fbc-0034074dba89\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" Aug 04 10:47:13.487603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.487580 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr"] Aug 04 10:47:13.490561 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.490545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" Aug 04 10:47:13.491892 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.491871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kks8\" (UniqueName: \"kubernetes.io/projected/94bd4f50-2139-4e73-9fbc-0034074dba89-kube-api-access-2kks8\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7\" (UID: \"94bd4f50-2139-4e73-9fbc-0034074dba89\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" Aug 04 10:47:13.495738 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.495720 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" Aug 04 10:47:13.499156 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.499138 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr"] Aug 04 10:47:13.584893 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.584144 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64vkp\" (UniqueName: \"kubernetes.io/projected/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-kube-api-access-64vkp\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb\" (UID: \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" Aug 04 10:47:13.584893 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.584192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfg9f\" (UniqueName: \"kubernetes.io/projected/ad7d3373-7879-45bc-8e26-93ac88ec9671-kube-api-access-pfg9f\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr\" (UID: \"ad7d3373-7879-45bc-8e26-93ac88ec9671\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" Aug 04 10:47:13.584893 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.584241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad7d3373-7879-45bc-8e26-93ac88ec9671-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr\" (UID: \"ad7d3373-7879-45bc-8e26-93ac88ec9671\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" Aug 04 10:47:13.584893 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.584285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb\" (UID: \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" Aug 04 10:47:13.584893 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.584309 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad7d3373-7879-45bc-8e26-93ac88ec9671-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr\" (UID: \"ad7d3373-7879-45bc-8e26-93ac88ec9671\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" Aug 04 10:47:13.584893 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.584343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb\" (UID: \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" Aug 04 10:47:13.584893 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.584822 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb\" (UID: \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" Aug 04 10:47:13.585251 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.584949 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb\" (UID: \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" Aug 04 10:47:13.592833 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.592805 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" Aug 04 10:47:13.593307 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.593282 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64vkp\" (UniqueName: \"kubernetes.io/projected/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-kube-api-access-64vkp\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb\" (UID: \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" Aug 04 10:47:13.637856 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.637822 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv"] Aug 04 10:47:13.638677 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:47:13.638643 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eeced55_bf8d_462a_bda6_3fda40dd5b0b.slice/crio-9ef295dca7a162b3531a97a64dc4fb6e063c3b1974736def5c4f8c7b5069261a WatchSource:0}: Error finding container 9ef295dca7a162b3531a97a64dc4fb6e063c3b1974736def5c4f8c7b5069261a: Status 404 returned error can't find the container with id 9ef295dca7a162b3531a97a64dc4fb6e063c3b1974736def5c4f8c7b5069261a Aug 04 10:47:13.685651 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.685616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfg9f\" (UniqueName: \"kubernetes.io/projected/ad7d3373-7879-45bc-8e26-93ac88ec9671-kube-api-access-pfg9f\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr\" (UID: \"ad7d3373-7879-45bc-8e26-93ac88ec9671\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" Aug 04 10:47:13.685798 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.685683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad7d3373-7879-45bc-8e26-93ac88ec9671-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr\" (UID: \"ad7d3373-7879-45bc-8e26-93ac88ec9671\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" Aug 04 10:47:13.685798 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.685731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad7d3373-7879-45bc-8e26-93ac88ec9671-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr\" (UID: \"ad7d3373-7879-45bc-8e26-93ac88ec9671\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" Aug 04 10:47:13.686150 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.686127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad7d3373-7879-45bc-8e26-93ac88ec9671-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr\" (UID: \"ad7d3373-7879-45bc-8e26-93ac88ec9671\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" Aug 04 10:47:13.686227 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.686182 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad7d3373-7879-45bc-8e26-93ac88ec9671-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr\" (UID: \"ad7d3373-7879-45bc-8e26-93ac88ec9671\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" Aug 04 10:47:13.695421 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.695391 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfg9f\" (UniqueName: \"kubernetes.io/projected/ad7d3373-7879-45bc-8e26-93ac88ec9671-kube-api-access-pfg9f\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr\" (UID: \"ad7d3373-7879-45bc-8e26-93ac88ec9671\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" Aug 04 10:47:13.708909 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.708877 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" Aug 04 10:47:13.737284 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.737256 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7"] Aug 04 10:47:13.739295 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:47:13.739268 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bd4f50_2139_4e73_9fbc_0034074dba89.slice/crio-67e88776225f68b812ce1d5f378dc6a6eb87070324ac62b542d6c2bbe5c7c4fb WatchSource:0}: Error finding container 67e88776225f68b812ce1d5f378dc6a6eb87070324ac62b542d6c2bbe5c7c4fb: Status 404 returned error can't find the container with id 67e88776225f68b812ce1d5f378dc6a6eb87070324ac62b542d6c2bbe5c7c4fb Aug 04 10:47:13.830708 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.830683 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" Aug 04 10:47:13.856068 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:47:13.855973 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b3f072_2ac1_4a89_b7a9_d14a01f06f77.slice/crio-63aef7560995510f4f066eca00404fd3832cbd351e7b268cf8fbb0f509ed179e WatchSource:0}: Error finding container 63aef7560995510f4f066eca00404fd3832cbd351e7b268cf8fbb0f509ed179e: Status 404 returned error can't find the container with id 63aef7560995510f4f066eca00404fd3832cbd351e7b268cf8fbb0f509ed179e Aug 04 10:47:13.856197 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.856078 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb"] Aug 04 10:47:13.983810 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:13.983781 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr"] Aug 04 10:47:14.027560 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:47:14.027454 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7d3373_7879_45bc_8e26_93ac88ec9671.slice/crio-34227add06d804dba3aa000debecef8e9d8d0389b1779286707386cf43e48013 WatchSource:0}: Error finding container 34227add06d804dba3aa000debecef8e9d8d0389b1779286707386cf43e48013: Status 404 returned error can't find the container with id 34227add06d804dba3aa000debecef8e9d8d0389b1779286707386cf43e48013 Aug 04 10:47:14.163468 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:14.163434 2576 generic.go:356] "Generic (PLEG): container finished" podID="0eeced55-bf8d-462a-bda6-3fda40dd5b0b" containerID="9bfdb57129205d0fab0c4b229b40157f1d6d80f889d7235c8faf37f0c47472f9" exitCode=0 Aug 04 10:47:14.163643 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:14.163540 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" event={"ID":"0eeced55-bf8d-462a-bda6-3fda40dd5b0b","Type":"ContainerDied","Data":"9bfdb57129205d0fab0c4b229b40157f1d6d80f889d7235c8faf37f0c47472f9"} Aug 04 10:47:14.163643 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:14.163594 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" event={"ID":"0eeced55-bf8d-462a-bda6-3fda40dd5b0b","Type":"ContainerStarted","Data":"9ef295dca7a162b3531a97a64dc4fb6e063c3b1974736def5c4f8c7b5069261a"} Aug 04 10:47:14.165340 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:14.165317 2576 generic.go:356] "Generic (PLEG): container finished" podID="ad7d3373-7879-45bc-8e26-93ac88ec9671" containerID="846a31ad917840beab191eb171f516c545fa4b5ec493d7ea0dd2dba08e2c02d9" exitCode=0 Aug 04 10:47:14.165421 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:14.165373 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" event={"ID":"ad7d3373-7879-45bc-8e26-93ac88ec9671","Type":"ContainerDied","Data":"846a31ad917840beab191eb171f516c545fa4b5ec493d7ea0dd2dba08e2c02d9"} Aug 04 10:47:14.165561 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:14.165422 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" event={"ID":"ad7d3373-7879-45bc-8e26-93ac88ec9671","Type":"ContainerStarted","Data":"34227add06d804dba3aa000debecef8e9d8d0389b1779286707386cf43e48013"} Aug 04 10:47:14.167126 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:14.167103 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2b3f072-2ac1-4a89-b7a9-d14a01f06f77" containerID="66e7ac82682e4a1a283ac98c0435dac1ddcaf929a94ef3974454e74946b6bae4" exitCode=0 Aug 04 10:47:14.167229 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:14.167188 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" event={"ID":"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77","Type":"ContainerDied","Data":"66e7ac82682e4a1a283ac98c0435dac1ddcaf929a94ef3974454e74946b6bae4"} Aug 04 10:47:14.167291 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:14.167232 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" event={"ID":"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77","Type":"ContainerStarted","Data":"63aef7560995510f4f066eca00404fd3832cbd351e7b268cf8fbb0f509ed179e"} Aug 04 10:47:14.168848 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:14.168830 2576 generic.go:356] "Generic (PLEG): container finished" podID="94bd4f50-2139-4e73-9fbc-0034074dba89" containerID="159139d1c5503a208f9bc35b4bdbd4fc0aeb9c2ae12bb4ce6c351559369d461b" exitCode=0 Aug 04 10:47:14.168925 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:14.168855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" event={"ID":"94bd4f50-2139-4e73-9fbc-0034074dba89","Type":"ContainerDied","Data":"159139d1c5503a208f9bc35b4bdbd4fc0aeb9c2ae12bb4ce6c351559369d461b"} Aug 04 10:47:14.168925 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:14.168898 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" event={"ID":"94bd4f50-2139-4e73-9fbc-0034074dba89","Type":"ContainerStarted","Data":"67e88776225f68b812ce1d5f378dc6a6eb87070324ac62b542d6c2bbe5c7c4fb"} Aug 04 10:47:15.176180 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:15.176137 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2b3f072-2ac1-4a89-b7a9-d14a01f06f77" containerID="c2165c405f9db3a4cb4b6e961cd3ff2a15b635758dce30338e14015b1e69cc30" exitCode=0 Aug 04 10:47:15.176615 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:15.176214 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" event={"ID":"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77","Type":"ContainerDied","Data":"c2165c405f9db3a4cb4b6e961cd3ff2a15b635758dce30338e14015b1e69cc30"} Aug 04 10:47:15.178224 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:15.178203 2576 generic.go:356] "Generic (PLEG): container finished" podID="0eeced55-bf8d-462a-bda6-3fda40dd5b0b" containerID="5dfcba2e96338a690b496dee766d3e70ac9b0509d86c0a40d865576957c44a99" exitCode=0 Aug 04 10:47:15.178330 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:15.178238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" event={"ID":"0eeced55-bf8d-462a-bda6-3fda40dd5b0b","Type":"ContainerDied","Data":"5dfcba2e96338a690b496dee766d3e70ac9b0509d86c0a40d865576957c44a99"} Aug 04 10:47:16.185742 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:16.185705 2576 generic.go:356] "Generic (PLEG): container finished" podID="0eeced55-bf8d-462a-bda6-3fda40dd5b0b" containerID="89eb77afe5c2767af95844b2e8b778b1a9991731f5058753413ba485578bfc29" exitCode=0 Aug 04 10:47:16.186182 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:16.185803 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" event={"ID":"0eeced55-bf8d-462a-bda6-3fda40dd5b0b","Type":"ContainerDied","Data":"89eb77afe5c2767af95844b2e8b778b1a9991731f5058753413ba485578bfc29"} Aug 04 10:47:16.187539 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:16.187497 2576 generic.go:356] "Generic (PLEG): container finished" podID="ad7d3373-7879-45bc-8e26-93ac88ec9671" containerID="5fbce2c69901324a64c400148a65d525090f117711e21157ff167349a5547d70" exitCode=0 Aug 04 10:47:16.187637 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:16.187591 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" event={"ID":"ad7d3373-7879-45bc-8e26-93ac88ec9671","Type":"ContainerDied","Data":"5fbce2c69901324a64c400148a65d525090f117711e21157ff167349a5547d70"} Aug 04 10:47:16.189572 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:16.189551 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2b3f072-2ac1-4a89-b7a9-d14a01f06f77" containerID="238d671b8aefac00105bb8b342fbeb7805ff0b1cb9e6239e3ed0201483028398" exitCode=0 Aug 04 10:47:16.189707 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:16.189615 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" event={"ID":"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77","Type":"ContainerDied","Data":"238d671b8aefac00105bb8b342fbeb7805ff0b1cb9e6239e3ed0201483028398"} Aug 04 10:47:16.191364 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:16.191346 2576 generic.go:356] "Generic (PLEG): container finished" podID="94bd4f50-2139-4e73-9fbc-0034074dba89" containerID="d5d900aafe8fb9c3f735c4dbdda555f3cd1476deeffe7eb80373a7c5d7627409" exitCode=0 Aug 04 10:47:16.191439 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:16.191397 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" event={"ID":"94bd4f50-2139-4e73-9fbc-0034074dba89","Type":"ContainerDied","Data":"d5d900aafe8fb9c3f735c4dbdda555f3cd1476deeffe7eb80373a7c5d7627409"} Aug 04 10:47:17.199439 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.199403 2576 generic.go:356] "Generic (PLEG): container finished" podID="ad7d3373-7879-45bc-8e26-93ac88ec9671" containerID="b1fd2a9b74c2bc66c4bc4881970714b37c8fff3e20c7d9bf1ec639eeabb00ac0" exitCode=0 Aug 04 10:47:17.199891 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.199496 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" event={"ID":"ad7d3373-7879-45bc-8e26-93ac88ec9671","Type":"ContainerDied","Data":"b1fd2a9b74c2bc66c4bc4881970714b37c8fff3e20c7d9bf1ec639eeabb00ac0"} Aug 04 10:47:17.201573 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.201551 2576 generic.go:356] "Generic (PLEG): container finished" podID="94bd4f50-2139-4e73-9fbc-0034074dba89" containerID="a17f2c7793ff7c18e736000d6ee173b0580a741f94827a6f3757dc7746cf1d6e" exitCode=0 Aug 04 10:47:17.201684 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.201631 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" event={"ID":"94bd4f50-2139-4e73-9fbc-0034074dba89","Type":"ContainerDied","Data":"a17f2c7793ff7c18e736000d6ee173b0580a741f94827a6f3757dc7746cf1d6e"} Aug 04 10:47:17.337413 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.337386 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" Aug 04 10:47:17.382646 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.382618 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" Aug 04 10:47:17.421673 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.421640 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-util\") pod \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\" (UID: \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\") " Aug 04 10:47:17.421854 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.421694 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-bundle\") pod \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\" (UID: \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\") " Aug 04 10:47:17.421854 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.421736 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-util\") pod \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\" (UID: \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\") " Aug 04 10:47:17.421854 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.421754 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64vkp\" (UniqueName: \"kubernetes.io/projected/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-kube-api-access-64vkp\") pod \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\" (UID: \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\") " Aug 04 10:47:17.421854 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.421781 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-bundle\") pod \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\" (UID: \"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77\") " Aug 04 10:47:17.422073 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.421873 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lwsr\" (UniqueName: \"kubernetes.io/projected/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-kube-api-access-4lwsr\") pod \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\" (UID: \"0eeced55-bf8d-462a-bda6-3fda40dd5b0b\") " Aug 04 10:47:17.422529 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.422460 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-bundle" (OuterVolumeSpecName: "bundle") pod "0eeced55-bf8d-462a-bda6-3fda40dd5b0b" (UID: "0eeced55-bf8d-462a-bda6-3fda40dd5b0b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:47:17.422643 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.422546 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-bundle" (OuterVolumeSpecName: "bundle") pod "a2b3f072-2ac1-4a89-b7a9-d14a01f06f77" (UID: "a2b3f072-2ac1-4a89-b7a9-d14a01f06f77"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:47:17.424294 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.424264 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-kube-api-access-64vkp" (OuterVolumeSpecName: "kube-api-access-64vkp") pod "a2b3f072-2ac1-4a89-b7a9-d14a01f06f77" (UID: "a2b3f072-2ac1-4a89-b7a9-d14a01f06f77"). InnerVolumeSpecName "kube-api-access-64vkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:47:17.424573 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.424551 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-kube-api-access-4lwsr" (OuterVolumeSpecName: "kube-api-access-4lwsr") pod "0eeced55-bf8d-462a-bda6-3fda40dd5b0b" (UID: "0eeced55-bf8d-462a-bda6-3fda40dd5b0b"). InnerVolumeSpecName "kube-api-access-4lwsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:47:17.428774 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.428750 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-util" (OuterVolumeSpecName: "util") pod "a2b3f072-2ac1-4a89-b7a9-d14a01f06f77" (UID: "a2b3f072-2ac1-4a89-b7a9-d14a01f06f77"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:47:17.428914 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.428892 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-util" (OuterVolumeSpecName: "util") pod "0eeced55-bf8d-462a-bda6-3fda40dd5b0b" (UID: "0eeced55-bf8d-462a-bda6-3fda40dd5b0b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:47:17.522823 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.522738 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lwsr\" (UniqueName: \"kubernetes.io/projected/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-kube-api-access-4lwsr\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:47:17.522823 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.522770 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-util\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:47:17.522823 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.522779 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0eeced55-bf8d-462a-bda6-3fda40dd5b0b-bundle\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:47:17.522823 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.522788 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-util\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:47:17.522823 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.522799 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64vkp\" (UniqueName: \"kubernetes.io/projected/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-kube-api-access-64vkp\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:47:17.522823 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:17.522809 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2b3f072-2ac1-4a89-b7a9-d14a01f06f77-bundle\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:47:18.209253 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.209223 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" Aug 04 10:47:18.209253 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.209249 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539cmnv" event={"ID":"0eeced55-bf8d-462a-bda6-3fda40dd5b0b","Type":"ContainerDied","Data":"9ef295dca7a162b3531a97a64dc4fb6e063c3b1974736def5c4f8c7b5069261a"} Aug 04 10:47:18.209776 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.209296 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ef295dca7a162b3531a97a64dc4fb6e063c3b1974736def5c4f8c7b5069261a" Aug 04 10:47:18.211198 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.211178 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" Aug 04 10:47:18.211349 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.211212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9pxpb" event={"ID":"a2b3f072-2ac1-4a89-b7a9-d14a01f06f77","Type":"ContainerDied","Data":"63aef7560995510f4f066eca00404fd3832cbd351e7b268cf8fbb0f509ed179e"} Aug 04 10:47:18.211349 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.211258 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63aef7560995510f4f066eca00404fd3832cbd351e7b268cf8fbb0f509ed179e" Aug 04 10:47:18.350513 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.350474 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" Aug 04 10:47:18.376265 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.376241 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" Aug 04 10:47:18.431718 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.431669 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfg9f\" (UniqueName: \"kubernetes.io/projected/ad7d3373-7879-45bc-8e26-93ac88ec9671-kube-api-access-pfg9f\") pod \"ad7d3373-7879-45bc-8e26-93ac88ec9671\" (UID: \"ad7d3373-7879-45bc-8e26-93ac88ec9671\") " Aug 04 10:47:18.431718 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.431720 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad7d3373-7879-45bc-8e26-93ac88ec9671-util\") pod \"ad7d3373-7879-45bc-8e26-93ac88ec9671\" (UID: \"ad7d3373-7879-45bc-8e26-93ac88ec9671\") " Aug 04 10:47:18.432011 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.431750 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad7d3373-7879-45bc-8e26-93ac88ec9671-bundle\") pod \"ad7d3373-7879-45bc-8e26-93ac88ec9671\" (UID: \"ad7d3373-7879-45bc-8e26-93ac88ec9671\") " Aug 04 10:47:18.432011 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.431787 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94bd4f50-2139-4e73-9fbc-0034074dba89-bundle\") pod \"94bd4f50-2139-4e73-9fbc-0034074dba89\" (UID: \"94bd4f50-2139-4e73-9fbc-0034074dba89\") " Aug 04 10:47:18.432011 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.431862 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94bd4f50-2139-4e73-9fbc-0034074dba89-util\") pod \"94bd4f50-2139-4e73-9fbc-0034074dba89\" (UID: \"94bd4f50-2139-4e73-9fbc-0034074dba89\") " Aug 04 10:47:18.432011 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.431894 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kks8\" (UniqueName: \"kubernetes.io/projected/94bd4f50-2139-4e73-9fbc-0034074dba89-kube-api-access-2kks8\") pod \"94bd4f50-2139-4e73-9fbc-0034074dba89\" (UID: \"94bd4f50-2139-4e73-9fbc-0034074dba89\") " Aug 04 10:47:18.432601 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.432559 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7d3373-7879-45bc-8e26-93ac88ec9671-bundle" (OuterVolumeSpecName: "bundle") pod "ad7d3373-7879-45bc-8e26-93ac88ec9671" (UID: "ad7d3373-7879-45bc-8e26-93ac88ec9671"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:47:18.432845 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.432651 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bd4f50-2139-4e73-9fbc-0034074dba89-bundle" (OuterVolumeSpecName: "bundle") pod "94bd4f50-2139-4e73-9fbc-0034074dba89" (UID: "94bd4f50-2139-4e73-9fbc-0034074dba89"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:47:18.434775 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.434748 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bd4f50-2139-4e73-9fbc-0034074dba89-kube-api-access-2kks8" (OuterVolumeSpecName: "kube-api-access-2kks8") pod "94bd4f50-2139-4e73-9fbc-0034074dba89" (UID: "94bd4f50-2139-4e73-9fbc-0034074dba89"). InnerVolumeSpecName "kube-api-access-2kks8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:47:18.434870 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.434843 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7d3373-7879-45bc-8e26-93ac88ec9671-kube-api-access-pfg9f" (OuterVolumeSpecName: "kube-api-access-pfg9f") pod "ad7d3373-7879-45bc-8e26-93ac88ec9671" (UID: "ad7d3373-7879-45bc-8e26-93ac88ec9671"). InnerVolumeSpecName "kube-api-access-pfg9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:47:18.438756 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.438732 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7d3373-7879-45bc-8e26-93ac88ec9671-util" (OuterVolumeSpecName: "util") pod "ad7d3373-7879-45bc-8e26-93ac88ec9671" (UID: "ad7d3373-7879-45bc-8e26-93ac88ec9671"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:47:18.439022 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.439002 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bd4f50-2139-4e73-9fbc-0034074dba89-util" (OuterVolumeSpecName: "util") pod "94bd4f50-2139-4e73-9fbc-0034074dba89" (UID: "94bd4f50-2139-4e73-9fbc-0034074dba89"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:47:18.533138 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.533045 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfg9f\" (UniqueName: \"kubernetes.io/projected/ad7d3373-7879-45bc-8e26-93ac88ec9671-kube-api-access-pfg9f\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:47:18.533138 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.533080 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad7d3373-7879-45bc-8e26-93ac88ec9671-util\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:47:18.533138 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.533090 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad7d3373-7879-45bc-8e26-93ac88ec9671-bundle\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:47:18.533138 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.533098 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94bd4f50-2139-4e73-9fbc-0034074dba89-bundle\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:47:18.533138 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.533106 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94bd4f50-2139-4e73-9fbc-0034074dba89-util\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:47:18.533138 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:18.533113 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kks8\" (UniqueName: \"kubernetes.io/projected/94bd4f50-2139-4e73-9fbc-0034074dba89-kube-api-access-2kks8\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:47:19.218702 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:19.218670 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" Aug 04 10:47:19.219149 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:19.218666 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a05djjr" event={"ID":"ad7d3373-7879-45bc-8e26-93ac88ec9671","Type":"ContainerDied","Data":"34227add06d804dba3aa000debecef8e9d8d0389b1779286707386cf43e48013"} Aug 04 10:47:19.219149 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:19.218814 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34227add06d804dba3aa000debecef8e9d8d0389b1779286707386cf43e48013" Aug 04 10:47:19.220489 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:19.220471 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" Aug 04 10:47:19.220604 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:19.220461 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbr8xh7" event={"ID":"94bd4f50-2139-4e73-9fbc-0034074dba89","Type":"ContainerDied","Data":"67e88776225f68b812ce1d5f378dc6a6eb87070324ac62b542d6c2bbe5c7c4fb"} Aug 04 10:47:19.220604 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:19.220547 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67e88776225f68b812ce1d5f378dc6a6eb87070324ac62b542d6c2bbe5c7c4fb" Aug 04 10:47:21.887053 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:21.887017 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:24.481215 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:24.481181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:24.866652 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:24.866609 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:25.295176 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:25.295099 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:27.742082 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:27.742052 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:28.494149 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:28.494111 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:28.884699 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:28.884667 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:28.998952 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:28.998921 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:30.815574 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:30.815536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:30.922266 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:30.922233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:31.169315 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:31.169286 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:31.499548 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:31.499450 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:31.774767 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:31.774688 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:33.171346 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171319 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-zl75f"] Aug 04 10:47:33.171718 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171687 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad7d3373-7879-45bc-8e26-93ac88ec9671" containerName="util" Aug 04 10:47:33.171718 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171699 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad7d3373-7879-45bc-8e26-93ac88ec9671" containerName="util" Aug 04 10:47:33.171718 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171708 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eeced55-bf8d-462a-bda6-3fda40dd5b0b" containerName="util" Aug 04 10:47:33.171718 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171714 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eeced55-bf8d-462a-bda6-3fda40dd5b0b" containerName="util" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171721 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94bd4f50-2139-4e73-9fbc-0034074dba89" containerName="extract" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171728 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="94bd4f50-2139-4e73-9fbc-0034074dba89" containerName="extract" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171738 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94bd4f50-2139-4e73-9fbc-0034074dba89" containerName="util" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171743 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="94bd4f50-2139-4e73-9fbc-0034074dba89" containerName="util" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171753 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad7d3373-7879-45bc-8e26-93ac88ec9671" containerName="pull" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171758 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad7d3373-7879-45bc-8e26-93ac88ec9671" containerName="pull" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171767 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad7d3373-7879-45bc-8e26-93ac88ec9671" containerName="extract" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171772 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad7d3373-7879-45bc-8e26-93ac88ec9671" containerName="extract" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171779 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eeced55-bf8d-462a-bda6-3fda40dd5b0b" containerName="extract" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171800 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eeced55-bf8d-462a-bda6-3fda40dd5b0b" containerName="extract" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171806 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94bd4f50-2139-4e73-9fbc-0034074dba89" containerName="pull" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171812 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="94bd4f50-2139-4e73-9fbc-0034074dba89" containerName="pull" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171821 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b3f072-2ac1-4a89-b7a9-d14a01f06f77" containerName="util" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171826 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b3f072-2ac1-4a89-b7a9-d14a01f06f77" containerName="util" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171833 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b3f072-2ac1-4a89-b7a9-d14a01f06f77" containerName="pull" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171838 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b3f072-2ac1-4a89-b7a9-d14a01f06f77" containerName="pull" Aug 04 10:47:33.171841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171845 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b3f072-2ac1-4a89-b7a9-d14a01f06f77" containerName="extract" Aug 04 10:47:33.172288 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171850 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b3f072-2ac1-4a89-b7a9-d14a01f06f77" containerName="extract" Aug 04 10:47:33.172288 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171857 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eeced55-bf8d-462a-bda6-3fda40dd5b0b" containerName="pull" Aug 04 10:47:33.172288 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171862 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eeced55-bf8d-462a-bda6-3fda40dd5b0b" containerName="pull" Aug 04 10:47:33.172288 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171916 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b3f072-2ac1-4a89-b7a9-d14a01f06f77" containerName="extract" Aug 04 10:47:33.172288 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171925 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eeced55-bf8d-462a-bda6-3fda40dd5b0b" containerName="extract" Aug 04 10:47:33.172288 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171932 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="94bd4f50-2139-4e73-9fbc-0034074dba89" containerName="extract" Aug 04 10:47:33.172288 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.171939 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad7d3373-7879-45bc-8e26-93ac88ec9671" containerName="extract" Aug 04 10:47:33.176168 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.176150 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-zl75f" Aug 04 10:47:33.178901 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.178877 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 10:47:33.179640 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.179623 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Aug 04 10:47:33.179986 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.179972 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-v5l7f\"" Aug 04 10:47:33.180588 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.180566 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 10:47:33.192599 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.192574 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-zl75f"] Aug 04 10:47:33.271599 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.271564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8gd2\" (UniqueName: \"kubernetes.io/projected/8c56e508-b1b2-4591-9442-cc5568f3fc0e-kube-api-access-z8gd2\") pod \"dns-operator-controller-manager-5985564776-zl75f\" (UID: \"8c56e508-b1b2-4591-9442-cc5568f3fc0e\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-zl75f" Aug 04 10:47:33.373066 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.373036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8gd2\" (UniqueName: \"kubernetes.io/projected/8c56e508-b1b2-4591-9442-cc5568f3fc0e-kube-api-access-z8gd2\") pod \"dns-operator-controller-manager-5985564776-zl75f\" (UID: \"8c56e508-b1b2-4591-9442-cc5568f3fc0e\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-zl75f" Aug 04 10:47:33.383418 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.383387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8gd2\" (UniqueName: \"kubernetes.io/projected/8c56e508-b1b2-4591-9442-cc5568f3fc0e-kube-api-access-z8gd2\") pod \"dns-operator-controller-manager-5985564776-zl75f\" (UID: \"8c56e508-b1b2-4591-9442-cc5568f3fc0e\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-zl75f" Aug 04 10:47:33.486777 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.486687 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-zl75f" Aug 04 10:47:33.647472 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:33.647445 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-zl75f"] Aug 04 10:47:33.650688 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:47:33.650659 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c56e508_b1b2_4591_9442_cc5568f3fc0e.slice/crio-3b42f7b0bc7077e1c22b21e4eec349180b76238913ffeaa9cbf479c97a6130ec WatchSource:0}: Error finding container 3b42f7b0bc7077e1c22b21e4eec349180b76238913ffeaa9cbf479c97a6130ec: Status 404 returned error can't find the container with id 3b42f7b0bc7077e1c22b21e4eec349180b76238913ffeaa9cbf479c97a6130ec Aug 04 10:47:34.298605 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:34.298571 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-zl75f" event={"ID":"8c56e508-b1b2-4591-9442-cc5568f3fc0e","Type":"ContainerStarted","Data":"3b42f7b0bc7077e1c22b21e4eec349180b76238913ffeaa9cbf479c97a6130ec"} Aug 04 10:47:34.492401 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:34.492367 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:34.590829 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:34.590791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:36.311692 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:36.311658 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-zl75f" event={"ID":"8c56e508-b1b2-4591-9442-cc5568f3fc0e","Type":"ContainerStarted","Data":"73ccfad99a9708e0ecef7d24141ebc89016960f3afa7ad1af6afe0329f8394b1"} Aug 04 10:47:36.312334 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:36.311798 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-zl75f" Aug 04 10:47:36.336297 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:36.336203 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-zl75f" podStartSLOduration=0.906011495 podStartE2EDuration="3.336186549s" totalImagesPullingTime="2.430175054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:47:36.33614704 +0000 UTC m=+645.433940438" watchObservedRunningTime="2026-08-04 10:47:36.336186549 +0000 UTC m=+645.433979939" Aug 04 10:47:36.606917 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:36.606829 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:36.821870 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:36.821797 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:38.490335 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:38.490304 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:38.595603 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:38.595572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:38.691576 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:38.691543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:38.792228 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:38.792140 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:38.897242 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:38.897205 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:39.233793 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:39.233760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:47.013571 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:47.013539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:47.123591 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:47.123559 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:47.231940 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:47.231909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:47.320657 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:47.320568 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-zl75f" Aug 04 10:47:49.447711 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:49.447670 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:49.582990 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:49.582946 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:53.177542 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:53.177490 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:53.844308 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:53.844267 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:53.947786 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:53.947753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:57.647843 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:57.647806 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:57.750258 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:57.750227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:57.848591 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:57.848557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:57.979523 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:57.979425 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:47:58.077738 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:47:58.077707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:08.055421 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:08.055386 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:08.157085 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:08.157052 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:10.457653 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:10.457616 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:10.558098 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:10.558060 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:11.841553 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:11.841481 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:15.162818 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:15.162774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:15.254228 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:15.254197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:20.919211 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:20.919175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:21.203131 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:21.203051 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:23.267636 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.267604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:23.366219 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.366182 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:23.481617 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.481581 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:23.641295 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.641259 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:48:23.647348 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.647321 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr" Aug 04 10:48:23.651363 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.651336 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 04 10:48:23.651363 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.651357 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-56r9c\"" Aug 04 10:48:23.655829 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.655801 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:48:23.687026 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.686989 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:48:23.829417 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.829382 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ed8ffdb-754b-4bed-9124-cc309f57cf04-config-file\") pod \"limitador-limitador-85d9db4d5d-x2gkr\" (UID: \"8ed8ffdb-754b-4bed-9124-cc309f57cf04\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr" Aug 04 10:48:23.829645 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.829431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4x7x\" (UniqueName: \"kubernetes.io/projected/8ed8ffdb-754b-4bed-9124-cc309f57cf04-kube-api-access-b4x7x\") pod \"limitador-limitador-85d9db4d5d-x2gkr\" (UID: \"8ed8ffdb-754b-4bed-9124-cc309f57cf04\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr" Aug 04 10:48:23.930445 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.930343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ed8ffdb-754b-4bed-9124-cc309f57cf04-config-file\") pod \"limitador-limitador-85d9db4d5d-x2gkr\" (UID: \"8ed8ffdb-754b-4bed-9124-cc309f57cf04\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr" Aug 04 10:48:23.930445 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.930402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4x7x\" (UniqueName: \"kubernetes.io/projected/8ed8ffdb-754b-4bed-9124-cc309f57cf04-kube-api-access-b4x7x\") pod \"limitador-limitador-85d9db4d5d-x2gkr\" (UID: \"8ed8ffdb-754b-4bed-9124-cc309f57cf04\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr" Aug 04 10:48:23.931008 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.930987 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ed8ffdb-754b-4bed-9124-cc309f57cf04-config-file\") pod \"limitador-limitador-85d9db4d5d-x2gkr\" (UID: \"8ed8ffdb-754b-4bed-9124-cc309f57cf04\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr" Aug 04 10:48:23.939196 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.939166 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4x7x\" (UniqueName: \"kubernetes.io/projected/8ed8ffdb-754b-4bed-9124-cc309f57cf04-kube-api-access-b4x7x\") pod \"limitador-limitador-85d9db4d5d-x2gkr\" (UID: \"8ed8ffdb-754b-4bed-9124-cc309f57cf04\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr" Aug 04 10:48:23.960315 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:23.960272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr" Aug 04 10:48:24.107129 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:24.107101 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:48:24.109336 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:48:24.109307 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed8ffdb_754b_4bed_9124_cc309f57cf04.slice/crio-ac6a9649b10535df5d2594c5063ea26504965b7b8b7f26126193dfe87ca47b2e WatchSource:0}: Error finding container ac6a9649b10535df5d2594c5063ea26504965b7b8b7f26126193dfe87ca47b2e: Status 404 returned error can't find the container with id ac6a9649b10535df5d2594c5063ea26504965b7b8b7f26126193dfe87ca47b2e Aug 04 10:48:24.145099 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:24.145070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:24.225043 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:24.224966 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-cs8vf"] Aug 04 10:48:24.229583 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:24.229566 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-cs8vf" Aug 04 10:48:24.232020 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:24.231985 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ddqb6\"" Aug 04 10:48:24.234397 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:24.234376 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-cs8vf"] Aug 04 10:48:24.237373 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:24.237346 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:24.333895 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:24.333864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tqws\" (UniqueName: \"kubernetes.io/projected/55652169-a51c-4931-be19-9db76831247d-kube-api-access-8tqws\") pod \"authorino-c548b5d7d-cs8vf\" (UID: \"55652169-a51c-4931-be19-9db76831247d\") " pod="kuadrant-system/authorino-c548b5d7d-cs8vf" Aug 04 10:48:24.435143 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:24.435104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tqws\" (UniqueName: \"kubernetes.io/projected/55652169-a51c-4931-be19-9db76831247d-kube-api-access-8tqws\") pod \"authorino-c548b5d7d-cs8vf\" (UID: \"55652169-a51c-4931-be19-9db76831247d\") " pod="kuadrant-system/authorino-c548b5d7d-cs8vf" Aug 04 10:48:24.443530 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:24.443472 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tqws\" (UniqueName: \"kubernetes.io/projected/55652169-a51c-4931-be19-9db76831247d-kube-api-access-8tqws\") pod \"authorino-c548b5d7d-cs8vf\" (UID: \"55652169-a51c-4931-be19-9db76831247d\") " pod="kuadrant-system/authorino-c548b5d7d-cs8vf" Aug 04 10:48:24.540329 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:24.540251 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-cs8vf" Aug 04 10:48:24.580189 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:24.580156 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr" event={"ID":"8ed8ffdb-754b-4bed-9124-cc309f57cf04","Type":"ContainerStarted","Data":"ac6a9649b10535df5d2594c5063ea26504965b7b8b7f26126193dfe87ca47b2e"} Aug 04 10:48:24.669651 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:24.669627 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-cs8vf"] Aug 04 10:48:24.671595 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:48:24.671566 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55652169_a51c_4931_be19_9db76831247d.slice/crio-bd1b0ab49c89d96823f0280c4886fce30e5b75b8c98041762df3b89375e60aae WatchSource:0}: Error finding container bd1b0ab49c89d96823f0280c4886fce30e5b75b8c98041762df3b89375e60aae: Status 404 returned error can't find the container with id bd1b0ab49c89d96823f0280c4886fce30e5b75b8c98041762df3b89375e60aae Aug 04 10:48:25.587350 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:25.587318 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr" event={"ID":"8ed8ffdb-754b-4bed-9124-cc309f57cf04","Type":"ContainerStarted","Data":"7963b016eaac53286f28888c0b788720ec0f9363e51ec9ab61f4c1599b5a7642"} Aug 04 10:48:25.587795 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:25.587491 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr" Aug 04 10:48:25.588537 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:25.588496 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-cs8vf" event={"ID":"55652169-a51c-4931-be19-9db76831247d","Type":"ContainerStarted","Data":"bd1b0ab49c89d96823f0280c4886fce30e5b75b8c98041762df3b89375e60aae"} Aug 04 10:48:25.606578 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:25.606527 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr" podStartSLOduration=1.4908813730000001 podStartE2EDuration="2.606513846s" totalImagesPullingTime="1.115632473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:48:25.604793374 +0000 UTC m=+694.702586769" watchObservedRunningTime="2026-08-04 10:48:25.606513846 +0000 UTC m=+694.704307231" Aug 04 10:48:27.602416 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:27.602375 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-cs8vf" event={"ID":"55652169-a51c-4931-be19-9db76831247d","Type":"ContainerStarted","Data":"0e2eb0097a6b57541ff1d3bfcc60c54848ace5ed4af0b22e4abef4e019836efa"} Aug 04 10:48:27.618221 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:27.618171 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-cs8vf" podStartSLOduration=1.546009868 podStartE2EDuration="3.618156378s" totalImagesPullingTime="2.07214651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:48:27.616729316 +0000 UTC m=+696.714522712" watchObservedRunningTime="2026-08-04 10:48:27.618156378 +0000 UTC m=+696.715949773" Aug 04 10:48:36.596162 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:36.596127 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr" Aug 04 10:48:39.334106 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:39.334077 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:42.404068 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:42.404031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:51.691306 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:51.691228 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:51.919012 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:51.918982 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:52.169808 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:52.169775 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7555667bcd-hvllz"] Aug 04 10:48:52.173156 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:52.173140 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7555667bcd-hvllz" Aug 04 10:48:52.176467 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:52.176444 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 04 10:48:52.181797 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:52.181774 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7555667bcd-hvllz"] Aug 04 10:48:52.286687 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:52.286589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhxt7\" (UniqueName: \"kubernetes.io/projected/929a6811-50a4-4e92-8d81-9ec6b539a519-kube-api-access-rhxt7\") pod \"authorino-7555667bcd-hvllz\" (UID: \"929a6811-50a4-4e92-8d81-9ec6b539a519\") " pod="kuadrant-system/authorino-7555667bcd-hvllz" Aug 04 10:48:52.286841 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:52.286746 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/929a6811-50a4-4e92-8d81-9ec6b539a519-tls-cert\") pod \"authorino-7555667bcd-hvllz\" (UID: \"929a6811-50a4-4e92-8d81-9ec6b539a519\") " pod="kuadrant-system/authorino-7555667bcd-hvllz" Aug 04 10:48:52.388199 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:52.388165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/929a6811-50a4-4e92-8d81-9ec6b539a519-tls-cert\") pod \"authorino-7555667bcd-hvllz\" (UID: \"929a6811-50a4-4e92-8d81-9ec6b539a519\") " pod="kuadrant-system/authorino-7555667bcd-hvllz" Aug 04 10:48:52.388371 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:52.388215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhxt7\" (UniqueName: \"kubernetes.io/projected/929a6811-50a4-4e92-8d81-9ec6b539a519-kube-api-access-rhxt7\") pod \"authorino-7555667bcd-hvllz\" (UID: \"929a6811-50a4-4e92-8d81-9ec6b539a519\") " pod="kuadrant-system/authorino-7555667bcd-hvllz" Aug 04 10:48:52.390689 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:52.390668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/929a6811-50a4-4e92-8d81-9ec6b539a519-tls-cert\") pod \"authorino-7555667bcd-hvllz\" (UID: \"929a6811-50a4-4e92-8d81-9ec6b539a519\") " pod="kuadrant-system/authorino-7555667bcd-hvllz" Aug 04 10:48:52.396272 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:52.396253 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhxt7\" (UniqueName: \"kubernetes.io/projected/929a6811-50a4-4e92-8d81-9ec6b539a519-kube-api-access-rhxt7\") pod \"authorino-7555667bcd-hvllz\" (UID: \"929a6811-50a4-4e92-8d81-9ec6b539a519\") " pod="kuadrant-system/authorino-7555667bcd-hvllz" Aug 04 10:48:52.482653 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:52.482618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7555667bcd-hvllz" Aug 04 10:48:52.617101 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:52.617072 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7555667bcd-hvllz"] Aug 04 10:48:52.621899 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:48:52.621871 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod929a6811_50a4_4e92_8d81_9ec6b539a519.slice/crio-c70fe3f9b005e3d3ac24210b963ef772a7d6538bfd46107048a1af76fccf63ad WatchSource:0}: Error finding container c70fe3f9b005e3d3ac24210b963ef772a7d6538bfd46107048a1af76fccf63ad: Status 404 returned error can't find the container with id c70fe3f9b005e3d3ac24210b963ef772a7d6538bfd46107048a1af76fccf63ad Aug 04 10:48:52.751284 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:52.751246 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7555667bcd-hvllz" event={"ID":"929a6811-50a4-4e92-8d81-9ec6b539a519","Type":"ContainerStarted","Data":"c70fe3f9b005e3d3ac24210b963ef772a7d6538bfd46107048a1af76fccf63ad"} Aug 04 10:48:53.761935 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:53.761838 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7555667bcd-hvllz" event={"ID":"929a6811-50a4-4e92-8d81-9ec6b539a519","Type":"ContainerStarted","Data":"01a0b1e377485f45856769618e7538e350c349d0a89f599384b061c8852bd4d3"} Aug 04 10:48:53.785201 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:53.785149 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7555667bcd-hvllz" podStartSLOduration=0.989899337 podStartE2EDuration="1.785132905s" totalImagesPullingTime="795.233568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:48:53.782985473 +0000 UTC m=+722.880778865" watchObservedRunningTime="2026-08-04 10:48:53.785132905 +0000 UTC m=+722.882926313" Aug 04 10:48:53.811765 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:53.811727 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-cs8vf"] Aug 04 10:48:53.811947 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:53.811911 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-cs8vf" podUID="55652169-a51c-4931-be19-9db76831247d" containerName="authorino" containerID="cri-o://0e2eb0097a6b57541ff1d3bfcc60c54848ace5ed4af0b22e4abef4e019836efa" gracePeriod=30 Aug 04 10:48:54.049656 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:54.049632 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-cs8vf" Aug 04 10:48:54.104121 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:54.104092 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tqws\" (UniqueName: \"kubernetes.io/projected/55652169-a51c-4931-be19-9db76831247d-kube-api-access-8tqws\") pod \"55652169-a51c-4931-be19-9db76831247d\" (UID: \"55652169-a51c-4931-be19-9db76831247d\") " Aug 04 10:48:54.106244 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:54.106213 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55652169-a51c-4931-be19-9db76831247d-kube-api-access-8tqws" (OuterVolumeSpecName: "kube-api-access-8tqws") pod "55652169-a51c-4931-be19-9db76831247d" (UID: "55652169-a51c-4931-be19-9db76831247d"). InnerVolumeSpecName "kube-api-access-8tqws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:48:54.205554 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:54.205520 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tqws\" (UniqueName: \"kubernetes.io/projected/55652169-a51c-4931-be19-9db76831247d-kube-api-access-8tqws\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:48:54.768511 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:54.768416 2576 generic.go:356] "Generic (PLEG): container finished" podID="55652169-a51c-4931-be19-9db76831247d" containerID="0e2eb0097a6b57541ff1d3bfcc60c54848ace5ed4af0b22e4abef4e019836efa" exitCode=0 Aug 04 10:48:54.768511 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:54.768456 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-cs8vf" Aug 04 10:48:54.768511 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:54.768486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-cs8vf" event={"ID":"55652169-a51c-4931-be19-9db76831247d","Type":"ContainerDied","Data":"0e2eb0097a6b57541ff1d3bfcc60c54848ace5ed4af0b22e4abef4e019836efa"} Aug 04 10:48:54.769035 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:54.768542 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-cs8vf" event={"ID":"55652169-a51c-4931-be19-9db76831247d","Type":"ContainerDied","Data":"bd1b0ab49c89d96823f0280c4886fce30e5b75b8c98041762df3b89375e60aae"} Aug 04 10:48:54.769035 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:54.768559 2576 scope.go:117] "RemoveContainer" containerID="0e2eb0097a6b57541ff1d3bfcc60c54848ace5ed4af0b22e4abef4e019836efa" Aug 04 10:48:54.777702 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:54.777684 2576 scope.go:117] "RemoveContainer" containerID="0e2eb0097a6b57541ff1d3bfcc60c54848ace5ed4af0b22e4abef4e019836efa" Aug 04 10:48:54.777958 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:48:54.777939 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2eb0097a6b57541ff1d3bfcc60c54848ace5ed4af0b22e4abef4e019836efa\": container with ID starting with 0e2eb0097a6b57541ff1d3bfcc60c54848ace5ed4af0b22e4abef4e019836efa not found: ID does not exist" containerID="0e2eb0097a6b57541ff1d3bfcc60c54848ace5ed4af0b22e4abef4e019836efa" Aug 04 10:48:54.778026 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:54.777966 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2eb0097a6b57541ff1d3bfcc60c54848ace5ed4af0b22e4abef4e019836efa"} err="failed to get container status \"0e2eb0097a6b57541ff1d3bfcc60c54848ace5ed4af0b22e4abef4e019836efa\": rpc error: code = NotFound desc = could not find container \"0e2eb0097a6b57541ff1d3bfcc60c54848ace5ed4af0b22e4abef4e019836efa\": container with ID starting with 0e2eb0097a6b57541ff1d3bfcc60c54848ace5ed4af0b22e4abef4e019836efa not found: ID does not exist" Aug 04 10:48:54.790179 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:54.790151 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-cs8vf"] Aug 04 10:48:54.794421 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:54.794398 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-cs8vf"] Aug 04 10:48:55.437799 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:55.437765 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55652169-a51c-4931-be19-9db76831247d" path="/var/lib/kubelet/pods/55652169-a51c-4931-be19-9db76831247d/volumes" Aug 04 10:48:58.749002 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:58.748969 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:58.846978 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:58.846949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:48:58.938734 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:48:58.938697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:02.163858 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:02.163812 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:02.322296 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:02.322245 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:09.820910 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:09.820853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:13.624800 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:13.624741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:16.639798 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:16.639765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:16.719317 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:16.719283 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:17.824548 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:17.824494 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:33.790576 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:33.790544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:33.887671 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:33.887640 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:35.310293 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:35.310262 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:35.516193 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:35.516157 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:35.971653 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:35.971622 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:36.107626 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:36.107597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:36.324835 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:36.324758 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:49:45.539619 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:45.539584 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:45.656808 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:45.656772 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:50.134221 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:50.134189 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:50.361899 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:50.361857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:52.615806 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:52.615770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:49:52.992267 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:49:52.992172 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:50:07.519105 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:50:07.519073 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:50:08.188052 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:50:08.188019 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:50:17.985051 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:50:17.984955 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:50:18.877336 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:50:18.877294 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:50:41.544172 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:50:41.544139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:50:42.811645 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:50:42.811604 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:50:46.992367 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:50:46.992334 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:50:49.188056 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:50:49.188023 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:50:52.362838 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:50:52.362800 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:50:59.579760 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:50:59.579716 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:52:15.262379 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.262344 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-854cb55995-l5hbl"] Aug 04 10:52:15.264784 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.262748 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55652169-a51c-4931-be19-9db76831247d" containerName="authorino" Aug 04 10:52:15.264784 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.262760 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="55652169-a51c-4931-be19-9db76831247d" containerName="authorino" Aug 04 10:52:15.264784 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.262821 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="55652169-a51c-4931-be19-9db76831247d" containerName="authorino" Aug 04 10:52:15.265648 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.265633 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-854cb55995-l5hbl" Aug 04 10:52:15.274360 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.274338 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-854cb55995-l5hbl"] Aug 04 10:52:15.350580 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.350540 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/43b4b83d-a497-4006-a752-6d5fa2de3f3a-tls-cert\") pod \"authorino-854cb55995-l5hbl\" (UID: \"43b4b83d-a497-4006-a752-6d5fa2de3f3a\") " pod="kuadrant-system/authorino-854cb55995-l5hbl" Aug 04 10:52:15.350761 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.350652 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf74q\" (UniqueName: \"kubernetes.io/projected/43b4b83d-a497-4006-a752-6d5fa2de3f3a-kube-api-access-mf74q\") pod \"authorino-854cb55995-l5hbl\" (UID: \"43b4b83d-a497-4006-a752-6d5fa2de3f3a\") " pod="kuadrant-system/authorino-854cb55995-l5hbl" Aug 04 10:52:15.451484 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.451443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf74q\" (UniqueName: \"kubernetes.io/projected/43b4b83d-a497-4006-a752-6d5fa2de3f3a-kube-api-access-mf74q\") pod \"authorino-854cb55995-l5hbl\" (UID: \"43b4b83d-a497-4006-a752-6d5fa2de3f3a\") " pod="kuadrant-system/authorino-854cb55995-l5hbl" Aug 04 10:52:15.451681 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.451554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/43b4b83d-a497-4006-a752-6d5fa2de3f3a-tls-cert\") pod \"authorino-854cb55995-l5hbl\" (UID: \"43b4b83d-a497-4006-a752-6d5fa2de3f3a\") " pod="kuadrant-system/authorino-854cb55995-l5hbl" Aug 04 10:52:15.454002 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.453981 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/43b4b83d-a497-4006-a752-6d5fa2de3f3a-tls-cert\") pod \"authorino-854cb55995-l5hbl\" (UID: \"43b4b83d-a497-4006-a752-6d5fa2de3f3a\") " pod="kuadrant-system/authorino-854cb55995-l5hbl" Aug 04 10:52:15.459737 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.459716 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf74q\" (UniqueName: \"kubernetes.io/projected/43b4b83d-a497-4006-a752-6d5fa2de3f3a-kube-api-access-mf74q\") pod \"authorino-854cb55995-l5hbl\" (UID: \"43b4b83d-a497-4006-a752-6d5fa2de3f3a\") " pod="kuadrant-system/authorino-854cb55995-l5hbl" Aug 04 10:52:15.575844 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.575755 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-854cb55995-l5hbl" Aug 04 10:52:15.708081 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.708054 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-854cb55995-l5hbl"] Aug 04 10:52:15.710161 ip-10-0-131-159 kubenswrapper[2576]: W0804 10:52:15.710135 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43b4b83d_a497_4006_a752_6d5fa2de3f3a.slice/crio-c7983c69e7d489a6990792f4928f5241fb91d7de9900180ad1ae31122f793b55 WatchSource:0}: Error finding container c7983c69e7d489a6990792f4928f5241fb91d7de9900180ad1ae31122f793b55: Status 404 returned error can't find the container with id c7983c69e7d489a6990792f4928f5241fb91d7de9900180ad1ae31122f793b55 Aug 04 10:52:15.711537 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.711519 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 10:52:15.885345 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:15.885311 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-854cb55995-l5hbl" event={"ID":"43b4b83d-a497-4006-a752-6d5fa2de3f3a","Type":"ContainerStarted","Data":"c7983c69e7d489a6990792f4928f5241fb91d7de9900180ad1ae31122f793b55"} Aug 04 10:52:16.894987 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:16.894939 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-854cb55995-l5hbl" event={"ID":"43b4b83d-a497-4006-a752-6d5fa2de3f3a","Type":"ContainerStarted","Data":"1ce3276fcf68f3023395d9e90c02035cc050712feece053b4d126033bb0049f3"} Aug 04 10:52:16.913474 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:16.913408 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-854cb55995-l5hbl" podStartSLOduration=1.226238878 podStartE2EDuration="1.913387454s" totalImagesPullingTime="687.148576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:52:16.912358618 +0000 UTC m=+926.010152008" watchObservedRunningTime="2026-08-04 10:52:16.913387454 +0000 UTC m=+926.011180850" Aug 04 10:52:16.944053 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:16.944017 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7555667bcd-hvllz"] Aug 04 10:52:16.944261 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:16.944235 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7555667bcd-hvllz" podUID="929a6811-50a4-4e92-8d81-9ec6b539a519" containerName="authorino" containerID="cri-o://01a0b1e377485f45856769618e7538e350c349d0a89f599384b061c8852bd4d3" gracePeriod=30 Aug 04 10:52:17.210128 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.210096 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7555667bcd-hvllz" Aug 04 10:52:17.269190 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.269159 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhxt7\" (UniqueName: \"kubernetes.io/projected/929a6811-50a4-4e92-8d81-9ec6b539a519-kube-api-access-rhxt7\") pod \"929a6811-50a4-4e92-8d81-9ec6b539a519\" (UID: \"929a6811-50a4-4e92-8d81-9ec6b539a519\") " Aug 04 10:52:17.269361 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.269212 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/929a6811-50a4-4e92-8d81-9ec6b539a519-tls-cert\") pod \"929a6811-50a4-4e92-8d81-9ec6b539a519\" (UID: \"929a6811-50a4-4e92-8d81-9ec6b539a519\") " Aug 04 10:52:17.271345 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.271308 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929a6811-50a4-4e92-8d81-9ec6b539a519-kube-api-access-rhxt7" (OuterVolumeSpecName: "kube-api-access-rhxt7") pod "929a6811-50a4-4e92-8d81-9ec6b539a519" (UID: "929a6811-50a4-4e92-8d81-9ec6b539a519"). InnerVolumeSpecName "kube-api-access-rhxt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:52:17.280240 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.280213 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929a6811-50a4-4e92-8d81-9ec6b539a519-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "929a6811-50a4-4e92-8d81-9ec6b539a519" (UID: "929a6811-50a4-4e92-8d81-9ec6b539a519"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:52:17.370929 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.370895 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhxt7\" (UniqueName: \"kubernetes.io/projected/929a6811-50a4-4e92-8d81-9ec6b539a519-kube-api-access-rhxt7\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:52:17.370929 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.370924 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/929a6811-50a4-4e92-8d81-9ec6b539a519-tls-cert\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 10:52:17.474980 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.474901 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:52:17.573338 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.573299 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:52:17.902908 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.902866 2576 generic.go:356] "Generic (PLEG): container finished" podID="929a6811-50a4-4e92-8d81-9ec6b539a519" containerID="01a0b1e377485f45856769618e7538e350c349d0a89f599384b061c8852bd4d3" exitCode=0 Aug 04 10:52:17.903394 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.902932 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7555667bcd-hvllz" Aug 04 10:52:17.903394 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.902996 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7555667bcd-hvllz" event={"ID":"929a6811-50a4-4e92-8d81-9ec6b539a519","Type":"ContainerDied","Data":"01a0b1e377485f45856769618e7538e350c349d0a89f599384b061c8852bd4d3"} Aug 04 10:52:17.903394 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.903030 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7555667bcd-hvllz" event={"ID":"929a6811-50a4-4e92-8d81-9ec6b539a519","Type":"ContainerDied","Data":"c70fe3f9b005e3d3ac24210b963ef772a7d6538bfd46107048a1af76fccf63ad"} Aug 04 10:52:17.903394 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.903054 2576 scope.go:117] "RemoveContainer" containerID="01a0b1e377485f45856769618e7538e350c349d0a89f599384b061c8852bd4d3" Aug 04 10:52:17.918289 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.918266 2576 scope.go:117] "RemoveContainer" containerID="01a0b1e377485f45856769618e7538e350c349d0a89f599384b061c8852bd4d3" Aug 04 10:52:17.918736 ip-10-0-131-159 kubenswrapper[2576]: E0804 10:52:17.918704 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a0b1e377485f45856769618e7538e350c349d0a89f599384b061c8852bd4d3\": container with ID starting with 01a0b1e377485f45856769618e7538e350c349d0a89f599384b061c8852bd4d3 not found: ID does not exist" containerID="01a0b1e377485f45856769618e7538e350c349d0a89f599384b061c8852bd4d3" Aug 04 10:52:17.918912 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.918876 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a0b1e377485f45856769618e7538e350c349d0a89f599384b061c8852bd4d3"} err="failed to get container status \"01a0b1e377485f45856769618e7538e350c349d0a89f599384b061c8852bd4d3\": rpc error: code = NotFound desc = could not find container \"01a0b1e377485f45856769618e7538e350c349d0a89f599384b061c8852bd4d3\": container with ID starting with 01a0b1e377485f45856769618e7538e350c349d0a89f599384b061c8852bd4d3 not found: ID does not exist" Aug 04 10:52:17.920457 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.920435 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7555667bcd-hvllz"] Aug 04 10:52:17.925465 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:17.925441 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7555667bcd-hvllz"] Aug 04 10:52:18.517821 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:18.517791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:52:18.615568 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:18.615489 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:52:19.437970 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:19.437936 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="929a6811-50a4-4e92-8d81-9ec6b539a519" path="/var/lib/kubelet/pods/929a6811-50a4-4e92-8d81-9ec6b539a519/volumes" Aug 04 10:52:46.786596 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:46.786555 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:52:55.779317 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:52:55.779279 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:53:07.389639 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:53:07.389605 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:53:11.010184 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:53:11.010151 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:54:23.317938 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:54:23.317906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:54:24.776121 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:54:24.776087 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:54:46.282564 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:54:46.282469 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:54:54.382778 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:54:54.382742 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:55:25.220632 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:55:25.220596 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:55:25.307147 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:55:25.307112 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:55:25.439288 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:55:25.439256 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:55:27.453827 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:55:27.453796 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:55:29.109089 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:55:29.109052 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:55:31.222480 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:55:31.222448 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:55:31.332602 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:55:31.332570 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:55:31.445102 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:55:31.445071 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:55:31.614387 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:55:31.614350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:55:37.400451 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:55:37.400417 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:55:37.500327 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:55:37.500296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:55:53.181343 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:55:53.181303 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:56:00.375596 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:56:00.375556 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:56:15.789649 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:56:15.789561 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:56:21.688239 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:56:21.682612 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:56:31.282808 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:56:31.282770 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:56:37.583152 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:56:37.583113 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:56:51.784630 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:56:51.784594 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:56:57.093443 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:56:57.093398 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:57:02.194094 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:57:02.194054 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:57:17.989345 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:57:17.989308 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:57:23.882260 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:57:23.882225 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:57:34.586297 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:57:34.586260 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:57:45.996215 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:57:45.996138 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:57:46.068666 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:57:46.068621 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:57:57.596578 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:57:57.596541 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:58:08.088369 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:58:08.088333 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:58:13.298186 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:58:13.298133 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:58:20.491254 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:58:20.491213 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:58:28.393671 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:58:28.393629 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:58:33.488453 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:58:33.488413 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:58:39.389187 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:58:39.389155 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:58:46.884887 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:58:46.884838 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:58:59.085243 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:58:59.085206 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:59:02.405892 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:59:02.405857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 10:59:05.890442 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:59:05.890401 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:59:16.492925 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:59:16.492826 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:59:21.587919 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:59:21.587889 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:59:27.688319 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:59:27.688280 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:59:37.102536 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:59:37.102484 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:59:42.894118 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:59:42.894078 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 10:59:53.297232 ip-10-0-131-159 kubenswrapper[2576]: I0804 10:59:53.297196 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:00:00.143795 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:00.143750 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764020-m8p6r"] Aug 04 11:00:00.144199 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:00.144141 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="929a6811-50a4-4e92-8d81-9ec6b539a519" containerName="authorino" Aug 04 11:00:00.144199 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:00.144153 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="929a6811-50a4-4e92-8d81-9ec6b539a519" containerName="authorino" Aug 04 11:00:00.144276 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:00.144246 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="929a6811-50a4-4e92-8d81-9ec6b539a519" containerName="authorino" Aug 04 11:00:00.147332 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:00.147314 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764020-m8p6r" Aug 04 11:00:00.150272 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:00.150253 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tgsgx\"" Aug 04 11:00:00.156988 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:00.156968 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764020-m8p6r" Aug 04 11:00:00.163303 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:00.163278 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764020-m8p6r"] Aug 04 11:00:00.297407 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:00.297382 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764020-m8p6r"] Aug 04 11:00:00.299051 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:00:00.299026 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3116da_8cc0_40ad_8795_40cdb70741a7.slice/crio-e0961de564d9556ccba0dfc5b3bdc297780b4128ee53f4763d2cccb6ea3d59f3 WatchSource:0}: Error finding container e0961de564d9556ccba0dfc5b3bdc297780b4128ee53f4763d2cccb6ea3d59f3: Status 404 returned error can't find the container with id e0961de564d9556ccba0dfc5b3bdc297780b4128ee53f4763d2cccb6ea3d59f3 Aug 04 11:00:00.300997 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:00.300976 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 11:00:00.494174 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:00.494079 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764020-m8p6r" event={"ID":"ef3116da-8cc0-40ad-8795-40cdb70741a7","Type":"ContainerStarted","Data":"e0961de564d9556ccba0dfc5b3bdc297780b4128ee53f4763d2cccb6ea3d59f3"} Aug 04 11:00:03.514947 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:03.514908 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764020-m8p6r" event={"ID":"ef3116da-8cc0-40ad-8795-40cdb70741a7","Type":"ContainerStarted","Data":"391f47dd0452f2e88807718c03866a31ecd8a872f717d2049fea42fe8480e980"} Aug 04 11:00:03.530647 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:03.530593 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764020-m8p6r" podStartSLOduration=1.141935684 podStartE2EDuration="3.530575018s" totalImagesPullingTime="2.388639334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:00:03.529428906 +0000 UTC m=+1392.627222301" watchObservedRunningTime="2026-08-04 11:00:03.530575018 +0000 UTC m=+1392.628368412" Aug 04 11:00:04.522204 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:04.522167 2576 generic.go:356] "Generic (PLEG): container finished" podID="ef3116da-8cc0-40ad-8795-40cdb70741a7" containerID="391f47dd0452f2e88807718c03866a31ecd8a872f717d2049fea42fe8480e980" exitCode=0 Aug 04 11:00:04.522657 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:04.522264 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764020-m8p6r" event={"ID":"ef3116da-8cc0-40ad-8795-40cdb70741a7","Type":"ContainerDied","Data":"391f47dd0452f2e88807718c03866a31ecd8a872f717d2049fea42fe8480e980"} Aug 04 11:00:05.660196 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:05.660172 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764020-m8p6r" Aug 04 11:00:06.534349 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:06.534317 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764020-m8p6r" Aug 04 11:00:06.534550 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:06.534355 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764020-m8p6r" event={"ID":"ef3116da-8cc0-40ad-8795-40cdb70741a7","Type":"ContainerDied","Data":"e0961de564d9556ccba0dfc5b3bdc297780b4128ee53f4763d2cccb6ea3d59f3"} Aug 04 11:00:06.534550 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:06.534404 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0961de564d9556ccba0dfc5b3bdc297780b4128ee53f4763d2cccb6ea3d59f3" Aug 04 11:00:20.596175 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:20.596141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:00:24.630088 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:00:24.630034 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:02:10.992593 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:02:10.992555 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:02:15.976333 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:02:15.976295 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:02:18.364685 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:02:18.364638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:02:41.282181 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:02:41.282141 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:03:02.585924 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:03:02.585885 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:03:12.477339 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:03:12.477305 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:03:18.284417 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:03:18.284376 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:03:23.482365 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:03:23.482326 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:03:33.276853 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:03:33.276810 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:03:38.185838 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:03:38.185802 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:03:48.484620 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:03:48.484532 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:03:53.780558 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:03:53.780522 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:04:03.678111 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:04:03.678071 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:04:10.476610 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:04:10.476568 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:04:12.230952 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:04:12.230910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:04:19.483165 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:04:19.483127 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:04:41.376959 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:04:41.376925 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:04:48.582316 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:04:48.582279 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:04:59.303785 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:04:59.303748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:05:02.292098 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:05:02.292063 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:05:07.005684 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:05:07.005654 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:05:14.278998 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:05:14.278906 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:05:32.793953 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:05:32.793918 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:05:34.584801 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:05:34.584762 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:05:38.863532 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:05:38.863481 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:05:44.579834 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:05:44.579797 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:05:50.289179 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:05:50.289137 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:05:59.181728 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:05:59.181690 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:06:06.379102 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:06:06.379063 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:06:12.395856 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:06:12.395818 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:06:18.785035 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:06:18.784995 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:06:26.488852 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:06:26.488811 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:06:30.783480 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:06:30.783439 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:06:40.685693 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:06:40.685654 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:06:47.881824 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:06:47.881727 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:07:03.478692 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:07:03.478647 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:07:09.982527 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:07:09.982454 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:07:16.476345 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:07:16.476311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:07:18.292045 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:07:18.292008 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:07:21.878927 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:07:21.878884 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:07:29.374393 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:07:29.374351 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:07:35.786638 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:07:35.786604 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:07:40.687898 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:07:40.687857 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:07:46.788261 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:07:46.788213 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:07:56.580689 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:07:56.580644 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:08:03.781376 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:08:03.781337 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:08:13.584546 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:08:13.584432 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:08:19.879839 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:08:19.879804 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:08:30.485333 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:08:30.485299 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:08:45.773667 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:08:45.773633 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:08:51.972236 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:08:51.972198 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:08:55.190614 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:08:55.190573 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:08:58.079781 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:08:58.079742 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:09:09.482626 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:09.482584 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:09:17.781105 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:17.781072 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:09:21.062882 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:21.062816 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:09:23.283374 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:23.283334 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:09:24.147791 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:24.147754 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:09:24.240682 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:24.240642 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:09:24.644745 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:24.644707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:09:30.554960 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:30.554929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:09:30.688327 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:30.688294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:09:30.831771 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:30.831736 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:09:31.130060 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:31.129967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:09:31.299667 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:31.299628 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:09:33.317860 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:33.317817 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:09:33.419681 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:33.419641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:09:34.422915 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:34.422876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:09:35.108047 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:35.107997 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:09:35.234440 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:35.234394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:09:37.488071 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:37.488030 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:09:58.863879 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:58.863844 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq"] Aug 04 11:09:58.864356 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:58.864221 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3116da-8cc0-40ad-8795-40cdb70741a7" containerName="cleanup" Aug 04 11:09:58.864356 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:58.864233 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3116da-8cc0-40ad-8795-40cdb70741a7" containerName="cleanup" Aug 04 11:09:58.864356 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:58.864309 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3116da-8cc0-40ad-8795-40cdb70741a7" containerName="cleanup" Aug 04 11:09:58.867123 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:58.867106 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq" Aug 04 11:09:58.869356 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:58.869334 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tgsgx\"" Aug 04 11:09:58.869491 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:58.869372 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 11:09:58.873568 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:58.873545 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq"] Aug 04 11:09:58.976634 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:58.976590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq\" (UID: \"9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq" Aug 04 11:09:59.077413 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:59.077374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq\" (UID: \"9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq" Aug 04 11:09:59.078070 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:59.078050 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq\" (UID: \"9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq" Aug 04 11:09:59.180106 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:59.180018 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq" Aug 04 11:09:59.315838 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:59.315810 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq"] Aug 04 11:09:59.317613 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:09:59.317568 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d99dc27_e0d8_4c32_a7ba_e6328dc1b71b.slice/crio-cfb06fdebcf0de4a5750e98164a6d9836116d03b4a1bb9afbef0c2bcbd4bb453 WatchSource:0}: Error finding container cfb06fdebcf0de4a5750e98164a6d9836116d03b4a1bb9afbef0c2bcbd4bb453: Status 404 returned error can't find the container with id cfb06fdebcf0de4a5750e98164a6d9836116d03b4a1bb9afbef0c2bcbd4bb453 Aug 04 11:09:59.862346 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:59.862312 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq" event={"ID":"9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b","Type":"ContainerStarted","Data":"337c22e9d726fd16cd74f888c7c54fd441003cc587e3460d596b738412afd3f2"} Aug 04 11:09:59.862545 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:59.862362 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq" event={"ID":"9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b","Type":"ContainerStarted","Data":"cfb06fdebcf0de4a5750e98164a6d9836116d03b4a1bb9afbef0c2bcbd4bb453"} Aug 04 11:09:59.879367 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:09:59.879317 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq" podStartSLOduration=1.87930263 podStartE2EDuration="1.87930263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:09:59.87743506 +0000 UTC m=+1988.975228454" watchObservedRunningTime="2026-08-04 11:09:59.87930263 +0000 UTC m=+1988.977096028" Aug 04 11:10:00.868957 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:00.868918 2576 generic.go:356] "Generic (PLEG): container finished" podID="9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b" containerID="337c22e9d726fd16cd74f888c7c54fd441003cc587e3460d596b738412afd3f2" exitCode=0 Aug 04 11:10:00.869148 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:00.869006 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq" event={"ID":"9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b","Type":"ContainerDied","Data":"337c22e9d726fd16cd74f888c7c54fd441003cc587e3460d596b738412afd3f2"} Aug 04 11:10:02.006302 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:02.006276 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq" Aug 04 11:10:02.104530 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:02.104462 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b-maas-api-service-ca\") pod \"9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b\" (UID: \"9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b\") " Aug 04 11:10:02.104839 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:02.104816 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b" (UID: "9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:10:02.205813 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:02.205725 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b-maas-api-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:10:02.881092 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:02.881062 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq" Aug 04 11:10:02.881269 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:02.881064 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq" event={"ID":"9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b","Type":"ContainerDied","Data":"cfb06fdebcf0de4a5750e98164a6d9836116d03b4a1bb9afbef0c2bcbd4bb453"} Aug 04 11:10:02.881269 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:02.881189 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfb06fdebcf0de4a5750e98164a6d9836116d03b4a1bb9afbef0c2bcbd4bb453" Aug 04 11:10:04.886353 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:04.886294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:04.988846 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:04.988815 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:05.046930 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:05.046889 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq"] Aug 04 11:10:05.049497 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:05.049476 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6279d3cc-8kjpq"] Aug 04 11:10:05.054296 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:05.054262 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:05.409833 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:05.409801 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:05.437263 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:05.437223 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b" path="/var/lib/kubelet/pods/9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b/volumes" Aug 04 11:10:05.509691 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:05.509647 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:05.706090 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:05.705999 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:05.807594 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:05.807563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:05.911949 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:05.911902 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:06.231084 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:06.231049 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:07.510132 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:07.510096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:07.889214 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:07.889176 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:10:08.662477 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.662440 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:08.791560 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.791524 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd"] Aug 04 11:10:08.791955 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.791941 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b" containerName="revoke-keys" Aug 04 11:10:08.791998 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.791956 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b" containerName="revoke-keys" Aug 04 11:10:08.792037 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.792027 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d99dc27-e0d8-4c32-a7ba-e6328dc1b71b" containerName="revoke-keys" Aug 04 11:10:08.796443 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.796426 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:08.799979 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.799954 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 11:10:08.800141 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.799980 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 11:10:08.800141 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.800136 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-93c76061-serving-cert\"" Aug 04 11:10:08.800264 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.800186 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmlb4\"" Aug 04 11:10:08.807767 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.807733 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd"] Aug 04 11:10:08.866805 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.866749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3508ae8-bff4-478d-afaf-cefac86676aa-maas-api-tls\") pod \"maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd\" (UID: \"d3508ae8-bff4-478d-afaf-cefac86676aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:08.866805 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.866805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3508ae8-bff4-478d-afaf-cefac86676aa-openshift-service-ca\") pod \"maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd\" (UID: \"d3508ae8-bff4-478d-afaf-cefac86676aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:08.867016 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.866988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8dkf\" (UniqueName: \"kubernetes.io/projected/d3508ae8-bff4-478d-afaf-cefac86676aa-kube-api-access-z8dkf\") pod \"maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd\" (UID: \"d3508ae8-bff4-478d-afaf-cefac86676aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:08.909019 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.908986 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:08.967759 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.967659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dkf\" (UniqueName: \"kubernetes.io/projected/d3508ae8-bff4-478d-afaf-cefac86676aa-kube-api-access-z8dkf\") pod \"maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd\" (UID: \"d3508ae8-bff4-478d-afaf-cefac86676aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:08.967759 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.967719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3508ae8-bff4-478d-afaf-cefac86676aa-maas-api-tls\") pod \"maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd\" (UID: \"d3508ae8-bff4-478d-afaf-cefac86676aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:08.967759 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.967747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3508ae8-bff4-478d-afaf-cefac86676aa-openshift-service-ca\") pod \"maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd\" (UID: \"d3508ae8-bff4-478d-afaf-cefac86676aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:08.968000 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:10:08.967880 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-serving-cert: secret "maas-api-e2e-migrate-93c76061-serving-cert" not found Aug 04 11:10:08.968000 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:10:08.967949 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3508ae8-bff4-478d-afaf-cefac86676aa-maas-api-tls podName:d3508ae8-bff4-478d-afaf-cefac86676aa nodeName:}" failed. No retries permitted until 2026-08-04 11:10:09.46793245 +0000 UTC m=+1998.565725824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3508ae8-bff4-478d-afaf-cefac86676aa-maas-api-tls") pod "maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" (UID: "d3508ae8-bff4-478d-afaf-cefac86676aa") : secret "maas-api-e2e-migrate-93c76061-serving-cert" not found Aug 04 11:10:08.968406 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.968388 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3508ae8-bff4-478d-afaf-cefac86676aa-openshift-service-ca\") pod \"maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd\" (UID: \"d3508ae8-bff4-478d-afaf-cefac86676aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:08.982575 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:08.982532 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8dkf\" (UniqueName: \"kubernetes.io/projected/d3508ae8-bff4-478d-afaf-cefac86676aa-kube-api-access-z8dkf\") pod \"maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd\" (UID: \"d3508ae8-bff4-478d-afaf-cefac86676aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:09.089323 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:09.089290 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:09.462312 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:09.462266 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:09.473293 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:09.473263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3508ae8-bff4-478d-afaf-cefac86676aa-maas-api-tls\") pod \"maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd\" (UID: \"d3508ae8-bff4-478d-afaf-cefac86676aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:09.475786 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:09.475760 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3508ae8-bff4-478d-afaf-cefac86676aa-maas-api-tls\") pod \"maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd\" (UID: \"d3508ae8-bff4-478d-afaf-cefac86676aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:09.560348 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:09.560308 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:09.708881 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:09.708846 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:09.855322 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:09.855297 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd"] Aug 04 11:10:09.857622 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:10:09.857575 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3508ae8_bff4_478d_afaf_cefac86676aa.slice/crio-3552ab09edbb2932d78e533d00105606b3b438879f79afc61ffe34bd5337e1dd WatchSource:0}: Error finding container 3552ab09edbb2932d78e533d00105606b3b438879f79afc61ffe34bd5337e1dd: Status 404 returned error can't find the container with id 3552ab09edbb2932d78e533d00105606b3b438879f79afc61ffe34bd5337e1dd Aug 04 11:10:09.858955 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:09.858940 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 11:10:09.924274 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:09.924238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" event={"ID":"d3508ae8-bff4-478d-afaf-cefac86676aa","Type":"ContainerStarted","Data":"3552ab09edbb2932d78e533d00105606b3b438879f79afc61ffe34bd5337e1dd"} Aug 04 11:10:12.949173 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:12.949129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" event={"ID":"d3508ae8-bff4-478d-afaf-cefac86676aa","Type":"ContainerStarted","Data":"c9daa5c9e93188974ab6da878980b50b9aeb1260c38629de13f891ef9c6e219e"} Aug 04 11:10:12.949593 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:12.949253 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:12.964962 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:12.964898 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" podStartSLOduration=2.774445525 podStartE2EDuration="4.964879226s" totalImagesPullingTime="2.190433701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:10:12.96465118 +0000 UTC m=+2002.062444590" watchObservedRunningTime="2026-08-04 11:10:12.964879226 +0000 UTC m=+2002.062672621" Aug 04 11:10:15.183697 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:15.183662 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:10:18.960200 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:18.960168 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:33.623488 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:33.623449 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj"] Aug 04 11:10:33.630917 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:33.630887 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj" Aug 04 11:10:33.633345 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:33.633319 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tgsgx\"" Aug 04 11:10:33.634682 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:33.634658 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj"] Aug 04 11:10:33.686061 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:33.686018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b135b1-edca-477a-86a3-d2515036594d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj\" (UID: \"63b135b1-edca-477a-86a3-d2515036594d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj" Aug 04 11:10:33.786782 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:33.786724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b135b1-edca-477a-86a3-d2515036594d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj\" (UID: \"63b135b1-edca-477a-86a3-d2515036594d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj" Aug 04 11:10:33.787371 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:33.787349 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b135b1-edca-477a-86a3-d2515036594d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj\" (UID: \"63b135b1-edca-477a-86a3-d2515036594d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj" Aug 04 11:10:33.945083 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:33.944983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj" Aug 04 11:10:34.089755 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:34.089731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj"] Aug 04 11:10:34.091613 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:10:34.091583 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b135b1_edca_477a_86a3_d2515036594d.slice/crio-a5d9872ecc4a4d593c8353b786bb82cb902bfc4680fa2891f1bc2b40d2dfbca6 WatchSource:0}: Error finding container a5d9872ecc4a4d593c8353b786bb82cb902bfc4680fa2891f1bc2b40d2dfbca6: Status 404 returned error can't find the container with id a5d9872ecc4a4d593c8353b786bb82cb902bfc4680fa2891f1bc2b40d2dfbca6 Aug 04 11:10:35.071110 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:35.071021 2576 generic.go:356] "Generic (PLEG): container finished" podID="63b135b1-edca-477a-86a3-d2515036594d" containerID="df1b2bc25a55d41f44adf52990cbd4f70ff6bca15394dae253dab1925c1ce653" exitCode=0 Aug 04 11:10:35.071472 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:35.071106 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj" event={"ID":"63b135b1-edca-477a-86a3-d2515036594d","Type":"ContainerDied","Data":"df1b2bc25a55d41f44adf52990cbd4f70ff6bca15394dae253dab1925c1ce653"} Aug 04 11:10:35.071472 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:35.071166 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj" event={"ID":"63b135b1-edca-477a-86a3-d2515036594d","Type":"ContainerStarted","Data":"a5d9872ecc4a4d593c8353b786bb82cb902bfc4680fa2891f1bc2b40d2dfbca6"} Aug 04 11:10:36.217080 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:36.217053 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj" Aug 04 11:10:36.310545 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:36.310514 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b135b1-edca-477a-86a3-d2515036594d-maas-api-service-ca\") pod \"63b135b1-edca-477a-86a3-d2515036594d\" (UID: \"63b135b1-edca-477a-86a3-d2515036594d\") " Aug 04 11:10:36.310830 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:36.310809 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b135b1-edca-477a-86a3-d2515036594d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63b135b1-edca-477a-86a3-d2515036594d" (UID: "63b135b1-edca-477a-86a3-d2515036594d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:10:36.411819 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:36.411785 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b135b1-edca-477a-86a3-d2515036594d-maas-api-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:10:37.083244 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:37.083215 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj" Aug 04 11:10:37.083445 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:37.083215 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj" event={"ID":"63b135b1-edca-477a-86a3-d2515036594d","Type":"ContainerDied","Data":"a5d9872ecc4a4d593c8353b786bb82cb902bfc4680fa2891f1bc2b40d2dfbca6"} Aug 04 11:10:37.083445 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:37.083337 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5d9872ecc4a4d593c8353b786bb82cb902bfc4680fa2891f1bc2b40d2dfbca6" Aug 04 11:10:39.507544 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.507492 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd"] Aug 04 11:10:39.508042 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.507768 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" podUID="d3508ae8-bff4-478d-afaf-cefac86676aa" containerName="maas-api" containerID="cri-o://c9daa5c9e93188974ab6da878980b50b9aeb1260c38629de13f891ef9c6e219e" gracePeriod=30 Aug 04 11:10:39.546136 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.546101 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:39.714180 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.714136 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj"] Aug 04 11:10:39.716406 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.716380 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-93c76061-wtrmj"] Aug 04 11:10:39.762342 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.762274 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:39.841440 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.841409 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8dkf\" (UniqueName: \"kubernetes.io/projected/d3508ae8-bff4-478d-afaf-cefac86676aa-kube-api-access-z8dkf\") pod \"d3508ae8-bff4-478d-afaf-cefac86676aa\" (UID: \"d3508ae8-bff4-478d-afaf-cefac86676aa\") " Aug 04 11:10:39.841662 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.841481 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3508ae8-bff4-478d-afaf-cefac86676aa-openshift-service-ca\") pod \"d3508ae8-bff4-478d-afaf-cefac86676aa\" (UID: \"d3508ae8-bff4-478d-afaf-cefac86676aa\") " Aug 04 11:10:39.841662 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.841553 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3508ae8-bff4-478d-afaf-cefac86676aa-maas-api-tls\") pod \"d3508ae8-bff4-478d-afaf-cefac86676aa\" (UID: \"d3508ae8-bff4-478d-afaf-cefac86676aa\") " Aug 04 11:10:39.841922 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.841897 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3508ae8-bff4-478d-afaf-cefac86676aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3508ae8-bff4-478d-afaf-cefac86676aa" (UID: "d3508ae8-bff4-478d-afaf-cefac86676aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:10:39.843906 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.843875 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3508ae8-bff4-478d-afaf-cefac86676aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3508ae8-bff4-478d-afaf-cefac86676aa" (UID: "d3508ae8-bff4-478d-afaf-cefac86676aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:10:39.844018 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.843989 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3508ae8-bff4-478d-afaf-cefac86676aa-kube-api-access-z8dkf" (OuterVolumeSpecName: "kube-api-access-z8dkf") pod "d3508ae8-bff4-478d-afaf-cefac86676aa" (UID: "d3508ae8-bff4-478d-afaf-cefac86676aa"). InnerVolumeSpecName "kube-api-access-z8dkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:10:39.943139 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.943104 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3508ae8-bff4-478d-afaf-cefac86676aa-openshift-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:10:39.943139 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.943133 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3508ae8-bff4-478d-afaf-cefac86676aa-maas-api-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:10:39.943139 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:39.943145 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8dkf\" (UniqueName: \"kubernetes.io/projected/d3508ae8-bff4-478d-afaf-cefac86676aa-kube-api-access-z8dkf\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:10:40.100451 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:40.100412 2576 generic.go:356] "Generic (PLEG): container finished" podID="d3508ae8-bff4-478d-afaf-cefac86676aa" containerID="c9daa5c9e93188974ab6da878980b50b9aeb1260c38629de13f891ef9c6e219e" exitCode=0 Aug 04 11:10:40.100662 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:40.100457 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" Aug 04 11:10:40.100662 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:40.100528 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" event={"ID":"d3508ae8-bff4-478d-afaf-cefac86676aa","Type":"ContainerDied","Data":"c9daa5c9e93188974ab6da878980b50b9aeb1260c38629de13f891ef9c6e219e"} Aug 04 11:10:40.100662 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:40.100582 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd" event={"ID":"d3508ae8-bff4-478d-afaf-cefac86676aa","Type":"ContainerDied","Data":"3552ab09edbb2932d78e533d00105606b3b438879f79afc61ffe34bd5337e1dd"} Aug 04 11:10:40.100662 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:40.100611 2576 scope.go:117] "RemoveContainer" containerID="c9daa5c9e93188974ab6da878980b50b9aeb1260c38629de13f891ef9c6e219e" Aug 04 11:10:40.110376 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:40.110355 2576 scope.go:117] "RemoveContainer" containerID="c9daa5c9e93188974ab6da878980b50b9aeb1260c38629de13f891ef9c6e219e" Aug 04 11:10:40.110696 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:10:40.110674 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9daa5c9e93188974ab6da878980b50b9aeb1260c38629de13f891ef9c6e219e\": container with ID starting with c9daa5c9e93188974ab6da878980b50b9aeb1260c38629de13f891ef9c6e219e not found: ID does not exist" containerID="c9daa5c9e93188974ab6da878980b50b9aeb1260c38629de13f891ef9c6e219e" Aug 04 11:10:40.110802 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:40.110704 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9daa5c9e93188974ab6da878980b50b9aeb1260c38629de13f891ef9c6e219e"} err="failed to get container status \"c9daa5c9e93188974ab6da878980b50b9aeb1260c38629de13f891ef9c6e219e\": rpc error: code = NotFound desc = could not find container \"c9daa5c9e93188974ab6da878980b50b9aeb1260c38629de13f891ef9c6e219e\": container with ID starting with c9daa5c9e93188974ab6da878980b50b9aeb1260c38629de13f891ef9c6e219e not found: ID does not exist" Aug 04 11:10:40.125784 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:40.125752 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd"] Aug 04 11:10:40.131921 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:40.131894 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-93c76061-65bddb9954-gjhqd"] Aug 04 11:10:41.438459 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:41.438423 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63b135b1-edca-477a-86a3-d2515036594d" path="/var/lib/kubelet/pods/63b135b1-edca-477a-86a3-d2515036594d/volumes" Aug 04 11:10:41.438849 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:41.438836 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3508ae8-bff4-478d-afaf-cefac86676aa" path="/var/lib/kubelet/pods/d3508ae8-bff4-478d-afaf-cefac86676aa/volumes" Aug 04 11:10:41.935175 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:41.935143 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:42.478819 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:42.478786 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:10:42.659490 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:42.659450 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:42.754933 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:42.754837 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:43.737320 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.737286 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw"] Aug 04 11:10:43.737966 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.737945 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63b135b1-edca-477a-86a3-d2515036594d" containerName="revoke-keys" Aug 04 11:10:43.738016 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.737971 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63b135b1-edca-477a-86a3-d2515036594d" containerName="revoke-keys" Aug 04 11:10:43.738016 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.737988 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3508ae8-bff4-478d-afaf-cefac86676aa" containerName="maas-api" Aug 04 11:10:43.738016 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.737997 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3508ae8-bff4-478d-afaf-cefac86676aa" containerName="maas-api" Aug 04 11:10:43.738121 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.738108 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="63b135b1-edca-477a-86a3-d2515036594d" containerName="revoke-keys" Aug 04 11:10:43.738156 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.738130 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3508ae8-bff4-478d-afaf-cefac86676aa" containerName="maas-api" Aug 04 11:10:43.743210 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.743190 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:10:43.745534 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.745497 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmlb4\"" Aug 04 11:10:43.745732 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.745528 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9269199f-serving-cert\"" Aug 04 11:10:43.745849 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.745609 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 11:10:43.746267 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.746252 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 11:10:43.751442 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.751418 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw"] Aug 04 11:10:43.776334 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.776299 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:43.781340 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.781310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a116db-6a05-423d-ab55-5d1f6b4e15ae-openshift-service-ca\") pod \"maas-api-e2e-ait-9269199f-5d74d68555-kfplw\" (UID: \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:10:43.781494 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.781399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9z47\" (UniqueName: \"kubernetes.io/projected/91a116db-6a05-423d-ab55-5d1f6b4e15ae-kube-api-access-b9z47\") pod \"maas-api-e2e-ait-9269199f-5d74d68555-kfplw\" (UID: \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:10:43.781494 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.781429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a116db-6a05-423d-ab55-5d1f6b4e15ae-maas-api-tls\") pod \"maas-api-e2e-ait-9269199f-5d74d68555-kfplw\" (UID: \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:10:43.875319 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.875278 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:43.882363 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.882341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a116db-6a05-423d-ab55-5d1f6b4e15ae-openshift-service-ca\") pod \"maas-api-e2e-ait-9269199f-5d74d68555-kfplw\" (UID: \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:10:43.882537 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.882518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9z47\" (UniqueName: \"kubernetes.io/projected/91a116db-6a05-423d-ab55-5d1f6b4e15ae-kube-api-access-b9z47\") pod \"maas-api-e2e-ait-9269199f-5d74d68555-kfplw\" (UID: \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:10:43.882584 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.882567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a116db-6a05-423d-ab55-5d1f6b4e15ae-maas-api-tls\") pod \"maas-api-e2e-ait-9269199f-5d74d68555-kfplw\" (UID: \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:10:43.883001 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.882984 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a116db-6a05-423d-ab55-5d1f6b4e15ae-openshift-service-ca\") pod \"maas-api-e2e-ait-9269199f-5d74d68555-kfplw\" (UID: \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:10:43.884897 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.884876 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a116db-6a05-423d-ab55-5d1f6b4e15ae-maas-api-tls\") pod \"maas-api-e2e-ait-9269199f-5d74d68555-kfplw\" (UID: \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:10:43.899089 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:43.899068 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9z47\" (UniqueName: \"kubernetes.io/projected/91a116db-6a05-423d-ab55-5d1f6b4e15ae-kube-api-access-b9z47\") pod \"maas-api-e2e-ait-9269199f-5d74d68555-kfplw\" (UID: \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:10:44.056171 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:44.056090 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:10:44.194984 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:44.194949 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw"] Aug 04 11:10:44.196884 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:10:44.196850 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a116db_6a05_423d_ab55_5d1f6b4e15ae.slice/crio-f452d24e6cd9187c90218e0bc0c948a55d615055ac989b41a5278b25a1a825d8 WatchSource:0}: Error finding container f452d24e6cd9187c90218e0bc0c948a55d615055ac989b41a5278b25a1a825d8: Status 404 returned error can't find the container with id f452d24e6cd9187c90218e0bc0c948a55d615055ac989b41a5278b25a1a825d8 Aug 04 11:10:44.292400 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:44.292365 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:44.385041 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:44.385006 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:10:45.133280 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:45.133238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" event={"ID":"91a116db-6a05-423d-ab55-5d1f6b4e15ae","Type":"ContainerStarted","Data":"f452d24e6cd9187c90218e0bc0c948a55d615055ac989b41a5278b25a1a825d8"} Aug 04 11:10:46.140766 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:46.140729 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" event={"ID":"91a116db-6a05-423d-ab55-5d1f6b4e15ae","Type":"ContainerStarted","Data":"69e7e543b1fe1c2712fa1e50265245d24a0487b96c53e91946b8a0790e0e294f"} Aug 04 11:10:46.141224 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:46.140795 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:10:46.159157 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:46.159103 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" podStartSLOduration=1.780338361 podStartE2EDuration="3.159086373s" totalImagesPullingTime="1.378748012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:10:46.156657381 +0000 UTC m=+2035.254450774" watchObservedRunningTime="2026-08-04 11:10:46.159086373 +0000 UTC m=+2035.256879767" Aug 04 11:10:49.986816 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:49.986763 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:10:52.152168 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:10:52.152138 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:11:02.477787 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:02.477743 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:24.379431 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.379348 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw"] Aug 04 11:11:24.380017 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.379611 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" podUID="91a116db-6a05-423d-ab55-5d1f6b4e15ae" containerName="maas-api" containerID="cri-o://69e7e543b1fe1c2712fa1e50265245d24a0487b96c53e91946b8a0790e0e294f" gracePeriod=30 Aug 04 11:11:24.411630 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.411597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:24.511999 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.511949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:24.575798 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.575763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:24.634423 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.634362 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:11:24.642935 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.642913 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a116db-6a05-423d-ab55-5d1f6b4e15ae-openshift-service-ca\") pod \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\" (UID: \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\") " Aug 04 11:11:24.643082 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.642941 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9z47\" (UniqueName: \"kubernetes.io/projected/91a116db-6a05-423d-ab55-5d1f6b4e15ae-kube-api-access-b9z47\") pod \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\" (UID: \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\") " Aug 04 11:11:24.643082 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.642970 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a116db-6a05-423d-ab55-5d1f6b4e15ae-maas-api-tls\") pod \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\" (UID: \"91a116db-6a05-423d-ab55-5d1f6b4e15ae\") " Aug 04 11:11:24.643285 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.643265 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a116db-6a05-423d-ab55-5d1f6b4e15ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91a116db-6a05-423d-ab55-5d1f6b4e15ae" (UID: "91a116db-6a05-423d-ab55-5d1f6b4e15ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:11:24.644980 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.644953 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a116db-6a05-423d-ab55-5d1f6b4e15ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91a116db-6a05-423d-ab55-5d1f6b4e15ae" (UID: "91a116db-6a05-423d-ab55-5d1f6b4e15ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:11:24.645078 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.645024 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a116db-6a05-423d-ab55-5d1f6b4e15ae-kube-api-access-b9z47" (OuterVolumeSpecName: "kube-api-access-b9z47") pod "91a116db-6a05-423d-ab55-5d1f6b4e15ae" (UID: "91a116db-6a05-423d-ab55-5d1f6b4e15ae"). InnerVolumeSpecName "kube-api-access-b9z47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:11:24.744084 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.744047 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91a116db-6a05-423d-ab55-5d1f6b4e15ae-openshift-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:11:24.744084 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.744079 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9z47\" (UniqueName: \"kubernetes.io/projected/91a116db-6a05-423d-ab55-5d1f6b4e15ae-kube-api-access-b9z47\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:11:24.744084 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:24.744089 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91a116db-6a05-423d-ab55-5d1f6b4e15ae-maas-api-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:11:25.371406 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:25.371369 2576 generic.go:356] "Generic (PLEG): container finished" podID="91a116db-6a05-423d-ab55-5d1f6b4e15ae" containerID="69e7e543b1fe1c2712fa1e50265245d24a0487b96c53e91946b8a0790e0e294f" exitCode=0 Aug 04 11:11:25.371406 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:25.371412 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" Aug 04 11:11:25.371659 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:25.371452 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" event={"ID":"91a116db-6a05-423d-ab55-5d1f6b4e15ae","Type":"ContainerDied","Data":"69e7e543b1fe1c2712fa1e50265245d24a0487b96c53e91946b8a0790e0e294f"} Aug 04 11:11:25.371659 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:25.371514 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw" event={"ID":"91a116db-6a05-423d-ab55-5d1f6b4e15ae","Type":"ContainerDied","Data":"f452d24e6cd9187c90218e0bc0c948a55d615055ac989b41a5278b25a1a825d8"} Aug 04 11:11:25.371659 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:25.371545 2576 scope.go:117] "RemoveContainer" containerID="69e7e543b1fe1c2712fa1e50265245d24a0487b96c53e91946b8a0790e0e294f" Aug 04 11:11:25.381988 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:25.381789 2576 scope.go:117] "RemoveContainer" containerID="69e7e543b1fe1c2712fa1e50265245d24a0487b96c53e91946b8a0790e0e294f" Aug 04 11:11:25.382226 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:11:25.382053 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e7e543b1fe1c2712fa1e50265245d24a0487b96c53e91946b8a0790e0e294f\": container with ID starting with 69e7e543b1fe1c2712fa1e50265245d24a0487b96c53e91946b8a0790e0e294f not found: ID does not exist" containerID="69e7e543b1fe1c2712fa1e50265245d24a0487b96c53e91946b8a0790e0e294f" Aug 04 11:11:25.382226 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:25.382076 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e7e543b1fe1c2712fa1e50265245d24a0487b96c53e91946b8a0790e0e294f"} err="failed to get container status \"69e7e543b1fe1c2712fa1e50265245d24a0487b96c53e91946b8a0790e0e294f\": rpc error: code = NotFound desc = could not find container \"69e7e543b1fe1c2712fa1e50265245d24a0487b96c53e91946b8a0790e0e294f\": container with ID starting with 69e7e543b1fe1c2712fa1e50265245d24a0487b96c53e91946b8a0790e0e294f not found: ID does not exist" Aug 04 11:11:25.393631 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:25.393596 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw"] Aug 04 11:11:25.397804 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:25.397781 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9269199f-5d74d68555-kfplw"] Aug 04 11:11:25.437881 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:25.437850 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91a116db-6a05-423d-ab55-5d1f6b4e15ae" path="/var/lib/kubelet/pods/91a116db-6a05-423d-ab55-5d1f6b4e15ae/volumes" Aug 04 11:11:25.724124 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:25.724031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:25.969615 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:25.969576 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:26.194042 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:26.194005 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:26.560318 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:26.560209 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:27.184974 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:27.184938 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:11:27.438042 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:27.437956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:28.988347 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:28.988311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:29.082371 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:29.082331 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:30.067516 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.067475 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp"] Aug 04 11:11:30.067885 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.067871 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91a116db-6a05-423d-ab55-5d1f6b4e15ae" containerName="maas-api" Aug 04 11:11:30.067929 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.067889 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="91a116db-6a05-423d-ab55-5d1f6b4e15ae" containerName="maas-api" Aug 04 11:11:30.067972 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.067963 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="91a116db-6a05-423d-ab55-5d1f6b4e15ae" containerName="maas-api" Aug 04 11:11:30.071253 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.071233 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:11:30.073950 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.073927 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-dfeef193-serving-cert\"" Aug 04 11:11:30.074048 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.073965 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 11:11:30.074729 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.074714 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 11:11:30.074834 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.074808 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmlb4\"" Aug 04 11:11:30.082407 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.082381 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp"] Aug 04 11:11:30.088058 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.088036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d4c261-8e9e-4819-8b16-31ceda7995ab-openshift-service-ca\") pod \"maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp\" (UID: \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:11:30.088162 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.088079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d4c261-8e9e-4819-8b16-31ceda7995ab-maas-api-tls\") pod \"maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp\" (UID: \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:11:30.088162 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.088122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5444d\" (UniqueName: \"kubernetes.io/projected/c7d4c261-8e9e-4819-8b16-31ceda7995ab-kube-api-access-5444d\") pod \"maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp\" (UID: \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:11:30.100479 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.100434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:30.188709 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.188671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d4c261-8e9e-4819-8b16-31ceda7995ab-openshift-service-ca\") pod \"maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp\" (UID: \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:11:30.188907 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.188726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d4c261-8e9e-4819-8b16-31ceda7995ab-maas-api-tls\") pod \"maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp\" (UID: \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:11:30.188907 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.188746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5444d\" (UniqueName: \"kubernetes.io/projected/c7d4c261-8e9e-4819-8b16-31ceda7995ab-kube-api-access-5444d\") pod \"maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp\" (UID: \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:11:30.188907 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:11:30.188849 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-serving-cert: secret "maas-api-e2e-derive-dfeef193-serving-cert" not found Aug 04 11:11:30.189023 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:11:30.188942 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d4c261-8e9e-4819-8b16-31ceda7995ab-maas-api-tls podName:c7d4c261-8e9e-4819-8b16-31ceda7995ab nodeName:}" failed. No retries permitted until 2026-08-04 11:11:30.688921819 +0000 UTC m=+2079.786715195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7d4c261-8e9e-4819-8b16-31ceda7995ab-maas-api-tls") pod "maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" (UID: "c7d4c261-8e9e-4819-8b16-31ceda7995ab") : secret "maas-api-e2e-derive-dfeef193-serving-cert" not found Aug 04 11:11:30.189391 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.189373 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d4c261-8e9e-4819-8b16-31ceda7995ab-openshift-service-ca\") pod \"maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp\" (UID: \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:11:30.198756 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.198734 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5444d\" (UniqueName: \"kubernetes.io/projected/c7d4c261-8e9e-4819-8b16-31ceda7995ab-kube-api-access-5444d\") pod \"maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp\" (UID: \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:11:30.279408 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.279374 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:30.635298 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.635259 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:30.692679 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.692641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d4c261-8e9e-4819-8b16-31ceda7995ab-maas-api-tls\") pod \"maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp\" (UID: \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:11:30.695057 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.695035 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d4c261-8e9e-4819-8b16-31ceda7995ab-maas-api-tls\") pod \"maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp\" (UID: \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:11:30.738213 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.738179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:11:30.983527 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:30.983420 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:11:31.120705 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:31.120681 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp"] Aug 04 11:11:31.122731 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:11:31.122704 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d4c261_8e9e_4819_8b16_31ceda7995ab.slice/crio-3150ad4ac86777a946dc116d92b2c927f0604d66bfeae1a63891d82e8d355aff WatchSource:0}: Error finding container 3150ad4ac86777a946dc116d92b2c927f0604d66bfeae1a63891d82e8d355aff: Status 404 returned error can't find the container with id 3150ad4ac86777a946dc116d92b2c927f0604d66bfeae1a63891d82e8d355aff Aug 04 11:11:31.409743 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:31.409709 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" event={"ID":"c7d4c261-8e9e-4819-8b16-31ceda7995ab","Type":"ContainerStarted","Data":"3150ad4ac86777a946dc116d92b2c927f0604d66bfeae1a63891d82e8d355aff"} Aug 04 11:11:32.416468 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:32.416433 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" event={"ID":"c7d4c261-8e9e-4819-8b16-31ceda7995ab","Type":"ContainerStarted","Data":"c90a0cc24790c86da0c64c9830d970f0138907b6bf911fc8531644f0f652b24f"} Aug 04 11:11:32.416887 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:32.416537 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:11:32.434726 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:32.434672 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" podStartSLOduration=1.975241665 podStartE2EDuration="2.434656946s" totalImagesPullingTime="459.415281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:11:32.433112197 +0000 UTC m=+2081.530905597" watchObservedRunningTime="2026-08-04 11:11:32.434656946 +0000 UTC m=+2081.532450400" Aug 04 11:11:33.681766 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:33.681734 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:11:38.428263 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:38.428233 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:11:49.445629 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:11:49.445596 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:00.691102 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:00.691066 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp"] Aug 04 11:12:00.691668 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:00.691292 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" podUID="c7d4c261-8e9e-4819-8b16-31ceda7995ab" containerName="maas-api" containerID="cri-o://c90a0cc24790c86da0c64c9830d970f0138907b6bf911fc8531644f0f652b24f" gracePeriod=30 Aug 04 11:12:00.728071 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:00.728025 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:00.824845 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:00.824809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:00.890755 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:00.890722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:00.945812 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:00.945742 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:12:00.955422 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:00.955390 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:01.040689 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.040653 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:01.070917 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.070887 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5444d\" (UniqueName: \"kubernetes.io/projected/c7d4c261-8e9e-4819-8b16-31ceda7995ab-kube-api-access-5444d\") pod \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\" (UID: \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\") " Aug 04 11:12:01.071115 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.070997 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d4c261-8e9e-4819-8b16-31ceda7995ab-maas-api-tls\") pod \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\" (UID: \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\") " Aug 04 11:12:01.071115 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.071038 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d4c261-8e9e-4819-8b16-31ceda7995ab-openshift-service-ca\") pod \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\" (UID: \"c7d4c261-8e9e-4819-8b16-31ceda7995ab\") " Aug 04 11:12:01.071648 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.071617 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d4c261-8e9e-4819-8b16-31ceda7995ab-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7d4c261-8e9e-4819-8b16-31ceda7995ab" (UID: "c7d4c261-8e9e-4819-8b16-31ceda7995ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:12:01.073452 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.073428 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d4c261-8e9e-4819-8b16-31ceda7995ab-kube-api-access-5444d" (OuterVolumeSpecName: "kube-api-access-5444d") pod "c7d4c261-8e9e-4819-8b16-31ceda7995ab" (UID: "c7d4c261-8e9e-4819-8b16-31ceda7995ab"). InnerVolumeSpecName "kube-api-access-5444d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:12:01.073583 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.073454 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d4c261-8e9e-4819-8b16-31ceda7995ab-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7d4c261-8e9e-4819-8b16-31ceda7995ab" (UID: "c7d4c261-8e9e-4819-8b16-31ceda7995ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:12:01.172145 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.172097 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5444d\" (UniqueName: \"kubernetes.io/projected/c7d4c261-8e9e-4819-8b16-31ceda7995ab-kube-api-access-5444d\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:12:01.172145 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.172139 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7d4c261-8e9e-4819-8b16-31ceda7995ab-maas-api-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:12:01.172145 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.172148 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7d4c261-8e9e-4819-8b16-31ceda7995ab-openshift-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:12:01.293007 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.292914 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:01.588021 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.587991 2576 generic.go:356] "Generic (PLEG): container finished" podID="c7d4c261-8e9e-4819-8b16-31ceda7995ab" containerID="c90a0cc24790c86da0c64c9830d970f0138907b6bf911fc8531644f0f652b24f" exitCode=0 Aug 04 11:12:01.588207 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.588042 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" Aug 04 11:12:01.588207 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.588078 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" event={"ID":"c7d4c261-8e9e-4819-8b16-31ceda7995ab","Type":"ContainerDied","Data":"c90a0cc24790c86da0c64c9830d970f0138907b6bf911fc8531644f0f652b24f"} Aug 04 11:12:01.588207 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.588135 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp" event={"ID":"c7d4c261-8e9e-4819-8b16-31ceda7995ab","Type":"ContainerDied","Data":"3150ad4ac86777a946dc116d92b2c927f0604d66bfeae1a63891d82e8d355aff"} Aug 04 11:12:01.588207 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.588164 2576 scope.go:117] "RemoveContainer" containerID="c90a0cc24790c86da0c64c9830d970f0138907b6bf911fc8531644f0f652b24f" Aug 04 11:12:01.597491 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.597467 2576 scope.go:117] "RemoveContainer" containerID="c90a0cc24790c86da0c64c9830d970f0138907b6bf911fc8531644f0f652b24f" Aug 04 11:12:01.597806 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:12:01.597783 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90a0cc24790c86da0c64c9830d970f0138907b6bf911fc8531644f0f652b24f\": container with ID starting with c90a0cc24790c86da0c64c9830d970f0138907b6bf911fc8531644f0f652b24f not found: ID does not exist" containerID="c90a0cc24790c86da0c64c9830d970f0138907b6bf911fc8531644f0f652b24f" Aug 04 11:12:01.597875 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.597814 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90a0cc24790c86da0c64c9830d970f0138907b6bf911fc8531644f0f652b24f"} err="failed to get container status \"c90a0cc24790c86da0c64c9830d970f0138907b6bf911fc8531644f0f652b24f\": rpc error: code = NotFound desc = could not find container \"c90a0cc24790c86da0c64c9830d970f0138907b6bf911fc8531644f0f652b24f\": container with ID starting with c90a0cc24790c86da0c64c9830d970f0138907b6bf911fc8531644f0f652b24f not found: ID does not exist" Aug 04 11:12:01.604494 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.604467 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp"] Aug 04 11:12:01.609877 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.609849 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-dfeef193-648b7f6f6-z9pbp"] Aug 04 11:12:01.763803 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.763770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:01.863614 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:01.863523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:03.437745 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:03.437711 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d4c261-8e9e-4819-8b16-31ceda7995ab" path="/var/lib/kubelet/pods/c7d4c261-8e9e-4819-8b16-31ceda7995ab/volumes" Aug 04 11:12:03.679942 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:03.679881 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:12:13.517811 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:13.517777 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:16.481902 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:16.481865 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:12:20.828278 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:20.828237 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:20.940325 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:20.940286 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:21.045300 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:21.045262 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:21.188592 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:21.188553 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:21.545173 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:21.545081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:47.521200 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:47.521166 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:54.336060 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:54.336026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:54.453101 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:54.453050 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:12:54.544240 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:12:54.544207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:10.428725 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:10.428686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:16.417633 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:16.417591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:16.488979 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:16.488945 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:16.595437 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:16.595391 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:16.683000 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:16.682906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:27.208235 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:27.208198 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:38.186213 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:38.186181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:44.219804 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:44.219771 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:44.323336 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:44.323296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:44.426391 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:44.426358 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:44.526481 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:44.526379 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:45.205744 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:45.205707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:45.295830 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:45.295798 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:45.399000 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:45.398962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:46.856136 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:46.856100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:48.888272 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:48.888233 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:13:53.335084 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:53.335044 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:53.435348 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:53.435315 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:53.999304 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:53.999271 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:13:55.680881 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:13:55.680840 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:14:00.929842 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:00.929801 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:01.031242 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:01.031207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:01.170325 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:01.170290 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:01.536914 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:01.536876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:02.096443 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:02.096408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:04.307177 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:04.307144 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:04.407360 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:04.407318 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:05.329367 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.329336 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns"] Aug 04 11:14:05.329848 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.329760 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d4c261-8e9e-4819-8b16-31ceda7995ab" containerName="maas-api" Aug 04 11:14:05.329848 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.329772 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d4c261-8e9e-4819-8b16-31ceda7995ab" containerName="maas-api" Aug 04 11:14:05.329962 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.329864 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d4c261-8e9e-4819-8b16-31ceda7995ab" containerName="maas-api" Aug 04 11:14:05.333223 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.333204 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:05.335807 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.335783 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 11:14:05.335938 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.335844 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7d02d259-serving-cert\"" Aug 04 11:14:05.335938 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.335806 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 11:14:05.335938 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.335918 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmlb4\"" Aug 04 11:14:05.344568 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.344527 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns"] Aug 04 11:14:05.378839 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.378788 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:05.402523 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.402474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563ab2a8-46cc-4600-bf9d-1446cfd860e1-openshift-service-ca\") pod \"maas-api-e2e-mt-7d02d259-5976b77db-9fqns\" (UID: \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:05.402704 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.402550 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563ab2a8-46cc-4600-bf9d-1446cfd860e1-maas-api-tls\") pod \"maas-api-e2e-mt-7d02d259-5976b77db-9fqns\" (UID: \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:05.402704 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.402610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6sng\" (UniqueName: \"kubernetes.io/projected/563ab2a8-46cc-4600-bf9d-1446cfd860e1-kube-api-access-v6sng\") pod \"maas-api-e2e-mt-7d02d259-5976b77db-9fqns\" (UID: \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:05.503464 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.503427 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563ab2a8-46cc-4600-bf9d-1446cfd860e1-openshift-service-ca\") pod \"maas-api-e2e-mt-7d02d259-5976b77db-9fqns\" (UID: \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:05.503464 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.503467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563ab2a8-46cc-4600-bf9d-1446cfd860e1-maas-api-tls\") pod \"maas-api-e2e-mt-7d02d259-5976b77db-9fqns\" (UID: \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:05.503744 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.503568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6sng\" (UniqueName: \"kubernetes.io/projected/563ab2a8-46cc-4600-bf9d-1446cfd860e1-kube-api-access-v6sng\") pod \"maas-api-e2e-mt-7d02d259-5976b77db-9fqns\" (UID: \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:05.504199 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.504179 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563ab2a8-46cc-4600-bf9d-1446cfd860e1-openshift-service-ca\") pod \"maas-api-e2e-mt-7d02d259-5976b77db-9fqns\" (UID: \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:05.506001 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.505978 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563ab2a8-46cc-4600-bf9d-1446cfd860e1-maas-api-tls\") pod \"maas-api-e2e-mt-7d02d259-5976b77db-9fqns\" (UID: \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:05.511547 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.511524 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6sng\" (UniqueName: \"kubernetes.io/projected/563ab2a8-46cc-4600-bf9d-1446cfd860e1-kube-api-access-v6sng\") pod \"maas-api-e2e-mt-7d02d259-5976b77db-9fqns\" (UID: \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:05.646909 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.646869 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:05.670677 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.670624 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:05.788469 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.788442 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns"] Aug 04 11:14:05.790308 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:14:05.790278 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563ab2a8_46cc_4600_bf9d_1446cfd860e1.slice/crio-6edf9540f848631bb03e2f7b47a463079268cf962212fb44224d71917a2d8c80 WatchSource:0}: Error finding container 6edf9540f848631bb03e2f7b47a463079268cf962212fb44224d71917a2d8c80: Status 404 returned error can't find the container with id 6edf9540f848631bb03e2f7b47a463079268cf962212fb44224d71917a2d8c80 Aug 04 11:14:05.862153 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.862114 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:05.954419 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:05.954330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:06.315919 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:06.315825 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" event={"ID":"563ab2a8-46cc-4600-bf9d-1446cfd860e1","Type":"ContainerStarted","Data":"6edf9540f848631bb03e2f7b47a463079268cf962212fb44224d71917a2d8c80"} Aug 04 11:14:07.322545 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:07.322484 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" event={"ID":"563ab2a8-46cc-4600-bf9d-1446cfd860e1","Type":"ContainerStarted","Data":"bfc4c1fb7229bfa1abf9fc543df9af2ae46cefb910b97678f8bfe6904a22dec3"} Aug 04 11:14:07.322949 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:07.322569 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:07.339608 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:07.339565 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" podStartSLOduration=1.594305926 podStartE2EDuration="2.339549769s" totalImagesPullingTime="745.243843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:14:07.338531689 +0000 UTC m=+2236.436325082" watchObservedRunningTime="2026-08-04 11:14:07.339549769 +0000 UTC m=+2236.437343158" Aug 04 11:14:08.795875 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:08.795838 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:14:12.464202 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:12.464167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:13.334333 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:13.334262 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:29.196920 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:29.196889 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:29.839737 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:29.839698 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:29.942336 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:29.942289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:30.294522 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:30.294406 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:36.219832 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:36.219795 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:36.321443 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:36.321394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:47.729428 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:47.729392 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns"] Aug 04 11:14:47.729932 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:47.729684 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" podUID="563ab2a8-46cc-4600-bf9d-1446cfd860e1" containerName="maas-api" containerID="cri-o://bfc4c1fb7229bfa1abf9fc543df9af2ae46cefb910b97678f8bfe6904a22dec3" gracePeriod=30 Aug 04 11:14:47.756141 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:47.756113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:47.855927 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:47.855894 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:47.923626 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:47.923522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:47.980270 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:47.980206 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:47.988134 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:47.988103 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:47.992050 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:47.992027 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563ab2a8-46cc-4600-bf9d-1446cfd860e1-openshift-service-ca\") pod \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\" (UID: \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\") " Aug 04 11:14:47.992145 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:47.992112 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6sng\" (UniqueName: \"kubernetes.io/projected/563ab2a8-46cc-4600-bf9d-1446cfd860e1-kube-api-access-v6sng\") pod \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\" (UID: \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\") " Aug 04 11:14:47.992226 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:47.992206 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563ab2a8-46cc-4600-bf9d-1446cfd860e1-maas-api-tls\") pod \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\" (UID: \"563ab2a8-46cc-4600-bf9d-1446cfd860e1\") " Aug 04 11:14:47.992422 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:47.992395 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563ab2a8-46cc-4600-bf9d-1446cfd860e1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "563ab2a8-46cc-4600-bf9d-1446cfd860e1" (UID: "563ab2a8-46cc-4600-bf9d-1446cfd860e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:14:47.994163 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:47.994141 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563ab2a8-46cc-4600-bf9d-1446cfd860e1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "563ab2a8-46cc-4600-bf9d-1446cfd860e1" (UID: "563ab2a8-46cc-4600-bf9d-1446cfd860e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:14:47.994267 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:47.994235 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563ab2a8-46cc-4600-bf9d-1446cfd860e1-kube-api-access-v6sng" (OuterVolumeSpecName: "kube-api-access-v6sng") pod "563ab2a8-46cc-4600-bf9d-1446cfd860e1" (UID: "563ab2a8-46cc-4600-bf9d-1446cfd860e1"). InnerVolumeSpecName "kube-api-access-v6sng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:14:48.093775 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:48.093739 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/563ab2a8-46cc-4600-bf9d-1446cfd860e1-openshift-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:14:48.093775 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:48.093766 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6sng\" (UniqueName: \"kubernetes.io/projected/563ab2a8-46cc-4600-bf9d-1446cfd860e1-kube-api-access-v6sng\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:14:48.093775 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:48.093776 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/563ab2a8-46cc-4600-bf9d-1446cfd860e1-maas-api-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:14:48.317985 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:48.317885 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:48.556200 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:48.556163 2576 generic.go:356] "Generic (PLEG): container finished" podID="563ab2a8-46cc-4600-bf9d-1446cfd860e1" containerID="bfc4c1fb7229bfa1abf9fc543df9af2ae46cefb910b97678f8bfe6904a22dec3" exitCode=0 Aug 04 11:14:48.556369 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:48.556210 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" Aug 04 11:14:48.556369 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:48.556253 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" event={"ID":"563ab2a8-46cc-4600-bf9d-1446cfd860e1","Type":"ContainerDied","Data":"bfc4c1fb7229bfa1abf9fc543df9af2ae46cefb910b97678f8bfe6904a22dec3"} Aug 04 11:14:48.556369 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:48.556308 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns" event={"ID":"563ab2a8-46cc-4600-bf9d-1446cfd860e1","Type":"ContainerDied","Data":"6edf9540f848631bb03e2f7b47a463079268cf962212fb44224d71917a2d8c80"} Aug 04 11:14:48.556369 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:48.556334 2576 scope.go:117] "RemoveContainer" containerID="bfc4c1fb7229bfa1abf9fc543df9af2ae46cefb910b97678f8bfe6904a22dec3" Aug 04 11:14:48.566374 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:48.566355 2576 scope.go:117] "RemoveContainer" containerID="bfc4c1fb7229bfa1abf9fc543df9af2ae46cefb910b97678f8bfe6904a22dec3" Aug 04 11:14:48.566651 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:14:48.566631 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc4c1fb7229bfa1abf9fc543df9af2ae46cefb910b97678f8bfe6904a22dec3\": container with ID starting with bfc4c1fb7229bfa1abf9fc543df9af2ae46cefb910b97678f8bfe6904a22dec3 not found: ID does not exist" containerID="bfc4c1fb7229bfa1abf9fc543df9af2ae46cefb910b97678f8bfe6904a22dec3" Aug 04 11:14:48.566727 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:48.566661 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc4c1fb7229bfa1abf9fc543df9af2ae46cefb910b97678f8bfe6904a22dec3"} err="failed to get container status \"bfc4c1fb7229bfa1abf9fc543df9af2ae46cefb910b97678f8bfe6904a22dec3\": rpc error: code = NotFound desc = could not find container \"bfc4c1fb7229bfa1abf9fc543df9af2ae46cefb910b97678f8bfe6904a22dec3\": container with ID starting with bfc4c1fb7229bfa1abf9fc543df9af2ae46cefb910b97678f8bfe6904a22dec3 not found: ID does not exist" Aug 04 11:14:48.578476 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:48.578391 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns"] Aug 04 11:14:48.581284 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:48.581261 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7d02d259-5976b77db-9fqns"] Aug 04 11:14:49.445148 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:49.445112 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="563ab2a8-46cc-4600-bf9d-1446cfd860e1" path="/var/lib/kubelet/pods/563ab2a8-46cc-4600-bf9d-1446cfd860e1/volumes" Aug 04 11:14:49.895339 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:49.895302 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:14:54.190779 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:54.190747 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:54.292945 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:54.292905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:54.466114 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:54.466026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:54.696841 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:54.696807 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:54.801030 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:54.800929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:54.899152 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:54.899118 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:14:55.012436 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:14:55.012388 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:15:00.130977 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:00.130931 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764035-85sd4"] Aug 04 11:15:00.131462 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:00.131351 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563ab2a8-46cc-4600-bf9d-1446cfd860e1" containerName="maas-api" Aug 04 11:15:00.131462 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:00.131365 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="563ab2a8-46cc-4600-bf9d-1446cfd860e1" containerName="maas-api" Aug 04 11:15:00.131462 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:00.131425 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="563ab2a8-46cc-4600-bf9d-1446cfd860e1" containerName="maas-api" Aug 04 11:15:00.134375 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:00.134358 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764035-85sd4" Aug 04 11:15:00.136889 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:00.136865 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tgsgx\"" Aug 04 11:15:00.142265 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:00.142175 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764035-85sd4"] Aug 04 11:15:00.145919 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:00.145893 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764035-85sd4" Aug 04 11:15:00.286673 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:00.286651 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764035-85sd4"] Aug 04 11:15:00.288874 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:15:00.288849 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892fea7e_b7bf_4516_aac2_2b7520c2c927.slice/crio-7dfd7cc8701dcb3f74d93949ecde62e56853963c726e545c34fcec03a2a099e3 WatchSource:0}: Error finding container 7dfd7cc8701dcb3f74d93949ecde62e56853963c726e545c34fcec03a2a099e3: Status 404 returned error can't find the container with id 7dfd7cc8701dcb3f74d93949ecde62e56853963c726e545c34fcec03a2a099e3 Aug 04 11:15:00.633781 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:00.633734 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764035-85sd4" event={"ID":"892fea7e-b7bf-4516-aac2-2b7520c2c927","Type":"ContainerStarted","Data":"7dfd7cc8701dcb3f74d93949ecde62e56853963c726e545c34fcec03a2a099e3"} Aug 04 11:15:00.997600 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:00.997564 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:15:01.166134 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:01.166100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:15:01.346974 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:01.346940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:15:01.446769 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:01.446737 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:15:01.640766 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:01.640677 2576 generic.go:356] "Generic (PLEG): container finished" podID="892fea7e-b7bf-4516-aac2-2b7520c2c927" containerID="5c4ac6af3088ab70c4826d40f2046a1b0fb189afd9e01d27113dc74b71f61dd5" exitCode=0 Aug 04 11:15:01.640917 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:01.640766 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764035-85sd4" event={"ID":"892fea7e-b7bf-4516-aac2-2b7520c2c927","Type":"ContainerDied","Data":"5c4ac6af3088ab70c4826d40f2046a1b0fb189afd9e01d27113dc74b71f61dd5"} Aug 04 11:15:02.784823 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:02.784797 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764035-85sd4" Aug 04 11:15:03.653920 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:03.653893 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764035-85sd4" Aug 04 11:15:03.654110 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:03.653893 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764035-85sd4" event={"ID":"892fea7e-b7bf-4516-aac2-2b7520c2c927","Type":"ContainerDied","Data":"7dfd7cc8701dcb3f74d93949ecde62e56853963c726e545c34fcec03a2a099e3"} Aug 04 11:15:03.654110 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:03.654022 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dfd7cc8701dcb3f74d93949ecde62e56853963c726e545c34fcec03a2a099e3" Aug 04 11:15:07.379475 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:07.379431 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:15:07.481200 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:07.481161 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:15:26.690519 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:15:26.690475 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:16:43.902269 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:16:43.902238 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:16:51.672663 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:16:51.672631 2576 scope.go:117] "RemoveContainer" containerID="df1b2bc25a55d41f44adf52990cbd4f70ff6bca15394dae253dab1925c1ce653" Aug 04 11:16:51.681651 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:16:51.681633 2576 scope.go:117] "RemoveContainer" containerID="337c22e9d726fd16cd74f888c7c54fd441003cc587e3460d596b738412afd3f2" Aug 04 11:17:10.248724 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:10.248691 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:17:10.544094 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:10.544009 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:17:10.652180 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:10.652146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:17:13.026690 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:13.026658 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:17:13.123385 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:13.123357 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:17:14.121309 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.121279 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv"] Aug 04 11:17:14.121764 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.121697 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="892fea7e-b7bf-4516-aac2-2b7520c2c927" containerName="cleanup" Aug 04 11:17:14.121764 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.121709 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="892fea7e-b7bf-4516-aac2-2b7520c2c927" containerName="cleanup" Aug 04 11:17:14.121844 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.121777 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="892fea7e-b7bf-4516-aac2-2b7520c2c927" containerName="cleanup" Aug 04 11:17:14.124855 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.124834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:17:14.127291 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.127268 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 11:17:14.127417 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.127309 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 11:17:14.128110 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.128079 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmlb4\"" Aug 04 11:17:14.128230 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.128215 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c88ef17b-serving-cert\"" Aug 04 11:17:14.135675 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.135654 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv"] Aug 04 11:17:14.155292 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.155255 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:17:14.280705 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.280677 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:17:14.295056 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.295030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9931bc7d-8f6e-4954-8a7a-6347f21d924a-openshift-service-ca\") pod \"maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv\" (UID: \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:17:14.295137 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.295063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jg5b\" (UniqueName: \"kubernetes.io/projected/9931bc7d-8f6e-4954-8a7a-6347f21d924a-kube-api-access-7jg5b\") pod \"maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv\" (UID: \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:17:14.295191 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.295174 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9931bc7d-8f6e-4954-8a7a-6347f21d924a-maas-api-tls\") pod \"maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv\" (UID: \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:17:14.396407 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.396319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9931bc7d-8f6e-4954-8a7a-6347f21d924a-maas-api-tls\") pod \"maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv\" (UID: \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:17:14.396407 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.396392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9931bc7d-8f6e-4954-8a7a-6347f21d924a-openshift-service-ca\") pod \"maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv\" (UID: \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:17:14.396622 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.396416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jg5b\" (UniqueName: \"kubernetes.io/projected/9931bc7d-8f6e-4954-8a7a-6347f21d924a-kube-api-access-7jg5b\") pod \"maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv\" (UID: \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:17:14.397120 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.397094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9931bc7d-8f6e-4954-8a7a-6347f21d924a-openshift-service-ca\") pod \"maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv\" (UID: \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:17:14.398838 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.398813 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9931bc7d-8f6e-4954-8a7a-6347f21d924a-maas-api-tls\") pod \"maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv\" (UID: \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:17:14.413631 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.413601 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jg5b\" (UniqueName: \"kubernetes.io/projected/9931bc7d-8f6e-4954-8a7a-6347f21d924a-kube-api-access-7jg5b\") pod \"maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv\" (UID: \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:17:14.436279 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.436242 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:17:14.451397 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.451289 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:17:14.587398 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.587373 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv"] Aug 04 11:17:14.589103 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:17:14.589062 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9931bc7d_8f6e_4954_8a7a_6347f21d924a.slice/crio-64570348a397f2dc7bf32bfdb8451d0ea01412168e9d3516b353b0f034485061 WatchSource:0}: Error finding container 64570348a397f2dc7bf32bfdb8451d0ea01412168e9d3516b353b0f034485061: Status 404 returned error can't find the container with id 64570348a397f2dc7bf32bfdb8451d0ea01412168e9d3516b353b0f034485061 Aug 04 11:17:14.590453 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.590438 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 11:17:14.696324 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.696244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:17:14.786177 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:14.786146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:17:15.380673 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:15.380629 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" event={"ID":"9931bc7d-8f6e-4954-8a7a-6347f21d924a","Type":"ContainerStarted","Data":"64570348a397f2dc7bf32bfdb8451d0ea01412168e9d3516b353b0f034485061"} Aug 04 11:17:16.388765 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:16.388732 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" event={"ID":"9931bc7d-8f6e-4954-8a7a-6347f21d924a","Type":"ContainerStarted","Data":"15a06798a8d362c20ddd25f7399a6d3577760c7901f78b83395bd29ab4e2010f"} Aug 04 11:17:16.389234 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:16.388814 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:17:21.076336 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:21.076276 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" podStartSLOduration=6.187430881 podStartE2EDuration="7.076258919s" totalImagesPullingTime="888.828038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:17:16.405792312 +0000 UTC m=+2425.503585708" watchObservedRunningTime="2026-08-04 11:17:21.076258919 +0000 UTC m=+2430.174052312" Aug 04 11:17:21.077031 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:21.077014 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:17:22.399942 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:22.399910 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:17:43.124766 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:43.124722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:17:59.288144 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:17:59.288107 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:18:01.693078 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:01.693042 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:18:04.936714 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:04.936679 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:18:05.038364 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:05.038321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:18:13.299607 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:13.299575 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:18:19.379522 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:19.379470 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:18:40.885312 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:40.885274 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:18:43.314605 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.314517 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv"] Aug 04 11:18:43.315035 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.314775 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" podUID="9931bc7d-8f6e-4954-8a7a-6347f21d924a" containerName="maas-api" containerID="cri-o://15a06798a8d362c20ddd25f7399a6d3577760c7901f78b83395bd29ab4e2010f" gracePeriod=30 Aug 04 11:18:43.355825 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.355785 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:18:43.442039 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.442000 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:18:43.561360 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.561307 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:18:43.569623 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.569567 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:18:43.637627 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.637590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:18:43.694779 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.694742 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9931bc7d-8f6e-4954-8a7a-6347f21d924a-maas-api-tls\") pod \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\" (UID: \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\") " Aug 04 11:18:43.694973 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.694928 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9931bc7d-8f6e-4954-8a7a-6347f21d924a-openshift-service-ca\") pod \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\" (UID: \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\") " Aug 04 11:18:43.694973 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.694964 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jg5b\" (UniqueName: \"kubernetes.io/projected/9931bc7d-8f6e-4954-8a7a-6347f21d924a-kube-api-access-7jg5b\") pod \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\" (UID: \"9931bc7d-8f6e-4954-8a7a-6347f21d924a\") " Aug 04 11:18:43.695307 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.695277 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9931bc7d-8f6e-4954-8a7a-6347f21d924a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9931bc7d-8f6e-4954-8a7a-6347f21d924a" (UID: "9931bc7d-8f6e-4954-8a7a-6347f21d924a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:18:43.697026 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.697000 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9931bc7d-8f6e-4954-8a7a-6347f21d924a-kube-api-access-7jg5b" (OuterVolumeSpecName: "kube-api-access-7jg5b") pod "9931bc7d-8f6e-4954-8a7a-6347f21d924a" (UID: "9931bc7d-8f6e-4954-8a7a-6347f21d924a"). InnerVolumeSpecName "kube-api-access-7jg5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:18:43.697026 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.697005 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9931bc7d-8f6e-4954-8a7a-6347f21d924a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9931bc7d-8f6e-4954-8a7a-6347f21d924a" (UID: "9931bc7d-8f6e-4954-8a7a-6347f21d924a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:18:43.796398 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.796341 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9931bc7d-8f6e-4954-8a7a-6347f21d924a-openshift-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:18:43.796398 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.796397 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jg5b\" (UniqueName: \"kubernetes.io/projected/9931bc7d-8f6e-4954-8a7a-6347f21d924a-kube-api-access-7jg5b\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:18:43.796398 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.796416 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9931bc7d-8f6e-4954-8a7a-6347f21d924a-maas-api-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:18:43.887311 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.887272 2576 generic.go:356] "Generic (PLEG): container finished" podID="9931bc7d-8f6e-4954-8a7a-6347f21d924a" containerID="15a06798a8d362c20ddd25f7399a6d3577760c7901f78b83395bd29ab4e2010f" exitCode=0 Aug 04 11:18:43.887496 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.887340 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" Aug 04 11:18:43.887496 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.887356 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" event={"ID":"9931bc7d-8f6e-4954-8a7a-6347f21d924a","Type":"ContainerDied","Data":"15a06798a8d362c20ddd25f7399a6d3577760c7901f78b83395bd29ab4e2010f"} Aug 04 11:18:43.887496 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.887433 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv" event={"ID":"9931bc7d-8f6e-4954-8a7a-6347f21d924a","Type":"ContainerDied","Data":"64570348a397f2dc7bf32bfdb8451d0ea01412168e9d3516b353b0f034485061"} Aug 04 11:18:43.887496 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.887456 2576 scope.go:117] "RemoveContainer" containerID="15a06798a8d362c20ddd25f7399a6d3577760c7901f78b83395bd29ab4e2010f" Aug 04 11:18:43.897700 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.897685 2576 scope.go:117] "RemoveContainer" containerID="15a06798a8d362c20ddd25f7399a6d3577760c7901f78b83395bd29ab4e2010f" Aug 04 11:18:43.897941 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:18:43.897922 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a06798a8d362c20ddd25f7399a6d3577760c7901f78b83395bd29ab4e2010f\": container with ID starting with 15a06798a8d362c20ddd25f7399a6d3577760c7901f78b83395bd29ab4e2010f not found: ID does not exist" containerID="15a06798a8d362c20ddd25f7399a6d3577760c7901f78b83395bd29ab4e2010f" Aug 04 11:18:43.897981 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.897949 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a06798a8d362c20ddd25f7399a6d3577760c7901f78b83395bd29ab4e2010f"} err="failed to get container status \"15a06798a8d362c20ddd25f7399a6d3577760c7901f78b83395bd29ab4e2010f\": rpc error: code = NotFound desc = could not find container \"15a06798a8d362c20ddd25f7399a6d3577760c7901f78b83395bd29ab4e2010f\": container with ID starting with 15a06798a8d362c20ddd25f7399a6d3577760c7901f78b83395bd29ab4e2010f not found: ID does not exist" Aug 04 11:18:43.910492 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.910468 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv"] Aug 04 11:18:43.915206 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:43.915179 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c88ef17b-558bf4f455-gfkcv"] Aug 04 11:18:45.438449 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:45.438416 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9931bc7d-8f6e-4954-8a7a-6347f21d924a" path="/var/lib/kubelet/pods/9931bc7d-8f6e-4954-8a7a-6347f21d924a/volumes" Aug 04 11:18:46.488458 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:46.488422 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:18:46.647705 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:46.647673 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:18:51.695512 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:51.695474 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:18:55.885238 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:55.885208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:18:55.954175 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:18:55.954146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:22.228023 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:22.227972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:22.422244 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:22.422208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:28.160909 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:28.160873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:34.000943 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:34.000909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:34.101611 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:34.101582 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:34.280912 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:34.280823 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:34.512202 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:34.512173 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:36.400116 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:36.400065 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:19:45.809289 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:45.809254 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:45.900848 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:45.900815 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:45.997168 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:45.997135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:46.620644 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:46.620609 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:46.731222 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:46.731186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:46.828902 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:46.828869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:47.591193 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:47.591156 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:19:48.128565 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:48.128525 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:54.129736 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:54.129695 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:19:55.028831 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:55.028718 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:55.129024 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:55.128992 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:19:55.689021 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:19:55.688968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:01.985828 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:01.985772 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:02.057555 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:02.057495 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:02.182806 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:02.182764 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:02.277551 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:02.277420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:02.492311 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:02.492274 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:20:12.761695 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:12.761660 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:12.988654 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:12.988619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:33.930234 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:33.930194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:41.965171 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:41.965136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:42.090158 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:42.090127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:42.308886 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:42.308799 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:42.696437 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:42.696398 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:43.096691 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:43.096655 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:46.122444 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:46.122405 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:46.250441 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:46.250406 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:47.415129 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.415090 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n"] Aug 04 11:20:47.415653 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.415555 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9931bc7d-8f6e-4954-8a7a-6347f21d924a" containerName="maas-api" Aug 04 11:20:47.415653 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.415568 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9931bc7d-8f6e-4954-8a7a-6347f21d924a" containerName="maas-api" Aug 04 11:20:47.415653 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.415643 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9931bc7d-8f6e-4954-8a7a-6347f21d924a" containerName="maas-api" Aug 04 11:20:47.419072 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.419052 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:20:47.421639 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.421614 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-dc2eb2-serving-cert\"" Aug 04 11:20:47.421768 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.421624 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 11:20:47.422097 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.422079 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 11:20:47.422486 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.422471 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmlb4\"" Aug 04 11:20:47.439708 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.439677 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n"] Aug 04 11:20:47.468398 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.468353 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:47.600258 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.600216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qrsg\" (UniqueName: \"kubernetes.io/projected/959f56f6-c264-44d2-96cb-dd5ee9dfe613-kube-api-access-4qrsg\") pod \"maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n\" (UID: \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:20:47.600469 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.600302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/959f56f6-c264-44d2-96cb-dd5ee9dfe613-maas-api-tls\") pod \"maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n\" (UID: \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:20:47.600469 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.600333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/959f56f6-c264-44d2-96cb-dd5ee9dfe613-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n\" (UID: \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:20:47.701814 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.701708 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/959f56f6-c264-44d2-96cb-dd5ee9dfe613-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n\" (UID: \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:20:47.701814 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.701804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qrsg\" (UniqueName: \"kubernetes.io/projected/959f56f6-c264-44d2-96cb-dd5ee9dfe613-kube-api-access-4qrsg\") pod \"maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n\" (UID: \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:20:47.702063 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.701870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/959f56f6-c264-44d2-96cb-dd5ee9dfe613-maas-api-tls\") pod \"maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n\" (UID: \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:20:47.702063 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:20:47.701960 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-serving-cert: secret "maas-api-e2e-inf-a-dc2eb2-serving-cert" not found Aug 04 11:20:47.702063 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:20:47.702025 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/959f56f6-c264-44d2-96cb-dd5ee9dfe613-maas-api-tls podName:959f56f6-c264-44d2-96cb-dd5ee9dfe613 nodeName:}" failed. No retries permitted until 2026-08-04 11:20:48.202008393 +0000 UTC m=+2637.299801765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/959f56f6-c264-44d2-96cb-dd5ee9dfe613-maas-api-tls") pod "maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" (UID: "959f56f6-c264-44d2-96cb-dd5ee9dfe613") : secret "maas-api-e2e-inf-a-dc2eb2-serving-cert" not found Aug 04 11:20:47.702488 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.702467 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/959f56f6-c264-44d2-96cb-dd5ee9dfe613-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n\" (UID: \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:20:47.713295 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:47.713269 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qrsg\" (UniqueName: \"kubernetes.io/projected/959f56f6-c264-44d2-96cb-dd5ee9dfe613-kube-api-access-4qrsg\") pod \"maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n\" (UID: \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:20:48.008197 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:48.008111 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:48.107031 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:48.107001 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:48.206030 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:48.205998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/959f56f6-c264-44d2-96cb-dd5ee9dfe613-maas-api-tls\") pod \"maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n\" (UID: \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:20:48.208635 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:48.208606 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/959f56f6-c264-44d2-96cb-dd5ee9dfe613-maas-api-tls\") pod \"maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n\" (UID: \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:20:48.308407 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:48.308325 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:48.331148 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:48.331108 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:20:48.479344 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:48.479314 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n"] Aug 04 11:20:48.481096 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:20:48.481068 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959f56f6_c264_44d2_96cb_dd5ee9dfe613.slice/crio-3cfa243f00d3b9dfdae2f99a895e9ab656b1c32b27c6abab1de9b8e23cab5cd9 WatchSource:0}: Error finding container 3cfa243f00d3b9dfdae2f99a895e9ab656b1c32b27c6abab1de9b8e23cab5cd9: Status 404 returned error can't find the container with id 3cfa243f00d3b9dfdae2f99a895e9ab656b1c32b27c6abab1de9b8e23cab5cd9 Aug 04 11:20:48.593195 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:48.593158 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" event={"ID":"959f56f6-c264-44d2-96cb-dd5ee9dfe613","Type":"ContainerStarted","Data":"3cfa243f00d3b9dfdae2f99a895e9ab656b1c32b27c6abab1de9b8e23cab5cd9"} Aug 04 11:20:49.600344 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:49.600305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" event={"ID":"959f56f6-c264-44d2-96cb-dd5ee9dfe613","Type":"ContainerStarted","Data":"e6c3d6cb9279df9bf594ca25342be01850e1a0f8319f2d1ca15e6bc2eb8bc207"} Aug 04 11:20:49.600720 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:49.600370 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:20:49.623234 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:49.623179 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" podStartSLOduration=2.021518083 podStartE2EDuration="2.623162049s" totalImagesPullingTime="601.643966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:20:49.621569654 +0000 UTC m=+2638.719363050" watchObservedRunningTime="2026-08-04 11:20:49.623162049 +0000 UTC m=+2638.720955444" Aug 04 11:20:49.999722 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:49.999638 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:20:54.654850 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:54.654820 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:54.733264 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:54.733220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:20:55.612541 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:20:55.612498 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:21:11.048162 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:11.048122 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:21:11.369874 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:11.369837 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:21:11.741137 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:11.741052 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:21:12.210120 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:12.210087 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:21:15.135313 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:15.135282 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:21:15.230807 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:15.230773 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:21:16.370405 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.370371 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s"] Aug 04 11:21:16.384773 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.384737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:21:16.384973 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.384819 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s"] Aug 04 11:21:16.387186 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.387161 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1dd437-serving-cert\"" Aug 04 11:21:16.406113 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.406083 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:21:16.465892 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.465849 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tcb6\" (UniqueName: \"kubernetes.io/projected/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-kube-api-access-4tcb6\") pod \"maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s\" (UID: \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:21:16.466114 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.466001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-maas-api-tls\") pod \"maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s\" (UID: \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:21:16.466114 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.466058 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s\" (UID: \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:21:16.567054 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.567025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-maas-api-tls\") pod \"maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s\" (UID: \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:21:16.567249 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.567068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s\" (UID: \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:21:16.567249 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.567117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tcb6\" (UniqueName: \"kubernetes.io/projected/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-kube-api-access-4tcb6\") pod \"maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s\" (UID: \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:21:16.567893 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.567864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s\" (UID: \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:21:16.569706 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.569683 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-maas-api-tls\") pod \"maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s\" (UID: \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:21:16.575607 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.575582 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tcb6\" (UniqueName: \"kubernetes.io/projected/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-kube-api-access-4tcb6\") pod \"maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s\" (UID: \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:21:16.620290 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.620251 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:21:16.696694 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.696592 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:21:16.842542 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.842468 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s"] Aug 04 11:21:16.844935 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:21:16.844906 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a0cc8a_10fc_40f7_942c_1308ad29eca9.slice/crio-9f4e3eebeb5796f33f159cc97444990cb9f1a856f8f47ee0d86166a883593fca WatchSource:0}: Error finding container 9f4e3eebeb5796f33f159cc97444990cb9f1a856f8f47ee0d86166a883593fca: Status 404 returned error can't find the container with id 9f4e3eebeb5796f33f159cc97444990cb9f1a856f8f47ee0d86166a883593fca Aug 04 11:21:16.968493 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:16.968388 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:21:17.067128 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:17.067092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:21:17.770240 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:17.770209 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" event={"ID":"e2a0cc8a-10fc-40f7-942c-1308ad29eca9","Type":"ContainerStarted","Data":"e496049ff18cde5c7e57b89244f4752bedababa57c459977933aecd533f86c62"} Aug 04 11:21:17.770699 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:17.770258 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" event={"ID":"e2a0cc8a-10fc-40f7-942c-1308ad29eca9","Type":"ContainerStarted","Data":"9f4e3eebeb5796f33f159cc97444990cb9f1a856f8f47ee0d86166a883593fca"} Aug 04 11:21:17.770699 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:17.770346 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:21:17.789942 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:17.789890 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" podStartSLOduration=1.4602056700000001 podStartE2EDuration="1.789871393s" totalImagesPullingTime="329.665723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:21:17.787611663 +0000 UTC m=+2666.885405059" watchObservedRunningTime="2026-08-04 11:21:17.789871393 +0000 UTC m=+2666.887664787" Aug 04 11:21:19.093768 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:19.093731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:21:23.781960 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:23.781927 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:21:40.094999 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:40.094943 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:21:48.330674 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:48.330591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:21:50.782416 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:50.782385 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:21:57.678394 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:21:57.678348 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:22:12.785028 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.782068 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz"] Aug 04 11:22:12.787351 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.787323 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.790745 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.790724 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1dd437\"/\"openshift-service-ca.crt\"" Aug 04 11:22:12.791305 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.791285 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1dd437\"/\"default-dockercfg-mzx4f\"" Aug 04 11:22:12.791393 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.791323 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1dd437\"/\"kube-root-ca.crt\"" Aug 04 11:22:12.791393 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.791321 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1dd437\"/\"test-model-1dd437-kserve-self-signed-certs\"" Aug 04 11:22:12.797007 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.796983 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz"] Aug 04 11:22:12.823010 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.822977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:22:12.887413 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.887371 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-tmp-dir\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.887413 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.887413 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb295dd0-b531-431e-adaa-333f62d54f08-tls-certs\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.887703 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.887436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2phk\" (UniqueName: \"kubernetes.io/projected/fb295dd0-b531-431e-adaa-333f62d54f08-kube-api-access-v2phk\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.887703 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.887523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-home\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.887703 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.887571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-model-cache\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.887703 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.887601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-dshm\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.988594 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.988556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-tmp-dir\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.988594 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.988591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb295dd0-b531-431e-adaa-333f62d54f08-tls-certs\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.988843 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.988608 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2phk\" (UniqueName: \"kubernetes.io/projected/fb295dd0-b531-431e-adaa-333f62d54f08-kube-api-access-v2phk\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.988843 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.988720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-home\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.988843 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.988771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-model-cache\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.988843 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.988795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-dshm\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.989071 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.988993 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-tmp-dir\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.989127 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.989074 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-home\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.989172 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.989142 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-model-cache\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.991015 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.990996 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-dshm\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.991317 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.991294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb295dd0-b531-431e-adaa-333f62d54f08-tls-certs\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:12.998839 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:12.998809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2phk\" (UniqueName: \"kubernetes.io/projected/fb295dd0-b531-431e-adaa-333f62d54f08-kube-api-access-v2phk\") pod \"test-model-1dd437-kserve-64f579d46c-92vwz\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:13.098568 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:13.098536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:13.238120 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:13.238093 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz"] Aug 04 11:22:13.241034 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:22:13.241003 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb295dd0_b531_431e_adaa_333f62d54f08.slice/crio-b08a3a78457bca1855686d1e0193a1e4aa325668f3874708657f86cd90f74921 WatchSource:0}: Error finding container b08a3a78457bca1855686d1e0193a1e4aa325668f3874708657f86cd90f74921: Status 404 returned error can't find the container with id b08a3a78457bca1855686d1e0193a1e4aa325668f3874708657f86cd90f74921 Aug 04 11:22:14.100346 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:14.100308 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" event={"ID":"fb295dd0-b531-431e-adaa-333f62d54f08","Type":"ContainerStarted","Data":"b08a3a78457bca1855686d1e0193a1e4aa325668f3874708657f86cd90f74921"} Aug 04 11:22:15.108125 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:15.108087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" event={"ID":"fb295dd0-b531-431e-adaa-333f62d54f08","Type":"ContainerStarted","Data":"30beffe799078e6d5b3a974beb178b247fe9fcb612399446def8e99f49b9121b"} Aug 04 11:22:15.128236 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:15.128182 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" podStartSLOduration=2.014872005 podStartE2EDuration="3.128164311s" totalImagesPullingTime="1.113292306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:22:15.126751237 +0000 UTC m=+2724.224544633" watchObservedRunningTime="2026-08-04 11:22:15.128164311 +0000 UTC m=+2724.225957714" Aug 04 11:22:15.587615 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:15.587578 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:22:23.099267 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:23.099224 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:23.099805 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:23.099540 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:23.112564 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:23.112535 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:23.166608 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:23.166580 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:22:26.886410 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:26.886371 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:22:59.669576 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:22:59.669544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:00.510731 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:00.510692 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz"] Aug 04 11:23:00.529121 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:00.529084 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:00.650274 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:00.650237 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt"] Aug 04 11:23:00.653834 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:00.653817 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt" Aug 04 11:23:00.656132 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:00.656105 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tgsgx\"" Aug 04 11:23:00.661910 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:00.661422 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt"] Aug 04 11:23:00.737297 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:00.737259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt\" (UID: \"9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt" Aug 04 11:23:00.838614 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:00.838579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt\" (UID: \"9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt" Aug 04 11:23:00.839247 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:00.839226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt\" (UID: \"9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt" Aug 04 11:23:00.966856 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:00.966799 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt" Aug 04 11:23:01.112749 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:01.112720 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt"] Aug 04 11:23:01.115062 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:23:01.115038 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca70e6c_ae2b_4e91_b3bd_dd0b2eb62156.slice/crio-fc0445cadab65af55f3a369c86d2892349f33b7debcea244d987cd846e2a3da6 WatchSource:0}: Error finding container fc0445cadab65af55f3a369c86d2892349f33b7debcea244d987cd846e2a3da6: Status 404 returned error can't find the container with id fc0445cadab65af55f3a369c86d2892349f33b7debcea244d987cd846e2a3da6 Aug 04 11:23:01.371046 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:01.370965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt" event={"ID":"9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156","Type":"ContainerStarted","Data":"fc0445cadab65af55f3a369c86d2892349f33b7debcea244d987cd846e2a3da6"} Aug 04 11:23:02.382216 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:02.382185 2576 generic.go:356] "Generic (PLEG): container finished" podID="9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156" containerID="5f665e38a31d781f2ace9cfde8b2199b67cf729204d1961574332db5e1569418" exitCode=0 Aug 04 11:23:02.382641 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:02.382283 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt" event={"ID":"9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156","Type":"ContainerDied","Data":"5f665e38a31d781f2ace9cfde8b2199b67cf729204d1961574332db5e1569418"} Aug 04 11:23:02.985316 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:02.985278 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:23:03.528887 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:03.528859 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt" Aug 04 11:23:03.665818 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:03.665737 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156-maas-api-service-ca\") pod \"9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156\" (UID: \"9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156\") " Aug 04 11:23:03.666068 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:03.666047 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156" (UID: "9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:23:03.766764 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:03.766729 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156-maas-api-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:23:04.395139 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:04.395109 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt" Aug 04 11:23:04.395317 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:04.395146 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt" event={"ID":"9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156","Type":"ContainerDied","Data":"fc0445cadab65af55f3a369c86d2892349f33b7debcea244d987cd846e2a3da6"} Aug 04 11:23:04.395317 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:04.395195 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc0445cadab65af55f3a369c86d2892349f33b7debcea244d987cd846e2a3da6" Aug 04 11:23:11.118559 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.118526 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n"] Aug 04 11:23:11.119041 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.118733 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" podUID="959f56f6-c264-44d2-96cb-dd5ee9dfe613" containerName="maas-api" containerID="cri-o://e6c3d6cb9279df9bf594ca25342be01850e1a0f8319f2d1ca15e6bc2eb8bc207" gracePeriod=30 Aug 04 11:23:11.145063 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.145030 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:11.218829 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.218792 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:11.281062 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.281030 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:11.326020 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.325982 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt"] Aug 04 11:23:11.329088 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.329062 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-dc2eb2-tcmdt"] Aug 04 11:23:11.347034 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.347009 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:11.438875 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.438841 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156" path="/var/lib/kubelet/pods/9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156/volumes" Aug 04 11:23:11.439658 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.439625 2576 generic.go:356] "Generic (PLEG): container finished" podID="959f56f6-c264-44d2-96cb-dd5ee9dfe613" containerID="e6c3d6cb9279df9bf594ca25342be01850e1a0f8319f2d1ca15e6bc2eb8bc207" exitCode=0 Aug 04 11:23:11.439767 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.439696 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" event={"ID":"959f56f6-c264-44d2-96cb-dd5ee9dfe613","Type":"ContainerDied","Data":"e6c3d6cb9279df9bf594ca25342be01850e1a0f8319f2d1ca15e6bc2eb8bc207"} Aug 04 11:23:11.583004 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.582977 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:23:11.630893 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.630869 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/959f56f6-c264-44d2-96cb-dd5ee9dfe613-maas-api-tls\") pod \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\" (UID: \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\") " Aug 04 11:23:11.631039 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.630906 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/959f56f6-c264-44d2-96cb-dd5ee9dfe613-openshift-service-ca\") pod \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\" (UID: \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\") " Aug 04 11:23:11.631039 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.630982 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qrsg\" (UniqueName: \"kubernetes.io/projected/959f56f6-c264-44d2-96cb-dd5ee9dfe613-kube-api-access-4qrsg\") pod \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\" (UID: \"959f56f6-c264-44d2-96cb-dd5ee9dfe613\") " Aug 04 11:23:11.631330 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.631305 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959f56f6-c264-44d2-96cb-dd5ee9dfe613-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "959f56f6-c264-44d2-96cb-dd5ee9dfe613" (UID: "959f56f6-c264-44d2-96cb-dd5ee9dfe613"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:23:11.632991 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.632969 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959f56f6-c264-44d2-96cb-dd5ee9dfe613-kube-api-access-4qrsg" (OuterVolumeSpecName: "kube-api-access-4qrsg") pod "959f56f6-c264-44d2-96cb-dd5ee9dfe613" (UID: "959f56f6-c264-44d2-96cb-dd5ee9dfe613"). InnerVolumeSpecName "kube-api-access-4qrsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:23:11.632991 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.632976 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/959f56f6-c264-44d2-96cb-dd5ee9dfe613-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "959f56f6-c264-44d2-96cb-dd5ee9dfe613" (UID: "959f56f6-c264-44d2-96cb-dd5ee9dfe613"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:23:11.688987 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.688893 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:11.731783 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.731750 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qrsg\" (UniqueName: \"kubernetes.io/projected/959f56f6-c264-44d2-96cb-dd5ee9dfe613-kube-api-access-4qrsg\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:23:11.731783 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.731780 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/959f56f6-c264-44d2-96cb-dd5ee9dfe613-maas-api-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:23:11.731783 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:11.731791 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/959f56f6-c264-44d2-96cb-dd5ee9dfe613-openshift-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:23:12.446357 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:12.446321 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" event={"ID":"959f56f6-c264-44d2-96cb-dd5ee9dfe613","Type":"ContainerDied","Data":"3cfa243f00d3b9dfdae2f99a895e9ab656b1c32b27c6abab1de9b8e23cab5cd9"} Aug 04 11:23:12.446357 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:12.446343 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n" Aug 04 11:23:12.446858 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:12.446386 2576 scope.go:117] "RemoveContainer" containerID="e6c3d6cb9279df9bf594ca25342be01850e1a0f8319f2d1ca15e6bc2eb8bc207" Aug 04 11:23:12.469192 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:12.469167 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n"] Aug 04 11:23:12.473206 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:12.473183 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dc2eb2-78f798657c-4zf5n"] Aug 04 11:23:13.441859 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:13.439686 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="959f56f6-c264-44d2-96cb-dd5ee9dfe613" path="/var/lib/kubelet/pods/959f56f6-c264-44d2-96cb-dd5ee9dfe613/volumes" Aug 04 11:23:13.475686 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:13.475653 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:23:15.531725 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.531666 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" podUID="fb295dd0-b531-431e-adaa-333f62d54f08" containerName="main" containerID="cri-o://30beffe799078e6d5b3a974beb178b247fe9fcb612399446def8e99f49b9121b" gracePeriod=45 Aug 04 11:23:15.782387 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.782308 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:23:15.868131 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.868093 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb295dd0-b531-431e-adaa-333f62d54f08-tls-certs\") pod \"fb295dd0-b531-431e-adaa-333f62d54f08\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " Aug 04 11:23:15.868131 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.868140 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-model-cache\") pod \"fb295dd0-b531-431e-adaa-333f62d54f08\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " Aug 04 11:23:15.868372 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.868166 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-dshm\") pod \"fb295dd0-b531-431e-adaa-333f62d54f08\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " Aug 04 11:23:15.868372 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.868192 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-tmp-dir\") pod \"fb295dd0-b531-431e-adaa-333f62d54f08\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " Aug 04 11:23:15.868372 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.868254 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2phk\" (UniqueName: \"kubernetes.io/projected/fb295dd0-b531-431e-adaa-333f62d54f08-kube-api-access-v2phk\") pod \"fb295dd0-b531-431e-adaa-333f62d54f08\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " Aug 04 11:23:15.868372 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.868297 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-home\") pod \"fb295dd0-b531-431e-adaa-333f62d54f08\" (UID: \"fb295dd0-b531-431e-adaa-333f62d54f08\") " Aug 04 11:23:15.868630 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.868459 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb295dd0-b531-431e-adaa-333f62d54f08" (UID: "fb295dd0-b531-431e-adaa-333f62d54f08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:23:15.868630 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.868498 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb295dd0-b531-431e-adaa-333f62d54f08" (UID: "fb295dd0-b531-431e-adaa-333f62d54f08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:23:15.868735 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.868633 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-home" (OuterVolumeSpecName: "home") pod "fb295dd0-b531-431e-adaa-333f62d54f08" (UID: "fb295dd0-b531-431e-adaa-333f62d54f08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:23:15.868735 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.868690 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-model-cache\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:23:15.868735 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.868711 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-tmp-dir\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:23:15.870542 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.870514 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-dshm" (OuterVolumeSpecName: "dshm") pod "fb295dd0-b531-431e-adaa-333f62d54f08" (UID: "fb295dd0-b531-431e-adaa-333f62d54f08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:23:15.870651 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.870540 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb295dd0-b531-431e-adaa-333f62d54f08-kube-api-access-v2phk" (OuterVolumeSpecName: "kube-api-access-v2phk") pod "fb295dd0-b531-431e-adaa-333f62d54f08" (UID: "fb295dd0-b531-431e-adaa-333f62d54f08"). InnerVolumeSpecName "kube-api-access-v2phk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:23:15.870651 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.870591 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb295dd0-b531-431e-adaa-333f62d54f08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb295dd0-b531-431e-adaa-333f62d54f08" (UID: "fb295dd0-b531-431e-adaa-333f62d54f08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:23:15.969848 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.969801 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-dshm\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:23:15.969848 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.969844 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2phk\" (UniqueName: \"kubernetes.io/projected/fb295dd0-b531-431e-adaa-333f62d54f08-kube-api-access-v2phk\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:23:15.969848 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.969855 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb295dd0-b531-431e-adaa-333f62d54f08-home\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:23:15.969848 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:15.969864 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb295dd0-b531-431e-adaa-333f62d54f08-tls-certs\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:23:16.469935 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:16.469899 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb295dd0-b531-431e-adaa-333f62d54f08" containerID="30beffe799078e6d5b3a974beb178b247fe9fcb612399446def8e99f49b9121b" exitCode=0 Aug 04 11:23:16.470128 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:16.469941 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" Aug 04 11:23:16.470128 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:16.469981 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" event={"ID":"fb295dd0-b531-431e-adaa-333f62d54f08","Type":"ContainerDied","Data":"30beffe799078e6d5b3a974beb178b247fe9fcb612399446def8e99f49b9121b"} Aug 04 11:23:16.470128 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:16.470022 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz" event={"ID":"fb295dd0-b531-431e-adaa-333f62d54f08","Type":"ContainerDied","Data":"b08a3a78457bca1855686d1e0193a1e4aa325668f3874708657f86cd90f74921"} Aug 04 11:23:16.470128 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:16.470041 2576 scope.go:117] "RemoveContainer" containerID="30beffe799078e6d5b3a974beb178b247fe9fcb612399446def8e99f49b9121b" Aug 04 11:23:16.480134 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:16.480118 2576 scope.go:117] "RemoveContainer" containerID="30beffe799078e6d5b3a974beb178b247fe9fcb612399446def8e99f49b9121b" Aug 04 11:23:16.480370 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:23:16.480349 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30beffe799078e6d5b3a974beb178b247fe9fcb612399446def8e99f49b9121b\": container with ID starting with 30beffe799078e6d5b3a974beb178b247fe9fcb612399446def8e99f49b9121b not found: ID does not exist" containerID="30beffe799078e6d5b3a974beb178b247fe9fcb612399446def8e99f49b9121b" Aug 04 11:23:16.480437 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:16.480383 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30beffe799078e6d5b3a974beb178b247fe9fcb612399446def8e99f49b9121b"} err="failed to get container status \"30beffe799078e6d5b3a974beb178b247fe9fcb612399446def8e99f49b9121b\": rpc error: code = NotFound desc = could not find container \"30beffe799078e6d5b3a974beb178b247fe9fcb612399446def8e99f49b9121b\": container with ID starting with 30beffe799078e6d5b3a974beb178b247fe9fcb612399446def8e99f49b9121b not found: ID does not exist" Aug 04 11:23:16.493125 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:16.493098 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz"] Aug 04 11:23:16.497304 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:16.497278 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1dd437/test-model-1dd437-kserve-64f579d46c-92vwz"] Aug 04 11:23:17.438205 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:17.438172 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb295dd0-b531-431e-adaa-333f62d54f08" path="/var/lib/kubelet/pods/fb295dd0-b531-431e-adaa-333f62d54f08/volumes" Aug 04 11:23:17.520356 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:17.520322 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:17.620992 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:17.620960 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:17.795595 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:17.795494 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:17.990847 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:17.990819 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:18.111204 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.111165 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq"] Aug 04 11:23:18.111612 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.111598 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb295dd0-b531-431e-adaa-333f62d54f08" containerName="main" Aug 04 11:23:18.111612 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.111613 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb295dd0-b531-431e-adaa-333f62d54f08" containerName="main" Aug 04 11:23:18.111720 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.111624 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="959f56f6-c264-44d2-96cb-dd5ee9dfe613" containerName="maas-api" Aug 04 11:23:18.111720 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.111629 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="959f56f6-c264-44d2-96cb-dd5ee9dfe613" containerName="maas-api" Aug 04 11:23:18.111720 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.111648 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156" containerName="revoke-keys" Aug 04 11:23:18.111720 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.111654 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156" containerName="revoke-keys" Aug 04 11:23:18.111720 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.111710 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb295dd0-b531-431e-adaa-333f62d54f08" containerName="main" Aug 04 11:23:18.111720 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.111718 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="959f56f6-c264-44d2-96cb-dd5ee9dfe613" containerName="maas-api" Aug 04 11:23:18.111962 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.111725 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ca70e6c-ae2b-4e91-b3bd-dd0b2eb62156" containerName="revoke-keys" Aug 04 11:23:18.116062 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.116044 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq" Aug 04 11:23:18.118732 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.118703 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tgsgx\"" Aug 04 11:23:18.122646 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.122617 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq"] Aug 04 11:23:18.189237 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.189201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d417b675-9133-40d8-812f-126c22c6bdcd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq\" (UID: \"d417b675-9133-40d8-812f-126c22c6bdcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq" Aug 04 11:23:18.290487 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.290440 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d417b675-9133-40d8-812f-126c22c6bdcd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq\" (UID: \"d417b675-9133-40d8-812f-126c22c6bdcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq" Aug 04 11:23:18.291119 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.291098 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d417b675-9133-40d8-812f-126c22c6bdcd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq\" (UID: \"d417b675-9133-40d8-812f-126c22c6bdcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq" Aug 04 11:23:18.429811 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.429723 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq" Aug 04 11:23:18.573569 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:18.573519 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq"] Aug 04 11:23:18.575009 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:23:18.574983 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd417b675_9133_40d8_812f_126c22c6bdcd.slice/crio-39848b5c7046fbec796c6d8657b2d52982802ede7adb44c49ae4b082ec4db39b WatchSource:0}: Error finding container 39848b5c7046fbec796c6d8657b2d52982802ede7adb44c49ae4b082ec4db39b: Status 404 returned error can't find the container with id 39848b5c7046fbec796c6d8657b2d52982802ede7adb44c49ae4b082ec4db39b Aug 04 11:23:19.495731 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:19.495634 2576 generic.go:356] "Generic (PLEG): container finished" podID="d417b675-9133-40d8-812f-126c22c6bdcd" containerID="7dad8fafb51e3d1fafd9237846194e072391172efb11c9c825c03e642eb5d3a6" exitCode=0 Aug 04 11:23:19.495731 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:19.495697 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq" event={"ID":"d417b675-9133-40d8-812f-126c22c6bdcd","Type":"ContainerDied","Data":"7dad8fafb51e3d1fafd9237846194e072391172efb11c9c825c03e642eb5d3a6"} Aug 04 11:23:19.495914 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:19.495772 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq" event={"ID":"d417b675-9133-40d8-812f-126c22c6bdcd","Type":"ContainerStarted","Data":"39848b5c7046fbec796c6d8657b2d52982802ede7adb44c49ae4b082ec4db39b"} Aug 04 11:23:20.637149 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:20.637125 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq" Aug 04 11:23:20.714845 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:20.714810 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d417b675-9133-40d8-812f-126c22c6bdcd-maas-api-service-ca\") pod \"d417b675-9133-40d8-812f-126c22c6bdcd\" (UID: \"d417b675-9133-40d8-812f-126c22c6bdcd\") " Aug 04 11:23:20.715177 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:20.715154 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d417b675-9133-40d8-812f-126c22c6bdcd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d417b675-9133-40d8-812f-126c22c6bdcd" (UID: "d417b675-9133-40d8-812f-126c22c6bdcd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:23:20.815774 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:20.815683 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d417b675-9133-40d8-812f-126c22c6bdcd-maas-api-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:23:21.508066 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:21.508041 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq" Aug 04 11:23:21.508066 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:21.508052 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq" event={"ID":"d417b675-9133-40d8-812f-126c22c6bdcd","Type":"ContainerDied","Data":"39848b5c7046fbec796c6d8657b2d52982802ede7adb44c49ae4b082ec4db39b"} Aug 04 11:23:21.508277 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:21.508099 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39848b5c7046fbec796c6d8657b2d52982802ede7adb44c49ae4b082ec4db39b" Aug 04 11:23:28.513779 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.513742 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s"] Aug 04 11:23:28.514264 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.514041 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" podUID="e2a0cc8a-10fc-40f7-942c-1308ad29eca9" containerName="maas-api" containerID="cri-o://e496049ff18cde5c7e57b89244f4752bedababa57c459977933aecd533f86c62" gracePeriod=30 Aug 04 11:23:28.543417 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.543389 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:28.609718 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.609685 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:28.672750 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.672722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:28.700788 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.700752 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq"] Aug 04 11:23:28.703402 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.703374 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1dd437-xm6fq"] Aug 04 11:23:28.732695 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.732650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:28.765947 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.765879 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:23:28.885652 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.885615 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tcb6\" (UniqueName: \"kubernetes.io/projected/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-kube-api-access-4tcb6\") pod \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\" (UID: \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\") " Aug 04 11:23:28.885828 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.885691 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-maas-api-tls\") pod \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\" (UID: \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\") " Aug 04 11:23:28.885878 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.885859 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-openshift-service-ca\") pod \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\" (UID: \"e2a0cc8a-10fc-40f7-942c-1308ad29eca9\") " Aug 04 11:23:28.886230 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.886205 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2a0cc8a-10fc-40f7-942c-1308ad29eca9" (UID: "e2a0cc8a-10fc-40f7-942c-1308ad29eca9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:23:28.887839 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.887818 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-kube-api-access-4tcb6" (OuterVolumeSpecName: "kube-api-access-4tcb6") pod "e2a0cc8a-10fc-40f7-942c-1308ad29eca9" (UID: "e2a0cc8a-10fc-40f7-942c-1308ad29eca9"). InnerVolumeSpecName "kube-api-access-4tcb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:23:28.887923 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.887829 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2a0cc8a-10fc-40f7-942c-1308ad29eca9" (UID: "e2a0cc8a-10fc-40f7-942c-1308ad29eca9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:23:28.987330 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.987292 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-openshift-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:23:28.987330 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.987321 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tcb6\" (UniqueName: \"kubernetes.io/projected/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-kube-api-access-4tcb6\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:23:28.987330 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:28.987331 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a0cc8a-10fc-40f7-942c-1308ad29eca9-maas-api-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:23:29.070151 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:29.070026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:29.438864 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:29.438830 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d417b675-9133-40d8-812f-126c22c6bdcd" path="/var/lib/kubelet/pods/d417b675-9133-40d8-812f-126c22c6bdcd/volumes" Aug 04 11:23:29.553476 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:29.553444 2576 generic.go:356] "Generic (PLEG): container finished" podID="e2a0cc8a-10fc-40f7-942c-1308ad29eca9" containerID="e496049ff18cde5c7e57b89244f4752bedababa57c459977933aecd533f86c62" exitCode=0 Aug 04 11:23:29.553950 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:29.553487 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" Aug 04 11:23:29.553950 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:29.553549 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" event={"ID":"e2a0cc8a-10fc-40f7-942c-1308ad29eca9","Type":"ContainerDied","Data":"e496049ff18cde5c7e57b89244f4752bedababa57c459977933aecd533f86c62"} Aug 04 11:23:29.553950 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:29.553603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s" event={"ID":"e2a0cc8a-10fc-40f7-942c-1308ad29eca9","Type":"ContainerDied","Data":"9f4e3eebeb5796f33f159cc97444990cb9f1a856f8f47ee0d86166a883593fca"} Aug 04 11:23:29.553950 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:29.553628 2576 scope.go:117] "RemoveContainer" containerID="e496049ff18cde5c7e57b89244f4752bedababa57c459977933aecd533f86c62" Aug 04 11:23:29.562974 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:29.562957 2576 scope.go:117] "RemoveContainer" containerID="e496049ff18cde5c7e57b89244f4752bedababa57c459977933aecd533f86c62" Aug 04 11:23:29.563247 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:23:29.563229 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e496049ff18cde5c7e57b89244f4752bedababa57c459977933aecd533f86c62\": container with ID starting with e496049ff18cde5c7e57b89244f4752bedababa57c459977933aecd533f86c62 not found: ID does not exist" containerID="e496049ff18cde5c7e57b89244f4752bedababa57c459977933aecd533f86c62" Aug 04 11:23:29.563291 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:29.563256 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e496049ff18cde5c7e57b89244f4752bedababa57c459977933aecd533f86c62"} err="failed to get container status \"e496049ff18cde5c7e57b89244f4752bedababa57c459977933aecd533f86c62\": rpc error: code = NotFound desc = could not find container \"e496049ff18cde5c7e57b89244f4752bedababa57c459977933aecd533f86c62\": container with ID starting with e496049ff18cde5c7e57b89244f4752bedababa57c459977933aecd533f86c62 not found: ID does not exist" Aug 04 11:23:29.570737 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:29.570711 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s"] Aug 04 11:23:29.576999 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:29.576974 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1dd437-58cc96c845-zwq6s"] Aug 04 11:23:30.777712 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:30.777674 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:23:31.443177 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:31.443140 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2a0cc8a-10fc-40f7-942c-1308ad29eca9" path="/var/lib/kubelet/pods/e2a0cc8a-10fc-40f7-942c-1308ad29eca9/volumes" Aug 04 11:23:35.022968 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:35.022939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:35.120941 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:35.120903 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:35.285917 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:35.285833 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:35.509165 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:35.509128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:35.861982 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:35.861950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:36.172063 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:36.171980 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:36.508157 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:36.508031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:38.795883 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:38.795850 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:38.899041 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:38.899004 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:39.895754 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.895715 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt"] Aug 04 11:23:39.896159 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.896121 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a0cc8a-10fc-40f7-942c-1308ad29eca9" containerName="maas-api" Aug 04 11:23:39.896159 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.896133 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a0cc8a-10fc-40f7-942c-1308ad29eca9" containerName="maas-api" Aug 04 11:23:39.896159 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.896158 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d417b675-9133-40d8-812f-126c22c6bdcd" containerName="revoke-keys" Aug 04 11:23:39.896270 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.896165 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d417b675-9133-40d8-812f-126c22c6bdcd" containerName="revoke-keys" Aug 04 11:23:39.896270 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.896234 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a0cc8a-10fc-40f7-942c-1308ad29eca9" containerName="maas-api" Aug 04 11:23:39.896270 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.896247 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d417b675-9133-40d8-812f-126c22c6bdcd" containerName="revoke-keys" Aug 04 11:23:39.904037 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.904013 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:23:39.907391 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.907366 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmlb4\"" Aug 04 11:23:39.907567 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.907436 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 11:23:39.907637 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.907585 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ba3f3c-serving-cert\"" Aug 04 11:23:39.908307 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.908288 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 11:23:39.924623 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.924589 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt"] Aug 04 11:23:39.943826 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.943799 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:39.994091 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.994056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acea64f5-a6bd-41a0-b9ca-460cc802389d-maas-api-tls\") pod \"maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt\" (UID: \"acea64f5-a6bd-41a0-b9ca-460cc802389d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:23:39.994278 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.994095 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea64f5-a6bd-41a0-b9ca-460cc802389d-openshift-service-ca\") pod \"maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt\" (UID: \"acea64f5-a6bd-41a0-b9ca-460cc802389d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:23:39.994278 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:39.994130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhgqp\" (UniqueName: \"kubernetes.io/projected/acea64f5-a6bd-41a0-b9ca-460cc802389d-kube-api-access-mhgqp\") pod \"maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt\" (UID: \"acea64f5-a6bd-41a0-b9ca-460cc802389d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:23:40.059370 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:40.059339 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:40.095007 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:40.094971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acea64f5-a6bd-41a0-b9ca-460cc802389d-maas-api-tls\") pod \"maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt\" (UID: \"acea64f5-a6bd-41a0-b9ca-460cc802389d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:23:40.095214 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:40.095012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea64f5-a6bd-41a0-b9ca-460cc802389d-openshift-service-ca\") pod \"maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt\" (UID: \"acea64f5-a6bd-41a0-b9ca-460cc802389d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:23:40.095214 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:40.095033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhgqp\" (UniqueName: \"kubernetes.io/projected/acea64f5-a6bd-41a0-b9ca-460cc802389d-kube-api-access-mhgqp\") pod \"maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt\" (UID: \"acea64f5-a6bd-41a0-b9ca-460cc802389d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:23:40.095824 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:40.095794 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea64f5-a6bd-41a0-b9ca-460cc802389d-openshift-service-ca\") pod \"maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt\" (UID: \"acea64f5-a6bd-41a0-b9ca-460cc802389d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:23:40.097677 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:40.097647 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acea64f5-a6bd-41a0-b9ca-460cc802389d-maas-api-tls\") pod \"maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt\" (UID: \"acea64f5-a6bd-41a0-b9ca-460cc802389d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:23:40.104236 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:40.104198 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhgqp\" (UniqueName: \"kubernetes.io/projected/acea64f5-a6bd-41a0-b9ca-460cc802389d-kube-api-access-mhgqp\") pod \"maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt\" (UID: \"acea64f5-a6bd-41a0-b9ca-460cc802389d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:23:40.214969 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:40.214880 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:23:40.356793 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:40.356764 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt"] Aug 04 11:23:40.358669 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:23:40.358642 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacea64f5_a6bd_41a0_b9ca_460cc802389d.slice/crio-5e75f5ba15f2ffbde874d4ddd5664bd22a8fd5d4a23a93e9a91b0443efa508c9 WatchSource:0}: Error finding container 5e75f5ba15f2ffbde874d4ddd5664bd22a8fd5d4a23a93e9a91b0443efa508c9: Status 404 returned error can't find the container with id 5e75f5ba15f2ffbde874d4ddd5664bd22a8fd5d4a23a93e9a91b0443efa508c9 Aug 04 11:23:40.360462 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:40.360440 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 11:23:40.572207 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:40.572120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:40.624171 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:40.624135 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" event={"ID":"acea64f5-a6bd-41a0-b9ca-460cc802389d","Type":"ContainerStarted","Data":"5e75f5ba15f2ffbde874d4ddd5664bd22a8fd5d4a23a93e9a91b0443efa508c9"} Aug 04 11:23:40.677208 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:40.677178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:41.631999 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:41.631963 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" event={"ID":"acea64f5-a6bd-41a0-b9ca-460cc802389d","Type":"ContainerStarted","Data":"bd9bbea9c29b4977812800663c34df5d47476a4f0b419af1cd7b64555bfe88c3"} Aug 04 11:23:41.632433 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:41.632098 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:23:41.651277 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:41.651209 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" podStartSLOduration=2.091094704 podStartE2EDuration="2.651192185s" totalImagesPullingTime="560.097481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:23:41.649090655 +0000 UTC m=+2810.746884050" watchObservedRunningTime="2026-08-04 11:23:41.651192185 +0000 UTC m=+2810.748985579" Aug 04 11:23:42.678971 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:42.678929 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:23:42.873919 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:42.873876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:42.930800 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:42.930693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:23:47.643029 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:23:47.642997 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:24:03.803427 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:03.803394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:04.221568 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:04.221477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:04.533614 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:04.533488 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:05.321476 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:05.321422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:07.920836 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:07.920793 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:08.020388 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:08.020298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:09.003283 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.003248 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg"] Aug 04 11:24:09.007161 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.007138 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:24:09.009678 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.009657 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-46695b-serving-cert\"" Aug 04 11:24:09.019219 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.019191 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg"] Aug 04 11:24:09.041840 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.041802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-openshift-service-ca\") pod \"maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg\" (UID: \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:24:09.042042 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.041903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-maas-api-tls\") pod \"maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg\" (UID: \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:24:09.042042 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.041965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpt9n\" (UniqueName: \"kubernetes.io/projected/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-kube-api-access-tpt9n\") pod \"maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg\" (UID: \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:24:09.070182 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.070152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:09.142652 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.142616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-openshift-service-ca\") pod \"maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg\" (UID: \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:24:09.142830 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.142687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-maas-api-tls\") pod \"maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg\" (UID: \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:24:09.142830 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.142731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpt9n\" (UniqueName: \"kubernetes.io/projected/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-kube-api-access-tpt9n\") pod \"maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg\" (UID: \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:24:09.142901 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:24:09.142838 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-serving-cert: secret "maas-api-e2e-api-b-46695b-serving-cert" not found Aug 04 11:24:09.142934 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:24:09.142915 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-maas-api-tls podName:6250a80a-132e-42c1-82e2-d5cd7daf1bd2 nodeName:}" failed. No retries permitted until 2026-08-04 11:24:09.642898511 +0000 UTC m=+2838.740691889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-maas-api-tls") pod "maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" (UID: "6250a80a-132e-42c1-82e2-d5cd7daf1bd2") : secret "maas-api-e2e-api-b-46695b-serving-cert" not found Aug 04 11:24:09.143230 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.143212 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-openshift-service-ca\") pod \"maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg\" (UID: \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:24:09.158369 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.158344 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpt9n\" (UniqueName: \"kubernetes.io/projected/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-kube-api-access-tpt9n\") pod \"maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg\" (UID: \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:24:09.426899 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.426862 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:09.645992 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.645960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-maas-api-tls\") pod \"maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg\" (UID: \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:24:09.648372 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.648341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-maas-api-tls\") pod \"maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg\" (UID: \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:24:09.667750 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.667722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:09.771394 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.771288 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:09.920729 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:09.920688 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:24:10.054150 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:10.054124 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg"] Aug 04 11:24:10.056109 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:24:10.056086 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6250a80a_132e_42c1_82e2_d5cd7daf1bd2.slice/crio-69267527eb5eb6a867cc97a4d524f106cfebf597a69e826375273229e72cccc0 WatchSource:0}: Error finding container 69267527eb5eb6a867cc97a4d524f106cfebf597a69e826375273229e72cccc0: Status 404 returned error can't find the container with id 69267527eb5eb6a867cc97a4d524f106cfebf597a69e826375273229e72cccc0 Aug 04 11:24:10.808797 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:10.808701 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" event={"ID":"6250a80a-132e-42c1-82e2-d5cd7daf1bd2","Type":"ContainerStarted","Data":"ece4e3a7c651ca36827d9ec8517bac29e805983a0df1f5f52cfc2c3d78e38e4e"} Aug 04 11:24:10.808953 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:10.808893 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:24:10.808953 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:10.808926 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" event={"ID":"6250a80a-132e-42c1-82e2-d5cd7daf1bd2","Type":"ContainerStarted","Data":"69267527eb5eb6a867cc97a4d524f106cfebf597a69e826375273229e72cccc0"} Aug 04 11:24:10.825618 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:10.825566 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" podStartSLOduration=2.324762947 podStartE2EDuration="2.825549948s" totalImagesPullingTime="500.787001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:24:10.824197264 +0000 UTC m=+2839.921990658" watchObservedRunningTime="2026-08-04 11:24:10.825549948 +0000 UTC m=+2839.923343338" Aug 04 11:24:12.387561 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:12.387523 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:24:16.820899 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:16.820871 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:24:32.911258 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:32.911224 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:35.705423 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:35.705384 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc"] Aug 04 11:24:35.708941 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:35.708923 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc" Aug 04 11:24:35.711302 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:35.711282 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tgsgx\"" Aug 04 11:24:35.715894 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:35.715855 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc"] Aug 04 11:24:35.780799 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:35.780763 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a963017b-35e8-46ff-8adf-90c1e61c9ec1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc\" (UID: \"a963017b-35e8-46ff-8adf-90c1e61c9ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc" Aug 04 11:24:35.881865 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:35.881832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a963017b-35e8-46ff-8adf-90c1e61c9ec1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc\" (UID: \"a963017b-35e8-46ff-8adf-90c1e61c9ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc" Aug 04 11:24:35.882541 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:35.882495 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a963017b-35e8-46ff-8adf-90c1e61c9ec1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc\" (UID: \"a963017b-35e8-46ff-8adf-90c1e61c9ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc" Aug 04 11:24:36.021786 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:36.021698 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc" Aug 04 11:24:36.158314 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:36.158289 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc"] Aug 04 11:24:36.161171 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:24:36.161132 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda963017b_35e8_46ff_8adf_90c1e61c9ec1.slice/crio-f3de1b1e887922f59006d2a673eeeaa8cfc7f4d5292ff6f5b1d51e035f191b83 WatchSource:0}: Error finding container f3de1b1e887922f59006d2a673eeeaa8cfc7f4d5292ff6f5b1d51e035f191b83: Status 404 returned error can't find the container with id f3de1b1e887922f59006d2a673eeeaa8cfc7f4d5292ff6f5b1d51e035f191b83 Aug 04 11:24:36.964156 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:36.964114 2576 generic.go:356] "Generic (PLEG): container finished" podID="a963017b-35e8-46ff-8adf-90c1e61c9ec1" containerID="8c2516ac27492555408d0f75b0df4b707dd2477f249a897bbf18914607729064" exitCode=0 Aug 04 11:24:36.964551 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:36.964165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc" event={"ID":"a963017b-35e8-46ff-8adf-90c1e61c9ec1","Type":"ContainerDied","Data":"8c2516ac27492555408d0f75b0df4b707dd2477f249a897bbf18914607729064"} Aug 04 11:24:36.964551 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:36.964243 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc" event={"ID":"a963017b-35e8-46ff-8adf-90c1e61c9ec1","Type":"ContainerStarted","Data":"f3de1b1e887922f59006d2a673eeeaa8cfc7f4d5292ff6f5b1d51e035f191b83"} Aug 04 11:24:38.101428 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:38.101399 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc" Aug 04 11:24:38.207948 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:38.207908 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a963017b-35e8-46ff-8adf-90c1e61c9ec1-maas-api-service-ca\") pod \"a963017b-35e8-46ff-8adf-90c1e61c9ec1\" (UID: \"a963017b-35e8-46ff-8adf-90c1e61c9ec1\") " Aug 04 11:24:38.208256 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:38.208234 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a963017b-35e8-46ff-8adf-90c1e61c9ec1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a963017b-35e8-46ff-8adf-90c1e61c9ec1" (UID: "a963017b-35e8-46ff-8adf-90c1e61c9ec1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:24:38.309302 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:38.309203 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a963017b-35e8-46ff-8adf-90c1e61c9ec1-maas-api-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:24:38.976350 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:38.976320 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc" Aug 04 11:24:38.976567 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:38.976323 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc" event={"ID":"a963017b-35e8-46ff-8adf-90c1e61c9ec1","Type":"ContainerDied","Data":"f3de1b1e887922f59006d2a673eeeaa8cfc7f4d5292ff6f5b1d51e035f191b83"} Aug 04 11:24:38.976567 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:38.976455 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3de1b1e887922f59006d2a673eeeaa8cfc7f4d5292ff6f5b1d51e035f191b83" Aug 04 11:24:51.167657 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.167570 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt"] Aug 04 11:24:51.168108 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.167829 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" podUID="acea64f5-a6bd-41a0-b9ca-460cc802389d" containerName="maas-api" containerID="cri-o://bd9bbea9c29b4977812800663c34df5d47476a4f0b419af1cd7b64555bfe88c3" gracePeriod=30 Aug 04 11:24:51.202118 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.202084 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:51.299633 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.299598 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:51.347206 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.347164 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc"] Aug 04 11:24:51.350229 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.350193 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba3f3c-lxlfc"] Aug 04 11:24:51.402099 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.402057 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:51.423550 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.423472 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:24:51.437944 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.437908 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a963017b-35e8-46ff-8adf-90c1e61c9ec1" path="/var/lib/kubelet/pods/a963017b-35e8-46ff-8adf-90c1e61c9ec1/volumes" Aug 04 11:24:51.528874 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.528822 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acea64f5-a6bd-41a0-b9ca-460cc802389d-maas-api-tls\") pod \"acea64f5-a6bd-41a0-b9ca-460cc802389d\" (UID: \"acea64f5-a6bd-41a0-b9ca-460cc802389d\") " Aug 04 11:24:51.529070 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.528937 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhgqp\" (UniqueName: \"kubernetes.io/projected/acea64f5-a6bd-41a0-b9ca-460cc802389d-kube-api-access-mhgqp\") pod \"acea64f5-a6bd-41a0-b9ca-460cc802389d\" (UID: \"acea64f5-a6bd-41a0-b9ca-460cc802389d\") " Aug 04 11:24:51.529070 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.529064 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea64f5-a6bd-41a0-b9ca-460cc802389d-openshift-service-ca\") pod \"acea64f5-a6bd-41a0-b9ca-460cc802389d\" (UID: \"acea64f5-a6bd-41a0-b9ca-460cc802389d\") " Aug 04 11:24:51.529425 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.529394 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acea64f5-a6bd-41a0-b9ca-460cc802389d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "acea64f5-a6bd-41a0-b9ca-460cc802389d" (UID: "acea64f5-a6bd-41a0-b9ca-460cc802389d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:24:51.531131 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.531107 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acea64f5-a6bd-41a0-b9ca-460cc802389d-kube-api-access-mhgqp" (OuterVolumeSpecName: "kube-api-access-mhgqp") pod "acea64f5-a6bd-41a0-b9ca-460cc802389d" (UID: "acea64f5-a6bd-41a0-b9ca-460cc802389d"). InnerVolumeSpecName "kube-api-access-mhgqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:24:51.531243 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.531188 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acea64f5-a6bd-41a0-b9ca-460cc802389d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "acea64f5-a6bd-41a0-b9ca-460cc802389d" (UID: "acea64f5-a6bd-41a0-b9ca-460cc802389d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:24:51.629916 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.629882 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acea64f5-a6bd-41a0-b9ca-460cc802389d-openshift-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:24:51.629916 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.629913 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acea64f5-a6bd-41a0-b9ca-460cc802389d-maas-api-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:24:51.629916 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.629922 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhgqp\" (UniqueName: \"kubernetes.io/projected/acea64f5-a6bd-41a0-b9ca-460cc802389d-kube-api-access-mhgqp\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:24:51.714760 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:51.714673 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:52.048630 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:52.048474 2576 generic.go:356] "Generic (PLEG): container finished" podID="acea64f5-a6bd-41a0-b9ca-460cc802389d" containerID="bd9bbea9c29b4977812800663c34df5d47476a4f0b419af1cd7b64555bfe88c3" exitCode=0 Aug 04 11:24:52.048630 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:52.048558 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" Aug 04 11:24:52.048834 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:52.048653 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" event={"ID":"acea64f5-a6bd-41a0-b9ca-460cc802389d","Type":"ContainerDied","Data":"bd9bbea9c29b4977812800663c34df5d47476a4f0b419af1cd7b64555bfe88c3"} Aug 04 11:24:52.048834 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:52.048705 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt" event={"ID":"acea64f5-a6bd-41a0-b9ca-460cc802389d","Type":"ContainerDied","Data":"5e75f5ba15f2ffbde874d4ddd5664bd22a8fd5d4a23a93e9a91b0443efa508c9"} Aug 04 11:24:52.048834 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:52.048728 2576 scope.go:117] "RemoveContainer" containerID="bd9bbea9c29b4977812800663c34df5d47476a4f0b419af1cd7b64555bfe88c3" Aug 04 11:24:52.058449 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:52.058430 2576 scope.go:117] "RemoveContainer" containerID="bd9bbea9c29b4977812800663c34df5d47476a4f0b419af1cd7b64555bfe88c3" Aug 04 11:24:52.058759 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:24:52.058739 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9bbea9c29b4977812800663c34df5d47476a4f0b419af1cd7b64555bfe88c3\": container with ID starting with bd9bbea9c29b4977812800663c34df5d47476a4f0b419af1cd7b64555bfe88c3 not found: ID does not exist" containerID="bd9bbea9c29b4977812800663c34df5d47476a4f0b419af1cd7b64555bfe88c3" Aug 04 11:24:52.058810 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:52.058769 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9bbea9c29b4977812800663c34df5d47476a4f0b419af1cd7b64555bfe88c3"} err="failed to get container status \"bd9bbea9c29b4977812800663c34df5d47476a4f0b419af1cd7b64555bfe88c3\": rpc error: code = NotFound desc = could not find container \"bd9bbea9c29b4977812800663c34df5d47476a4f0b419af1cd7b64555bfe88c3\": container with ID starting with bd9bbea9c29b4977812800663c34df5d47476a4f0b419af1cd7b64555bfe88c3 not found: ID does not exist" Aug 04 11:24:52.072456 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:52.072425 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt"] Aug 04 11:24:52.078839 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:52.078810 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ba3f3c-7d9cd56b46-n64jt"] Aug 04 11:24:53.438030 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:53.437994 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acea64f5-a6bd-41a0-b9ca-460cc802389d" path="/var/lib/kubelet/pods/acea64f5-a6bd-41a0-b9ca-460cc802389d/volumes" Aug 04 11:24:53.583813 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:53.583779 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:24:57.447942 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:57.447910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:57.558843 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:57.558801 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:57.698179 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:57.698091 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:57.916030 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:57.915995 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:24:58.032023 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.031940 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx"] Aug 04 11:24:58.032339 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.032327 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acea64f5-a6bd-41a0-b9ca-460cc802389d" containerName="maas-api" Aug 04 11:24:58.032386 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.032340 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="acea64f5-a6bd-41a0-b9ca-460cc802389d" containerName="maas-api" Aug 04 11:24:58.032386 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.032356 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a963017b-35e8-46ff-8adf-90c1e61c9ec1" containerName="revoke-keys" Aug 04 11:24:58.032386 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.032362 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a963017b-35e8-46ff-8adf-90c1e61c9ec1" containerName="revoke-keys" Aug 04 11:24:58.032474 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.032426 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a963017b-35e8-46ff-8adf-90c1e61c9ec1" containerName="revoke-keys" Aug 04 11:24:58.032474 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.032437 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="acea64f5-a6bd-41a0-b9ca-460cc802389d" containerName="maas-api" Aug 04 11:24:58.035947 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.035928 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx" Aug 04 11:24:58.038365 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.038344 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tgsgx\"" Aug 04 11:24:58.043167 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.043142 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx"] Aug 04 11:24:58.190052 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.190008 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e6db13-34fb-4676-b958-bccf6703dd7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-46695b-djqfx\" (UID: \"29e6db13-34fb-4676-b958-bccf6703dd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx" Aug 04 11:24:58.291387 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.291300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e6db13-34fb-4676-b958-bccf6703dd7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-46695b-djqfx\" (UID: \"29e6db13-34fb-4676-b958-bccf6703dd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx" Aug 04 11:24:58.292105 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.292083 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e6db13-34fb-4676-b958-bccf6703dd7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-46695b-djqfx\" (UID: \"29e6db13-34fb-4676-b958-bccf6703dd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx" Aug 04 11:24:58.347812 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.347775 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx" Aug 04 11:24:58.497087 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:58.497055 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx"] Aug 04 11:24:58.499416 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:24:58.499386 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e6db13_34fb_4676_b958_bccf6703dd7b.slice/crio-287ead87034af4ab98334ce43eaf4820750974758325a4e91ffe840f1a4c56fa WatchSource:0}: Error finding container 287ead87034af4ab98334ce43eaf4820750974758325a4e91ffe840f1a4c56fa: Status 404 returned error can't find the container with id 287ead87034af4ab98334ce43eaf4820750974758325a4e91ffe840f1a4c56fa Aug 04 11:24:59.097169 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:59.097132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx" event={"ID":"29e6db13-34fb-4676-b958-bccf6703dd7b","Type":"ContainerStarted","Data":"d37d896c83103255756fb6b76ca2c2ef72ec8d89322194af7f926597c42f2e87"} Aug 04 11:24:59.097356 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:59.097196 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx" event={"ID":"29e6db13-34fb-4676-b958-bccf6703dd7b","Type":"ContainerStarted","Data":"287ead87034af4ab98334ce43eaf4820750974758325a4e91ffe840f1a4c56fa"} Aug 04 11:24:59.113360 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:24:59.113302 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx" podStartSLOduration=1.113284951 podStartE2EDuration="1.113284951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:24:59.112136156 +0000 UTC m=+2888.209929549" watchObservedRunningTime="2026-08-04 11:24:59.113284951 +0000 UTC m=+2888.211078396" Aug 04 11:25:00.104847 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:00.104813 2576 generic.go:356] "Generic (PLEG): container finished" podID="29e6db13-34fb-4676-b958-bccf6703dd7b" containerID="d37d896c83103255756fb6b76ca2c2ef72ec8d89322194af7f926597c42f2e87" exitCode=0 Aug 04 11:25:00.105295 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:00.104855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx" event={"ID":"29e6db13-34fb-4676-b958-bccf6703dd7b","Type":"ContainerDied","Data":"d37d896c83103255756fb6b76ca2c2ef72ec8d89322194af7f926597c42f2e87"} Aug 04 11:25:01.244211 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:01.244185 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx" Aug 04 11:25:01.315578 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:01.315538 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e6db13-34fb-4676-b958-bccf6703dd7b-maas-api-service-ca\") pod \"29e6db13-34fb-4676-b958-bccf6703dd7b\" (UID: \"29e6db13-34fb-4676-b958-bccf6703dd7b\") " Aug 04 11:25:01.315865 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:01.315843 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e6db13-34fb-4676-b958-bccf6703dd7b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "29e6db13-34fb-4676-b958-bccf6703dd7b" (UID: "29e6db13-34fb-4676-b958-bccf6703dd7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:25:01.416285 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:01.416178 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e6db13-34fb-4676-b958-bccf6703dd7b-maas-api-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:25:02.117880 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:02.117839 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx" event={"ID":"29e6db13-34fb-4676-b958-bccf6703dd7b","Type":"ContainerDied","Data":"287ead87034af4ab98334ce43eaf4820750974758325a4e91ffe840f1a4c56fa"} Aug 04 11:25:02.118092 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:02.117900 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="287ead87034af4ab98334ce43eaf4820750974758325a4e91ffe840f1a4c56fa" Aug 04 11:25:02.118092 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:02.117855 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx" Aug 04 11:25:09.833230 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:09.833197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:13.628431 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:13.628401 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg"] Aug 04 11:25:13.628869 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:13.628605 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" podUID="6250a80a-132e-42c1-82e2-d5cd7daf1bd2" containerName="maas-api" containerID="cri-o://ece4e3a7c651ca36827d9ec8517bac29e805983a0df1f5f52cfc2c3d78e38e4e" gracePeriod=30 Aug 04 11:25:13.651081 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:13.651049 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:13.838331 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:13.838302 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx"] Aug 04 11:25:13.840418 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:13.840396 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-46695b-djqfx"] Aug 04 11:25:13.899092 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:13.899068 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:25:13.908310 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:13.908287 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-maas-api-tls\") pod \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\" (UID: \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\") " Aug 04 11:25:13.908390 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:13.908322 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpt9n\" (UniqueName: \"kubernetes.io/projected/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-kube-api-access-tpt9n\") pod \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\" (UID: \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\") " Aug 04 11:25:13.908390 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:13.908368 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-openshift-service-ca\") pod \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\" (UID: \"6250a80a-132e-42c1-82e2-d5cd7daf1bd2\") " Aug 04 11:25:13.908780 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:13.908754 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6250a80a-132e-42c1-82e2-d5cd7daf1bd2" (UID: "6250a80a-132e-42c1-82e2-d5cd7daf1bd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:25:13.910340 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:13.910323 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6250a80a-132e-42c1-82e2-d5cd7daf1bd2" (UID: "6250a80a-132e-42c1-82e2-d5cd7daf1bd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:25:13.910410 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:13.910393 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-kube-api-access-tpt9n" (OuterVolumeSpecName: "kube-api-access-tpt9n") pod "6250a80a-132e-42c1-82e2-d5cd7daf1bd2" (UID: "6250a80a-132e-42c1-82e2-d5cd7daf1bd2"). InnerVolumeSpecName "kube-api-access-tpt9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:25:13.930970 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:13.930938 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:14.009237 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:14.009194 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-maas-api-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:25:14.009237 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:14.009231 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpt9n\" (UniqueName: \"kubernetes.io/projected/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-kube-api-access-tpt9n\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:25:14.009237 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:14.009242 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6250a80a-132e-42c1-82e2-d5cd7daf1bd2-openshift-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:25:14.026267 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:14.026231 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:14.188061 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:14.187968 2576 generic.go:356] "Generic (PLEG): container finished" podID="6250a80a-132e-42c1-82e2-d5cd7daf1bd2" containerID="ece4e3a7c651ca36827d9ec8517bac29e805983a0df1f5f52cfc2c3d78e38e4e" exitCode=0 Aug 04 11:25:14.188061 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:14.188020 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" event={"ID":"6250a80a-132e-42c1-82e2-d5cd7daf1bd2","Type":"ContainerDied","Data":"ece4e3a7c651ca36827d9ec8517bac29e805983a0df1f5f52cfc2c3d78e38e4e"} Aug 04 11:25:14.188061 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:14.188036 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" Aug 04 11:25:14.188378 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:14.188086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg" event={"ID":"6250a80a-132e-42c1-82e2-d5cd7daf1bd2","Type":"ContainerDied","Data":"69267527eb5eb6a867cc97a4d524f106cfebf597a69e826375273229e72cccc0"} Aug 04 11:25:14.188378 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:14.188108 2576 scope.go:117] "RemoveContainer" containerID="ece4e3a7c651ca36827d9ec8517bac29e805983a0df1f5f52cfc2c3d78e38e4e" Aug 04 11:25:14.197777 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:14.197757 2576 scope.go:117] "RemoveContainer" containerID="ece4e3a7c651ca36827d9ec8517bac29e805983a0df1f5f52cfc2c3d78e38e4e" Aug 04 11:25:14.198091 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:25:14.198071 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece4e3a7c651ca36827d9ec8517bac29e805983a0df1f5f52cfc2c3d78e38e4e\": container with ID starting with ece4e3a7c651ca36827d9ec8517bac29e805983a0df1f5f52cfc2c3d78e38e4e not found: ID does not exist" containerID="ece4e3a7c651ca36827d9ec8517bac29e805983a0df1f5f52cfc2c3d78e38e4e" Aug 04 11:25:14.198143 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:14.198100 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece4e3a7c651ca36827d9ec8517bac29e805983a0df1f5f52cfc2c3d78e38e4e"} err="failed to get container status \"ece4e3a7c651ca36827d9ec8517bac29e805983a0df1f5f52cfc2c3d78e38e4e\": rpc error: code = NotFound desc = could not find container \"ece4e3a7c651ca36827d9ec8517bac29e805983a0df1f5f52cfc2c3d78e38e4e\": container with ID starting with ece4e3a7c651ca36827d9ec8517bac29e805983a0df1f5f52cfc2c3d78e38e4e not found: ID does not exist" Aug 04 11:25:14.212100 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:14.212066 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg"] Aug 04 11:25:14.216329 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:14.216300 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-46695b-7c75ccc8d7-4q2lg"] Aug 04 11:25:14.225002 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:14.224980 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:15.438211 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:15.438178 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29e6db13-34fb-4676-b958-bccf6703dd7b" path="/var/lib/kubelet/pods/29e6db13-34fb-4676-b958-bccf6703dd7b/volumes" Aug 04 11:25:15.438605 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:15.438590 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6250a80a-132e-42c1-82e2-d5cd7daf1bd2" path="/var/lib/kubelet/pods/6250a80a-132e-42c1-82e2-d5cd7daf1bd2/volumes" Aug 04 11:25:15.781400 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:15.781311 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:25:20.222007 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:20.221977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:20.374546 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:20.374493 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:20.492537 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:20.492422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:20.722273 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:20.722240 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:21.045858 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:21.045827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:21.343549 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:21.343494 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:21.470759 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:21.470728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:24.326159 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:24.326131 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:24.424106 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:24.424076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:25.480688 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.480655 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5"] Aug 04 11:25:25.481223 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.481203 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6250a80a-132e-42c1-82e2-d5cd7daf1bd2" containerName="maas-api" Aug 04 11:25:25.481295 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.481226 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6250a80a-132e-42c1-82e2-d5cd7daf1bd2" containerName="maas-api" Aug 04 11:25:25.481295 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.481240 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e6db13-34fb-4676-b958-bccf6703dd7b" containerName="revoke-keys" Aug 04 11:25:25.481295 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.481248 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e6db13-34fb-4676-b958-bccf6703dd7b" containerName="revoke-keys" Aug 04 11:25:25.481443 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.481353 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e6db13-34fb-4676-b958-bccf6703dd7b" containerName="revoke-keys" Aug 04 11:25:25.481443 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.481374 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6250a80a-132e-42c1-82e2-d5cd7daf1bd2" containerName="maas-api" Aug 04 11:25:25.485848 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.485827 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:25:25.488439 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.488418 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 11:25:25.489180 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.489145 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 11:25:25.489180 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.489165 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-ed8ca8-serving-cert\"" Aug 04 11:25:25.489357 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.489155 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmlb4\"" Aug 04 11:25:25.497295 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.497272 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5"] Aug 04 11:25:25.526779 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.526746 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:25.608051 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.608019 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe00718-f1a6-4f52-8aee-125b654ed662-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5\" (UID: \"dbe00718-f1a6-4f52-8aee-125b654ed662\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:25:25.608253 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.608066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmw4x\" (UniqueName: \"kubernetes.io/projected/dbe00718-f1a6-4f52-8aee-125b654ed662-kube-api-access-bmw4x\") pod \"maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5\" (UID: \"dbe00718-f1a6-4f52-8aee-125b654ed662\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:25:25.608253 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.608202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbe00718-f1a6-4f52-8aee-125b654ed662-maas-api-tls\") pod \"maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5\" (UID: \"dbe00718-f1a6-4f52-8aee-125b654ed662\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:25:25.709373 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.709325 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmw4x\" (UniqueName: \"kubernetes.io/projected/dbe00718-f1a6-4f52-8aee-125b654ed662-kube-api-access-bmw4x\") pod \"maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5\" (UID: \"dbe00718-f1a6-4f52-8aee-125b654ed662\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:25:25.709589 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.709412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbe00718-f1a6-4f52-8aee-125b654ed662-maas-api-tls\") pod \"maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5\" (UID: \"dbe00718-f1a6-4f52-8aee-125b654ed662\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:25:25.709589 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.709525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe00718-f1a6-4f52-8aee-125b654ed662-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5\" (UID: \"dbe00718-f1a6-4f52-8aee-125b654ed662\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:25:25.709749 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:25:25.709586 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-serving-cert: secret "maas-api-e2e-shared-a-ed8ca8-serving-cert" not found Aug 04 11:25:25.709749 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:25:25.709684 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe00718-f1a6-4f52-8aee-125b654ed662-maas-api-tls podName:dbe00718-f1a6-4f52-8aee-125b654ed662 nodeName:}" failed. No retries permitted until 2026-08-04 11:25:26.209660731 +0000 UTC m=+2915.307454114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dbe00718-f1a6-4f52-8aee-125b654ed662-maas-api-tls") pod "maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" (UID: "dbe00718-f1a6-4f52-8aee-125b654ed662") : secret "maas-api-e2e-shared-a-ed8ca8-serving-cert" not found Aug 04 11:25:25.710281 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.710261 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe00718-f1a6-4f52-8aee-125b654ed662-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5\" (UID: \"dbe00718-f1a6-4f52-8aee-125b654ed662\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:25:25.721277 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.721208 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmw4x\" (UniqueName: \"kubernetes.io/projected/dbe00718-f1a6-4f52-8aee-125b654ed662-kube-api-access-bmw4x\") pod \"maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5\" (UID: \"dbe00718-f1a6-4f52-8aee-125b654ed662\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:25:25.936560 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:25.936492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:26.040197 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:26.040165 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:26.182601 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:26.182570 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:26.213995 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:26.213907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbe00718-f1a6-4f52-8aee-125b654ed662-maas-api-tls\") pod \"maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5\" (UID: \"dbe00718-f1a6-4f52-8aee-125b654ed662\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:25:26.216327 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:26.216307 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbe00718-f1a6-4f52-8aee-125b654ed662-maas-api-tls\") pod \"maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5\" (UID: \"dbe00718-f1a6-4f52-8aee-125b654ed662\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:25:26.398263 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:26.398228 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:25:26.537210 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:26.537185 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5"] Aug 04 11:25:26.539388 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:25:26.539365 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe00718_f1a6_4f52_8aee_125b654ed662.slice/crio-0d6f8ab109792be77adfe097bbaae050b88ada5b7151ef1145c054bff1bbea19 WatchSource:0}: Error finding container 0d6f8ab109792be77adfe097bbaae050b88ada5b7151ef1145c054bff1bbea19: Status 404 returned error can't find the container with id 0d6f8ab109792be77adfe097bbaae050b88ada5b7151ef1145c054bff1bbea19 Aug 04 11:25:27.266271 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:27.266224 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" event={"ID":"dbe00718-f1a6-4f52-8aee-125b654ed662","Type":"ContainerStarted","Data":"f00b4cc06297521fc79906ca6b1340d27656784d5d180e639c23f9c4bbc73047"} Aug 04 11:25:27.266450 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:27.266294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" event={"ID":"dbe00718-f1a6-4f52-8aee-125b654ed662","Type":"ContainerStarted","Data":"0d6f8ab109792be77adfe097bbaae050b88ada5b7151ef1145c054bff1bbea19"} Aug 04 11:25:27.266450 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:27.266412 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:25:27.284626 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:27.284573 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" podStartSLOduration=1.7369148 podStartE2EDuration="2.284558406s" totalImagesPullingTime="547.643606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:25:27.282920278 +0000 UTC m=+2916.380713674" watchObservedRunningTime="2026-08-04 11:25:27.284558406 +0000 UTC m=+2916.382351793" Aug 04 11:25:28.488189 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:28.488154 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:25:33.277612 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:33.277580 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:25:46.207749 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:46.207718 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:49.270331 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:49.270290 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:49.608578 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:49.608549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:49.930122 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:49.930033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:50.424806 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:50.424768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:52.469325 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:52.469293 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:52.566576 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:52.566542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:53.570158 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.570129 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56"] Aug 04 11:25:53.575095 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.575076 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:25:53.577442 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.577424 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c74755-serving-cert\"" Aug 04 11:25:53.586986 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.586958 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56"] Aug 04 11:25:53.635150 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.635120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:53.758653 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.758623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56\" (UID: \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:25:53.758862 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.758686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpvmk\" (UniqueName: \"kubernetes.io/projected/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-kube-api-access-dpvmk\") pod \"maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56\" (UID: \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:25:53.758862 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.758817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-maas-api-tls\") pod \"maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56\" (UID: \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:25:53.859409 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.859377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-maas-api-tls\") pod \"maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56\" (UID: \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:25:53.859634 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.859436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56\" (UID: \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:25:53.859634 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.859480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpvmk\" (UniqueName: \"kubernetes.io/projected/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-kube-api-access-dpvmk\") pod \"maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56\" (UID: \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:25:53.860211 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.860172 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56\" (UID: \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:25:53.861887 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.861859 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-maas-api-tls\") pod \"maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56\" (UID: \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:25:53.871127 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.871098 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpvmk\" (UniqueName: \"kubernetes.io/projected/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-kube-api-access-dpvmk\") pod \"maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56\" (UID: \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:25:53.887054 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.887028 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:25:53.930744 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:53.930714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:54.034273 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:54.034248 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56"] Aug 04 11:25:54.034757 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:25:54.034725 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d67ddb_79d6_44bf_9e44_a7aef71c393e.slice/crio-2aff506c674faa26e505263a8f514ee4b91d19707284cd99eb6b46053bac2ff4 WatchSource:0}: Error finding container 2aff506c674faa26e505263a8f514ee4b91d19707284cd99eb6b46053bac2ff4: Status 404 returned error can't find the container with id 2aff506c674faa26e505263a8f514ee4b91d19707284cd99eb6b46053bac2ff4 Aug 04 11:25:54.323331 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:54.323249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:54.421989 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:54.421949 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" event={"ID":"f6d67ddb-79d6-44bf-9e44-a7aef71c393e","Type":"ContainerStarted","Data":"2aff506c674faa26e505263a8f514ee4b91d19707284cd99eb6b46053bac2ff4"} Aug 04 11:25:54.422998 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:54.422976 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:25:55.429257 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:55.429225 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" event={"ID":"f6d67ddb-79d6-44bf-9e44-a7aef71c393e","Type":"ContainerStarted","Data":"22f5222a16df153869fcab808eb73d41999ec1863c2195098513d702a13387bf"} Aug 04 11:25:55.429656 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:55.429369 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:25:55.459246 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:55.459196 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" podStartSLOduration=1.8741263849999998 podStartE2EDuration="2.45918023s" totalImagesPullingTime="585.053845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:25:55.457204597 +0000 UTC m=+2944.554997990" watchObservedRunningTime="2026-08-04 11:25:55.45918023 +0000 UTC m=+2944.556973623" Aug 04 11:25:56.589692 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:25:56.589660 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:26:01.446350 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:26:01.446323 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:26:15.891384 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:26:15.891338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:26:17.425640 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:26:17.425604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:26:20.698188 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:26:20.698156 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:26:22.991329 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:26:22.991292 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:26:42.499143 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:26:42.499108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:26:44.912738 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:26:44.912702 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:27:07.486291 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:27:07.486254 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:27:22.981511 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:27:22.981462 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:27:33.386429 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:27:33.386391 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:27:45.769007 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:27:45.768909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:27:49.382078 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:27:49.382047 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:28:04.885063 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:28:04.885025 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:28:14.881101 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:28:14.881066 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:28:29.676284 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:28:29.676249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:28:31.079300 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:28:31.079261 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:28:46.584659 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:28:46.584624 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:28:56.427493 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:28:56.427455 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:29:12.799160 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:29:12.799114 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:29:28.578212 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:29:28.578134 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:29:39.186749 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:29:39.186712 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:29:51.841048 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:29:51.841016 2576 scope.go:117] "RemoveContainer" containerID="7dad8fafb51e3d1fafd9237846194e072391172efb11c9c825c03e642eb5d3a6" Aug 04 11:29:51.850097 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:29:51.850076 2576 scope.go:117] "RemoveContainer" containerID="5f665e38a31d781f2ace9cfde8b2199b67cf729204d1961574332db5e1569418" Aug 04 11:29:54.385417 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:29:54.385377 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:30:00.134872 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.134814 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ed8ca8-29764050-7vxtf"] Aug 04 11:30:00.140083 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.140057 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c74755-29764050-8msxn"] Aug 04 11:30:00.140243 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.140179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ed8ca8-29764050-7vxtf" Aug 04 11:30:00.149552 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.143034 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tgsgx\"" Aug 04 11:30:00.149552 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.146656 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764050-xv6m5"] Aug 04 11:30:00.149552 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.147870 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c74755-29764050-8msxn" Aug 04 11:30:00.152223 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.152200 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764050-xv6m5" Aug 04 11:30:00.154007 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.153980 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c74755-29764050-8msxn"] Aug 04 11:30:00.157253 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.157225 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ed8ca8-29764050-7vxtf"] Aug 04 11:30:00.157370 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.157323 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ed8ca8-29764050-7vxtf" Aug 04 11:30:00.168193 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.168165 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c74755-29764050-8msxn" Aug 04 11:30:00.171194 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.171159 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764050-xv6m5"] Aug 04 11:30:00.202708 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.202684 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764050-xv6m5" Aug 04 11:30:00.328967 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.328906 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ed8ca8-29764050-7vxtf"] Aug 04 11:30:00.333524 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:30:00.333464 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3d8bf7_954d_44b3_b08e_f250ddeed083.slice/crio-cb827bc16832e58381089a0604b61e7ad66ceae541f555fbf9a9b098cb00c13b WatchSource:0}: Error finding container cb827bc16832e58381089a0604b61e7ad66ceae541f555fbf9a9b098cb00c13b: Status 404 returned error can't find the container with id cb827bc16832e58381089a0604b61e7ad66ceae541f555fbf9a9b098cb00c13b Aug 04 11:30:00.364343 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.364318 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c74755-29764050-8msxn"] Aug 04 11:30:00.367195 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:30:00.367163 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8189562_2ec7_41f6_a500_b720098b0759.slice/crio-92409b8b38e5e40c7a7f45c3e0953018dfbc653acc3c3b1177f67f597153d8c0 WatchSource:0}: Error finding container 92409b8b38e5e40c7a7f45c3e0953018dfbc653acc3c3b1177f67f597153d8c0: Status 404 returned error can't find the container with id 92409b8b38e5e40c7a7f45c3e0953018dfbc653acc3c3b1177f67f597153d8c0 Aug 04 11:30:00.389946 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.389918 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764050-xv6m5"] Aug 04 11:30:00.406910 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:30:00.406886 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7380efc_86ca_45c0_98ea_21fb4961d7fc.slice/crio-ae452def15e05fb8e6a33d9a71ccc4bc351718be81515fb06f8fa5db18b34243 WatchSource:0}: Error finding container ae452def15e05fb8e6a33d9a71ccc4bc351718be81515fb06f8fa5db18b34243: Status 404 returned error can't find the container with id ae452def15e05fb8e6a33d9a71ccc4bc351718be81515fb06f8fa5db18b34243 Aug 04 11:30:00.863489 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.863447 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ed8ca8-29764050-7vxtf" event={"ID":"cd3d8bf7-954d-44b3-b08e-f250ddeed083","Type":"ContainerStarted","Data":"aac725c9cc8cc164acb723ae08c72902d4d1b464ad2c6fc215db8e6c44cbcff9"} Aug 04 11:30:00.863705 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.863526 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ed8ca8-29764050-7vxtf" event={"ID":"cd3d8bf7-954d-44b3-b08e-f250ddeed083","Type":"ContainerStarted","Data":"cb827bc16832e58381089a0604b61e7ad66ceae541f555fbf9a9b098cb00c13b"} Aug 04 11:30:00.865152 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.865120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764050-xv6m5" event={"ID":"d7380efc-86ca-45c0-98ea-21fb4961d7fc","Type":"ContainerStarted","Data":"4eac532b2bb4444492af873fe43b6258a46627ddd156072f95f8ea98a439f4d7"} Aug 04 11:30:00.865298 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.865177 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764050-xv6m5" event={"ID":"d7380efc-86ca-45c0-98ea-21fb4961d7fc","Type":"ContainerStarted","Data":"ae452def15e05fb8e6a33d9a71ccc4bc351718be81515fb06f8fa5db18b34243"} Aug 04 11:30:00.866739 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.866712 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c74755-29764050-8msxn" event={"ID":"c8189562-2ec7-41f6-a500-b720098b0759","Type":"ContainerStarted","Data":"2e7a254ee6af4a7a5d50e6fb0704e2bd1eb199761e38d261d0281916f07d9661"} Aug 04 11:30:00.866853 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.866758 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c74755-29764050-8msxn" event={"ID":"c8189562-2ec7-41f6-a500-b720098b0759","Type":"ContainerStarted","Data":"92409b8b38e5e40c7a7f45c3e0953018dfbc653acc3c3b1177f67f597153d8c0"} Aug 04 11:30:00.878607 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.878454 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ed8ca8-29764050-7vxtf" podStartSLOduration=0.878436046 podStartE2EDuration="878.436046ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:30:00.878341855 +0000 UTC m=+3189.976135245" watchObservedRunningTime="2026-08-04 11:30:00.878436046 +0000 UTC m=+3189.976229443" Aug 04 11:30:00.895280 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.895177 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c74755-29764050-8msxn" podStartSLOduration=0.895160121 podStartE2EDuration="895.160121ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:30:00.892857453 +0000 UTC m=+3189.990650842" watchObservedRunningTime="2026-08-04 11:30:00.895160121 +0000 UTC m=+3189.992953514" Aug 04 11:30:00.907992 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:00.907939 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764050-xv6m5" podStartSLOduration=0.907921421 podStartE2EDuration="907.921421ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:30:00.906270239 +0000 UTC m=+3190.004063628" watchObservedRunningTime="2026-08-04 11:30:00.907921421 +0000 UTC m=+3190.005714838" Aug 04 11:30:01.878623 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:01.878591 2576 generic.go:356] "Generic (PLEG): container finished" podID="c8189562-2ec7-41f6-a500-b720098b0759" containerID="2e7a254ee6af4a7a5d50e6fb0704e2bd1eb199761e38d261d0281916f07d9661" exitCode=0 Aug 04 11:30:01.879014 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:01.878682 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c74755-29764050-8msxn" event={"ID":"c8189562-2ec7-41f6-a500-b720098b0759","Type":"ContainerDied","Data":"2e7a254ee6af4a7a5d50e6fb0704e2bd1eb199761e38d261d0281916f07d9661"} Aug 04 11:30:01.880290 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:01.880262 2576 generic.go:356] "Generic (PLEG): container finished" podID="cd3d8bf7-954d-44b3-b08e-f250ddeed083" containerID="aac725c9cc8cc164acb723ae08c72902d4d1b464ad2c6fc215db8e6c44cbcff9" exitCode=0 Aug 04 11:30:01.880415 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:01.880343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ed8ca8-29764050-7vxtf" event={"ID":"cd3d8bf7-954d-44b3-b08e-f250ddeed083","Type":"ContainerDied","Data":"aac725c9cc8cc164acb723ae08c72902d4d1b464ad2c6fc215db8e6c44cbcff9"} Aug 04 11:30:01.881939 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:01.881915 2576 generic.go:356] "Generic (PLEG): container finished" podID="d7380efc-86ca-45c0-98ea-21fb4961d7fc" containerID="4eac532b2bb4444492af873fe43b6258a46627ddd156072f95f8ea98a439f4d7" exitCode=0 Aug 04 11:30:01.882060 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:01.881952 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764050-xv6m5" event={"ID":"d7380efc-86ca-45c0-98ea-21fb4961d7fc","Type":"ContainerDied","Data":"4eac532b2bb4444492af873fe43b6258a46627ddd156072f95f8ea98a439f4d7"} Aug 04 11:30:03.036431 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:03.036400 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ed8ca8-29764050-7vxtf" Aug 04 11:30:03.073926 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:03.073894 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c74755-29764050-8msxn" Aug 04 11:30:03.076993 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:03.076972 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764050-xv6m5" Aug 04 11:30:03.894819 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:03.894792 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ed8ca8-29764050-7vxtf" Aug 04 11:30:03.894974 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:03.894828 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ed8ca8-29764050-7vxtf" event={"ID":"cd3d8bf7-954d-44b3-b08e-f250ddeed083","Type":"ContainerDied","Data":"cb827bc16832e58381089a0604b61e7ad66ceae541f555fbf9a9b098cb00c13b"} Aug 04 11:30:03.894974 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:03.894905 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb827bc16832e58381089a0604b61e7ad66ceae541f555fbf9a9b098cb00c13b" Aug 04 11:30:03.896136 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:03.896116 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764050-xv6m5" Aug 04 11:30:03.896136 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:03.896122 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764050-xv6m5" event={"ID":"d7380efc-86ca-45c0-98ea-21fb4961d7fc","Type":"ContainerDied","Data":"ae452def15e05fb8e6a33d9a71ccc4bc351718be81515fb06f8fa5db18b34243"} Aug 04 11:30:03.896303 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:03.896165 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae452def15e05fb8e6a33d9a71ccc4bc351718be81515fb06f8fa5db18b34243" Aug 04 11:30:03.897553 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:03.897525 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c74755-29764050-8msxn" Aug 04 11:30:03.897553 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:03.897542 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c74755-29764050-8msxn" event={"ID":"c8189562-2ec7-41f6-a500-b720098b0759","Type":"ContainerDied","Data":"92409b8b38e5e40c7a7f45c3e0953018dfbc653acc3c3b1177f67f597153d8c0"} Aug 04 11:30:03.897722 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:03.897583 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92409b8b38e5e40c7a7f45c3e0953018dfbc653acc3c3b1177f67f597153d8c0" Aug 04 11:30:10.088451 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:10.088410 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:30:20.683033 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:20.682993 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:30:36.383129 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:36.383095 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:30:51.877256 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:51.877160 2576 scope.go:117] "RemoveContainer" containerID="8c2516ac27492555408d0f75b0df4b707dd2477f249a897bbf18914607729064" Aug 04 11:30:51.879458 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:51.879435 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:30:55.980123 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:55.980094 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:30:56.693362 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:56.693328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:30:58.596483 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:30:58.596457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:31:01.983105 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:01.983065 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:31:08.883050 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.883016 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb"] Aug 04 11:31:08.883461 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.883411 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8189562-2ec7-41f6-a500-b720098b0759" containerName="cleanup" Aug 04 11:31:08.883461 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.883421 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8189562-2ec7-41f6-a500-b720098b0759" containerName="cleanup" Aug 04 11:31:08.883461 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.883436 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7380efc-86ca-45c0-98ea-21fb4961d7fc" containerName="cleanup" Aug 04 11:31:08.883461 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.883442 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7380efc-86ca-45c0-98ea-21fb4961d7fc" containerName="cleanup" Aug 04 11:31:08.883461 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.883456 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3d8bf7-954d-44b3-b08e-f250ddeed083" containerName="cleanup" Aug 04 11:31:08.883461 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.883463 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3d8bf7-954d-44b3-b08e-f250ddeed083" containerName="cleanup" Aug 04 11:31:08.883694 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.883543 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3d8bf7-954d-44b3-b08e-f250ddeed083" containerName="cleanup" Aug 04 11:31:08.883694 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.883551 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7380efc-86ca-45c0-98ea-21fb4961d7fc" containerName="cleanup" Aug 04 11:31:08.883694 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.883559 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8189562-2ec7-41f6-a500-b720098b0759" containerName="cleanup" Aug 04 11:31:08.886616 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.886599 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:08.888879 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.888855 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ed8ca8\"/\"openshift-service-ca.crt\"" Aug 04 11:31:08.889020 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.888890 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ed8ca8\"/\"default-dockercfg-855th\"" Aug 04 11:31:08.889779 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.889760 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ed8ca8\"/\"kube-root-ca.crt\"" Aug 04 11:31:08.889898 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.889761 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ed8ca8\"/\"sub-test-model-ed8ca8-kserve-self-signed-certs\"" Aug 04 11:31:08.894941 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.894915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:31:08.897044 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:08.897022 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb"] Aug 04 11:31:09.005806 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.005769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-tls-certs\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.005994 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.005840 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2l5l\" (UniqueName: \"kubernetes.io/projected/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-kube-api-access-z2l5l\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.005994 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.005897 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-tmp-dir\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.005994 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.005928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-model-cache\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.005994 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.005964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-dshm\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.005994 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.005984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-home\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.107307 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.107262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2l5l\" (UniqueName: \"kubernetes.io/projected/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-kube-api-access-z2l5l\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.107307 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.107303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-tmp-dir\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.107602 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.107324 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-model-cache\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.107602 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.107366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-dshm\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.107602 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.107393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-home\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.107602 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.107529 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-tls-certs\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.107819 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.107793 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-tmp-dir\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.107819 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.107811 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-home\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.107934 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.107846 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-model-cache\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.109813 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.109789 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-dshm\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.109978 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.109961 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-tls-certs\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.116171 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.116143 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2l5l\" (UniqueName: \"kubernetes.io/projected/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-kube-api-access-z2l5l\") pod \"sub-test-model-ed8ca8-kserve-78f7c46965-kkslb\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.196200 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.196110 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:09.336677 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:09.336648 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb"] Aug 04 11:31:09.338638 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:31:09.338610 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdfb0e1c_3be6_4ae1_8fda_cb8fae3b52cf.slice/crio-6c507dc06646c89c9284161ad3d48a6207131e8c9c39663f80670225ac8d8d17 WatchSource:0}: Error finding container 6c507dc06646c89c9284161ad3d48a6207131e8c9c39663f80670225ac8d8d17: Status 404 returned error can't find the container with id 6c507dc06646c89c9284161ad3d48a6207131e8c9c39663f80670225ac8d8d17 Aug 04 11:31:10.301470 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:10.301431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" event={"ID":"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf","Type":"ContainerStarted","Data":"b7385d25b1786978d2baff83fb9f0c4ffe24c9d3d2fcc8022cc732b0b4104b6e"} Aug 04 11:31:10.301470 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:10.301477 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" event={"ID":"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf","Type":"ContainerStarted","Data":"6c507dc06646c89c9284161ad3d48a6207131e8c9c39663f80670225ac8d8d17"} Aug 04 11:31:10.320112 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:10.320058 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" podStartSLOduration=2.320038716 podStartE2EDuration="2.320038716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:31:10.318032425 +0000 UTC m=+3259.415825818" watchObservedRunningTime="2026-08-04 11:31:10.320038716 +0000 UTC m=+3259.417832138" Aug 04 11:31:11.481953 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:11.481920 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:31:19.197110 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:19.197073 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:19.197593 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:19.197225 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:19.209751 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:19.209723 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:19.365451 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:19.365422 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:31:50.161475 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:50.161434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:31:51.900056 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:51.900022 2576 scope.go:117] "RemoveContainer" containerID="d37d896c83103255756fb6b76ca2c2ef72ec8d89322194af7f926597c42f2e87" Aug 04 11:31:52.784048 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:31:52.784007 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:32:29.382030 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:32:29.381946 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:32:41.794031 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:32:41.793987 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:32:53.082744 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:32:53.082709 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:32:57.451235 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:32:57.451194 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb"] Aug 04 11:32:58.295043 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:32:58.295013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:32:59.743205 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:32:59.743175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:33:03.991579 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:03.991538 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:33:10.800203 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:10.800149 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:33:12.470396 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.470359 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" podUID="fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf" containerName="main" containerID="cri-o://b7385d25b1786978d2baff83fb9f0c4ffe24c9d3d2fcc8022cc732b0b4104b6e" gracePeriod=45 Aug 04 11:33:12.718074 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.718043 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:33:12.752970 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.752902 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-tmp-dir\") pod \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " Aug 04 11:33:12.753124 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.752972 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-dshm\") pod \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " Aug 04 11:33:12.753124 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.753006 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-tls-certs\") pod \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " Aug 04 11:33:12.753124 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.753049 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2l5l\" (UniqueName: \"kubernetes.io/projected/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-kube-api-access-z2l5l\") pod \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " Aug 04 11:33:12.753124 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.753080 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-home\") pod \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " Aug 04 11:33:12.753340 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.753153 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf" (UID: "fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:33:12.753340 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.753208 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-model-cache\") pod \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\" (UID: \"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf\") " Aug 04 11:33:12.753340 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.753305 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-home" (OuterVolumeSpecName: "home") pod "fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf" (UID: "fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:33:12.753647 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.753603 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-tmp-dir\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:33:12.753647 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.753609 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf" (UID: "fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:33:12.753647 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.753627 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-home\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:33:12.755192 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.755167 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-dshm" (OuterVolumeSpecName: "dshm") pod "fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf" (UID: "fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:33:12.755576 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.755553 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf" (UID: "fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:33:12.755639 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.755583 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-kube-api-access-z2l5l" (OuterVolumeSpecName: "kube-api-access-z2l5l") pod "fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf" (UID: "fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf"). InnerVolumeSpecName "kube-api-access-z2l5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:33:12.854287 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.854246 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-dshm\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:33:12.854287 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.854279 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-tls-certs\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:33:12.854287 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.854292 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2l5l\" (UniqueName: \"kubernetes.io/projected/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-kube-api-access-z2l5l\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:33:12.854561 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:12.854302 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf-model-cache\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:33:13.028511 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:13.028413 2576 generic.go:356] "Generic (PLEG): container finished" podID="fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf" containerID="b7385d25b1786978d2baff83fb9f0c4ffe24c9d3d2fcc8022cc732b0b4104b6e" exitCode=0 Aug 04 11:33:13.028511 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:13.028453 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" Aug 04 11:33:13.028728 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:13.028512 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" event={"ID":"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf","Type":"ContainerDied","Data":"b7385d25b1786978d2baff83fb9f0c4ffe24c9d3d2fcc8022cc732b0b4104b6e"} Aug 04 11:33:13.028728 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:13.028559 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb" event={"ID":"fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf","Type":"ContainerDied","Data":"6c507dc06646c89c9284161ad3d48a6207131e8c9c39663f80670225ac8d8d17"} Aug 04 11:33:13.028728 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:13.028582 2576 scope.go:117] "RemoveContainer" containerID="b7385d25b1786978d2baff83fb9f0c4ffe24c9d3d2fcc8022cc732b0b4104b6e" Aug 04 11:33:13.040320 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:13.040306 2576 scope.go:117] "RemoveContainer" containerID="b7385d25b1786978d2baff83fb9f0c4ffe24c9d3d2fcc8022cc732b0b4104b6e" Aug 04 11:33:13.040580 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:33:13.040561 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7385d25b1786978d2baff83fb9f0c4ffe24c9d3d2fcc8022cc732b0b4104b6e\": container with ID starting with b7385d25b1786978d2baff83fb9f0c4ffe24c9d3d2fcc8022cc732b0b4104b6e not found: ID does not exist" containerID="b7385d25b1786978d2baff83fb9f0c4ffe24c9d3d2fcc8022cc732b0b4104b6e" Aug 04 11:33:13.040634 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:13.040589 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7385d25b1786978d2baff83fb9f0c4ffe24c9d3d2fcc8022cc732b0b4104b6e"} err="failed to get container status \"b7385d25b1786978d2baff83fb9f0c4ffe24c9d3d2fcc8022cc732b0b4104b6e\": rpc error: code = NotFound desc = could not find container \"b7385d25b1786978d2baff83fb9f0c4ffe24c9d3d2fcc8022cc732b0b4104b6e\": container with ID starting with b7385d25b1786978d2baff83fb9f0c4ffe24c9d3d2fcc8022cc732b0b4104b6e not found: ID does not exist" Aug 04 11:33:13.066238 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:13.066215 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb"] Aug 04 11:33:13.073887 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:13.073868 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ed8ca8/sub-test-model-ed8ca8-kserve-78f7c46965-kkslb"] Aug 04 11:33:13.438344 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:13.438312 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf" path="/var/lib/kubelet/pods/fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf/volumes" Aug 04 11:33:13.498399 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:13.498368 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:33:42.032148 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:42.032111 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:33:42.094213 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:42.094174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:33:48.900778 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:48.900668 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:33:51.386582 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:33:51.386548 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:34:11.287448 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:34:11.287406 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:34:11.387727 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:34:11.387689 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:34:33.206199 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:34:33.206168 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:34:35.212143 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:34:35.212108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:34:36.192894 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:34:36.192859 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:34:48.084411 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:34:48.084368 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:35:15.686559 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:15.686422 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:35:21.329048 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:21.329012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:35:22.072538 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:22.072495 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:35:22.982326 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:22.982287 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7401-140224698048176"] Aug 04 11:35:22.982763 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:22.982748 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf" containerName="main" Aug 04 11:35:22.982763 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:22.982764 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf" containerName="main" Aug 04 11:35:22.982838 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:22.982833 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdfb0e1c-3be6-4ae1-8fda-cb8fae3b52cf" containerName="main" Aug 04 11:35:22.986147 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:22.986129 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698048176" Aug 04 11:35:22.988625 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:22.988600 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2w6pj\"" Aug 04 11:35:22.992957 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:22.992929 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-140224698048176"] Aug 04 11:35:23.060255 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:23.060225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9kxd\" (UniqueName: \"kubernetes.io/projected/7ca530c3-5d58-40df-a444-feb1635764f6-kube-api-access-m9kxd\") pod \"test-curl-7401-140224698048176\" (UID: \"7ca530c3-5d58-40df-a444-feb1635764f6\") " pod="opendatahub/test-curl-7401-140224698048176" Aug 04 11:35:23.161105 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:23.161074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9kxd\" (UniqueName: \"kubernetes.io/projected/7ca530c3-5d58-40df-a444-feb1635764f6-kube-api-access-m9kxd\") pod \"test-curl-7401-140224698048176\" (UID: \"7ca530c3-5d58-40df-a444-feb1635764f6\") " pod="opendatahub/test-curl-7401-140224698048176" Aug 04 11:35:23.169546 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:23.169498 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9kxd\" (UniqueName: \"kubernetes.io/projected/7ca530c3-5d58-40df-a444-feb1635764f6-kube-api-access-m9kxd\") pod \"test-curl-7401-140224698048176\" (UID: \"7ca530c3-5d58-40df-a444-feb1635764f6\") " pod="opendatahub/test-curl-7401-140224698048176" Aug 04 11:35:23.297705 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:23.297624 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698048176" Aug 04 11:35:23.439700 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:23.439670 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-140224698048176"] Aug 04 11:35:23.440341 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:35:23.440320 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ca530c3_5d58_40df_a444_feb1635764f6.slice/crio-1ec2e796d78c97f97184edf9aa4cb0e87774894e78c98c05875520847e4615fe WatchSource:0}: Error finding container 1ec2e796d78c97f97184edf9aa4cb0e87774894e78c98c05875520847e4615fe: Status 404 returned error can't find the container with id 1ec2e796d78c97f97184edf9aa4cb0e87774894e78c98c05875520847e4615fe Aug 04 11:35:23.441703 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:23.441682 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 11:35:23.793249 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:23.793212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-140224698048176" event={"ID":"7ca530c3-5d58-40df-a444-feb1635764f6","Type":"ContainerStarted","Data":"1ec2e796d78c97f97184edf9aa4cb0e87774894e78c98c05875520847e4615fe"} Aug 04 11:35:24.800153 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:24.800119 2576 generic.go:356] "Generic (PLEG): container finished" podID="7ca530c3-5d58-40df-a444-feb1635764f6" containerID="5c634d7826421d744a8b8b253a0e5da6f23cba9a6beb7bc8f776ab0eace7e2bc" exitCode=0 Aug 04 11:35:24.800650 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:24.800174 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-140224698048176" event={"ID":"7ca530c3-5d58-40df-a444-feb1635764f6","Type":"ContainerDied","Data":"5c634d7826421d744a8b8b253a0e5da6f23cba9a6beb7bc8f776ab0eace7e2bc"} Aug 04 11:35:25.946157 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:25.946121 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698048176" Aug 04 11:35:25.967496 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:25.967465 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7401-140224698048176_7ca530c3-5d58-40df-a444-feb1635764f6/test-curl-7401-140224698048176/0.log" Aug 04 11:35:25.988787 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:25.988757 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9kxd\" (UniqueName: \"kubernetes.io/projected/7ca530c3-5d58-40df-a444-feb1635764f6-kube-api-access-m9kxd\") pod \"7ca530c3-5d58-40df-a444-feb1635764f6\" (UID: \"7ca530c3-5d58-40df-a444-feb1635764f6\") " Aug 04 11:35:25.990542 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:25.990515 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7401-140224698048176"] Aug 04 11:35:25.991045 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:25.991016 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca530c3-5d58-40df-a444-feb1635764f6-kube-api-access-m9kxd" (OuterVolumeSpecName: "kube-api-access-m9kxd") pod "7ca530c3-5d58-40df-a444-feb1635764f6" (UID: "7ca530c3-5d58-40df-a444-feb1635764f6"). InnerVolumeSpecName "kube-api-access-m9kxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:35:25.996413 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:25.996379 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7401-140224698048176"] Aug 04 11:35:26.090464 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.090433 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9kxd\" (UniqueName: \"kubernetes.io/projected/7ca530c3-5d58-40df-a444-feb1635764f6-kube-api-access-m9kxd\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:35:26.140599 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.140563 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7401-140224698038064"] Aug 04 11:35:26.140974 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.140961 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ca530c3-5d58-40df-a444-feb1635764f6" containerName="test-curl-7401-140224698048176" Aug 04 11:35:26.141016 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.140976 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ca530c3-5d58-40df-a444-feb1635764f6" containerName="test-curl-7401-140224698048176" Aug 04 11:35:26.141078 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.141069 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ca530c3-5d58-40df-a444-feb1635764f6" containerName="test-curl-7401-140224698048176" Aug 04 11:35:26.144260 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.144236 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698038064" Aug 04 11:35:26.152754 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.152728 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-140224698038064"] Aug 04 11:35:26.191469 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.191429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnpzr\" (UniqueName: \"kubernetes.io/projected/90800835-2e47-470b-90b5-a63bdf9cd958-kube-api-access-wnpzr\") pod \"test-curl-7401-140224698038064\" (UID: \"90800835-2e47-470b-90b5-a63bdf9cd958\") " pod="opendatahub/test-curl-7401-140224698038064" Aug 04 11:35:26.292144 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.292100 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnpzr\" (UniqueName: \"kubernetes.io/projected/90800835-2e47-470b-90b5-a63bdf9cd958-kube-api-access-wnpzr\") pod \"test-curl-7401-140224698038064\" (UID: \"90800835-2e47-470b-90b5-a63bdf9cd958\") " pod="opendatahub/test-curl-7401-140224698038064" Aug 04 11:35:26.300794 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.300764 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnpzr\" (UniqueName: \"kubernetes.io/projected/90800835-2e47-470b-90b5-a63bdf9cd958-kube-api-access-wnpzr\") pod \"test-curl-7401-140224698038064\" (UID: \"90800835-2e47-470b-90b5-a63bdf9cd958\") " pod="opendatahub/test-curl-7401-140224698038064" Aug 04 11:35:26.455004 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.454917 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698038064" Aug 04 11:35:26.594384 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.594360 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-140224698038064"] Aug 04 11:35:26.596060 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:35:26.596030 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90800835_2e47_470b_90b5_a63bdf9cd958.slice/crio-88d2c85a91ca3d4f8cbe8b209bf09f51028dd13c5ccb28850eac77c532ebe27d WatchSource:0}: Error finding container 88d2c85a91ca3d4f8cbe8b209bf09f51028dd13c5ccb28850eac77c532ebe27d: Status 404 returned error can't find the container with id 88d2c85a91ca3d4f8cbe8b209bf09f51028dd13c5ccb28850eac77c532ebe27d Aug 04 11:35:26.815327 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.815246 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698048176" Aug 04 11:35:26.815327 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.815287 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ec2e796d78c97f97184edf9aa4cb0e87774894e78c98c05875520847e4615fe" Aug 04 11:35:26.816423 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:26.816398 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-140224698038064" event={"ID":"90800835-2e47-470b-90b5-a63bdf9cd958","Type":"ContainerStarted","Data":"88d2c85a91ca3d4f8cbe8b209bf09f51028dd13c5ccb28850eac77c532ebe27d"} Aug 04 11:35:26.826038 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:35:26.826009 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7401-140224698048176\" is forbidden: User \"system:node:ip-10-0-131-159.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-159.ec2.internal' and this object" podUID="7ca530c3-5d58-40df-a444-feb1635764f6" pod="opendatahub/test-curl-7401-140224698048176" Aug 04 11:35:27.216285 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:27.216248 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:35:27.444227 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:27.444193 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ca530c3-5d58-40df-a444-feb1635764f6" path="/var/lib/kubelet/pods/7ca530c3-5d58-40df-a444-feb1635764f6/volumes" Aug 04 11:35:27.823286 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:27.823250 2576 generic.go:356] "Generic (PLEG): container finished" podID="90800835-2e47-470b-90b5-a63bdf9cd958" containerID="ff0155acba3abe54c342ddcf1014b3cfe89eeb54d08fc3216ea71108e75e82c6" exitCode=0 Aug 04 11:35:27.823455 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:27.823341 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-140224698038064" event={"ID":"90800835-2e47-470b-90b5-a63bdf9cd958","Type":"ContainerDied","Data":"ff0155acba3abe54c342ddcf1014b3cfe89eeb54d08fc3216ea71108e75e82c6"} Aug 04 11:35:28.967195 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:28.967166 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698038064" Aug 04 11:35:28.987736 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:28.987606 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7401-140224698038064_90800835-2e47-470b-90b5-a63bdf9cd958/test-curl-7401-140224698038064/0.log" Aug 04 11:35:29.011609 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.011581 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7401-140224698038064"] Aug 04 11:35:29.016379 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.016353 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7401-140224698038064"] Aug 04 11:35:29.017959 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.017939 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnpzr\" (UniqueName: \"kubernetes.io/projected/90800835-2e47-470b-90b5-a63bdf9cd958-kube-api-access-wnpzr\") pod \"90800835-2e47-470b-90b5-a63bdf9cd958\" (UID: \"90800835-2e47-470b-90b5-a63bdf9cd958\") " Aug 04 11:35:29.019930 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.019906 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90800835-2e47-470b-90b5-a63bdf9cd958-kube-api-access-wnpzr" (OuterVolumeSpecName: "kube-api-access-wnpzr") pod "90800835-2e47-470b-90b5-a63bdf9cd958" (UID: "90800835-2e47-470b-90b5-a63bdf9cd958"). InnerVolumeSpecName "kube-api-access-wnpzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:35:29.119189 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.119149 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnpzr\" (UniqueName: \"kubernetes.io/projected/90800835-2e47-470b-90b5-a63bdf9cd958-kube-api-access-wnpzr\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:35:29.172742 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.172704 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7401-140224698113312"] Aug 04 11:35:29.173111 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.173098 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90800835-2e47-470b-90b5-a63bdf9cd958" containerName="test-curl-7401-140224698038064" Aug 04 11:35:29.173154 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.173113 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90800835-2e47-470b-90b5-a63bdf9cd958" containerName="test-curl-7401-140224698038064" Aug 04 11:35:29.173208 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.173199 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="90800835-2e47-470b-90b5-a63bdf9cd958" containerName="test-curl-7401-140224698038064" Aug 04 11:35:29.176271 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.176255 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698113312" Aug 04 11:35:29.182957 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.182924 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-140224698113312"] Aug 04 11:35:29.220432 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.220399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7gsd\" (UniqueName: \"kubernetes.io/projected/60722e4c-043a-4abc-82ae-eba14a7b1a5d-kube-api-access-p7gsd\") pod \"test-curl-7401-140224698113312\" (UID: \"60722e4c-043a-4abc-82ae-eba14a7b1a5d\") " pod="opendatahub/test-curl-7401-140224698113312" Aug 04 11:35:29.321607 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.321560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7gsd\" (UniqueName: \"kubernetes.io/projected/60722e4c-043a-4abc-82ae-eba14a7b1a5d-kube-api-access-p7gsd\") pod \"test-curl-7401-140224698113312\" (UID: \"60722e4c-043a-4abc-82ae-eba14a7b1a5d\") " pod="opendatahub/test-curl-7401-140224698113312" Aug 04 11:35:29.330455 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.330425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7gsd\" (UniqueName: \"kubernetes.io/projected/60722e4c-043a-4abc-82ae-eba14a7b1a5d-kube-api-access-p7gsd\") pod \"test-curl-7401-140224698113312\" (UID: \"60722e4c-043a-4abc-82ae-eba14a7b1a5d\") " pod="opendatahub/test-curl-7401-140224698113312" Aug 04 11:35:29.437937 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.437843 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90800835-2e47-470b-90b5-a63bdf9cd958" path="/var/lib/kubelet/pods/90800835-2e47-470b-90b5-a63bdf9cd958/volumes" Aug 04 11:35:29.487196 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.487146 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698113312" Aug 04 11:35:29.626576 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.626552 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-140224698113312"] Aug 04 11:35:29.628200 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:35:29.628173 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60722e4c_043a_4abc_82ae_eba14a7b1a5d.slice/crio-97d347c55e3e259b2687f59f5dadbdf30f087e66d28d6ab77fd5a17483649824 WatchSource:0}: Error finding container 97d347c55e3e259b2687f59f5dadbdf30f087e66d28d6ab77fd5a17483649824: Status 404 returned error can't find the container with id 97d347c55e3e259b2687f59f5dadbdf30f087e66d28d6ab77fd5a17483649824 Aug 04 11:35:29.836105 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.836082 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698038064" Aug 04 11:35:29.836105 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.836093 2576 scope.go:117] "RemoveContainer" containerID="ff0155acba3abe54c342ddcf1014b3cfe89eeb54d08fc3216ea71108e75e82c6" Aug 04 11:35:29.837472 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:29.837438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-140224698113312" event={"ID":"60722e4c-043a-4abc-82ae-eba14a7b1a5d","Type":"ContainerStarted","Data":"97d347c55e3e259b2687f59f5dadbdf30f087e66d28d6ab77fd5a17483649824"} Aug 04 11:35:30.844972 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:30.844941 2576 generic.go:356] "Generic (PLEG): container finished" podID="60722e4c-043a-4abc-82ae-eba14a7b1a5d" containerID="b931c5708c227d416522e860018879c4ac87d3bad39b749fceb042d14e53d423" exitCode=0 Aug 04 11:35:30.845357 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:30.845013 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-140224698113312" event={"ID":"60722e4c-043a-4abc-82ae-eba14a7b1a5d","Type":"ContainerDied","Data":"b931c5708c227d416522e860018879c4ac87d3bad39b749fceb042d14e53d423"} Aug 04 11:35:31.983348 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:31.983309 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698113312" Aug 04 11:35:32.006307 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.006283 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7401-140224698113312_60722e4c-043a-4abc-82ae-eba14a7b1a5d/test-curl-7401-140224698113312/0.log" Aug 04 11:35:32.027158 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.027120 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7401-140224698113312"] Aug 04 11:35:32.029470 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.029447 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7401-140224698113312"] Aug 04 11:35:32.045389 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.045359 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7gsd\" (UniqueName: \"kubernetes.io/projected/60722e4c-043a-4abc-82ae-eba14a7b1a5d-kube-api-access-p7gsd\") pod \"60722e4c-043a-4abc-82ae-eba14a7b1a5d\" (UID: \"60722e4c-043a-4abc-82ae-eba14a7b1a5d\") " Aug 04 11:35:32.047479 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.047453 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60722e4c-043a-4abc-82ae-eba14a7b1a5d-kube-api-access-p7gsd" (OuterVolumeSpecName: "kube-api-access-p7gsd") pod "60722e4c-043a-4abc-82ae-eba14a7b1a5d" (UID: "60722e4c-043a-4abc-82ae-eba14a7b1a5d"). InnerVolumeSpecName "kube-api-access-p7gsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:35:32.146176 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.146088 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7gsd\" (UniqueName: \"kubernetes.io/projected/60722e4c-043a-4abc-82ae-eba14a7b1a5d-kube-api-access-p7gsd\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:35:32.150421 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.150392 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7401-140224698113312"] Aug 04 11:35:32.150826 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.150812 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60722e4c-043a-4abc-82ae-eba14a7b1a5d" containerName="test-curl-7401-140224698113312" Aug 04 11:35:32.150870 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.150828 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="60722e4c-043a-4abc-82ae-eba14a7b1a5d" containerName="test-curl-7401-140224698113312" Aug 04 11:35:32.150921 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.150912 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="60722e4c-043a-4abc-82ae-eba14a7b1a5d" containerName="test-curl-7401-140224698113312" Aug 04 11:35:32.153877 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.153860 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698113312" Aug 04 11:35:32.163073 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.163047 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-140224698113312"] Aug 04 11:35:32.246761 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.246719 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgcc7\" (UniqueName: \"kubernetes.io/projected/9a3bc981-ec0a-413d-9c61-c477b45c34e2-kube-api-access-pgcc7\") pod \"test-curl-7401-140224698113312\" (UID: \"9a3bc981-ec0a-413d-9c61-c477b45c34e2\") " pod="opendatahub/test-curl-7401-140224698113312" Aug 04 11:35:32.347612 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.347563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgcc7\" (UniqueName: \"kubernetes.io/projected/9a3bc981-ec0a-413d-9c61-c477b45c34e2-kube-api-access-pgcc7\") pod \"test-curl-7401-140224698113312\" (UID: \"9a3bc981-ec0a-413d-9c61-c477b45c34e2\") " pod="opendatahub/test-curl-7401-140224698113312" Aug 04 11:35:32.356496 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.356452 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgcc7\" (UniqueName: \"kubernetes.io/projected/9a3bc981-ec0a-413d-9c61-c477b45c34e2-kube-api-access-pgcc7\") pod \"test-curl-7401-140224698113312\" (UID: \"9a3bc981-ec0a-413d-9c61-c477b45c34e2\") " pod="opendatahub/test-curl-7401-140224698113312" Aug 04 11:35:32.464760 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.464653 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698113312" Aug 04 11:35:32.615388 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.615359 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-140224698113312"] Aug 04 11:35:32.617008 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:35:32.616984 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a3bc981_ec0a_413d_9c61_c477b45c34e2.slice/crio-731fbf8cf9c7fa9c4a9dd88abacba7a0a3fb3a842c0966aa5248062c591ffa1d WatchSource:0}: Error finding container 731fbf8cf9c7fa9c4a9dd88abacba7a0a3fb3a842c0966aa5248062c591ffa1d: Status 404 returned error can't find the container with id 731fbf8cf9c7fa9c4a9dd88abacba7a0a3fb3a842c0966aa5248062c591ffa1d Aug 04 11:35:32.857542 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.857493 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-140224698113312" event={"ID":"9a3bc981-ec0a-413d-9c61-c477b45c34e2","Type":"ContainerStarted","Data":"731fbf8cf9c7fa9c4a9dd88abacba7a0a3fb3a842c0966aa5248062c591ffa1d"} Aug 04 11:35:32.858729 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.858706 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698113312" Aug 04 11:35:32.858869 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.858725 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97d347c55e3e259b2687f59f5dadbdf30f087e66d28d6ab77fd5a17483649824" Aug 04 11:35:32.870697 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:32.870666 2576 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7401-140224698113312" oldPodUID="60722e4c-043a-4abc-82ae-eba14a7b1a5d" podUID="9a3bc981-ec0a-413d-9c61-c477b45c34e2" Aug 04 11:35:33.438126 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:33.438090 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60722e4c-043a-4abc-82ae-eba14a7b1a5d" path="/var/lib/kubelet/pods/60722e4c-043a-4abc-82ae-eba14a7b1a5d/volumes" Aug 04 11:35:33.866762 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:33.866730 2576 generic.go:356] "Generic (PLEG): container finished" podID="9a3bc981-ec0a-413d-9c61-c477b45c34e2" containerID="0ec1969e7f84a4d9763862dbb3e5ed45e5f961868dc748f0b0407d81bfbfd40c" exitCode=0 Aug 04 11:35:33.866940 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:33.866822 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-140224698113312" event={"ID":"9a3bc981-ec0a-413d-9c61-c477b45c34e2","Type":"ContainerDied","Data":"0ec1969e7f84a4d9763862dbb3e5ed45e5f961868dc748f0b0407d81bfbfd40c"} Aug 04 11:35:35.005350 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.005326 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698113312" Aug 04 11:35:35.025953 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.025925 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7401-140224698113312_9a3bc981-ec0a-413d-9c61-c477b45c34e2/test-curl-7401-140224698113312/0.log" Aug 04 11:35:35.049677 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.049642 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7401-140224698113312"] Aug 04 11:35:35.053734 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.053706 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7401-140224698113312"] Aug 04 11:35:35.074188 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.074161 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgcc7\" (UniqueName: \"kubernetes.io/projected/9a3bc981-ec0a-413d-9c61-c477b45c34e2-kube-api-access-pgcc7\") pod \"9a3bc981-ec0a-413d-9c61-c477b45c34e2\" (UID: \"9a3bc981-ec0a-413d-9c61-c477b45c34e2\") " Aug 04 11:35:35.076226 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.076202 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3bc981-ec0a-413d-9c61-c477b45c34e2-kube-api-access-pgcc7" (OuterVolumeSpecName: "kube-api-access-pgcc7") pod "9a3bc981-ec0a-413d-9c61-c477b45c34e2" (UID: "9a3bc981-ec0a-413d-9c61-c477b45c34e2"). InnerVolumeSpecName "kube-api-access-pgcc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:35:35.171251 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.171165 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7401-140224697975120"] Aug 04 11:35:35.171588 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.171574 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a3bc981-ec0a-413d-9c61-c477b45c34e2" containerName="test-curl-7401-140224698113312" Aug 04 11:35:35.171639 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.171590 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a3bc981-ec0a-413d-9c61-c477b45c34e2" containerName="test-curl-7401-140224698113312" Aug 04 11:35:35.171677 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.171661 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a3bc981-ec0a-413d-9c61-c477b45c34e2" containerName="test-curl-7401-140224698113312" Aug 04 11:35:35.174802 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.174785 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224697975120" Aug 04 11:35:35.175034 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.175015 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgcc7\" (UniqueName: \"kubernetes.io/projected/9a3bc981-ec0a-413d-9c61-c477b45c34e2-kube-api-access-pgcc7\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:35:35.184322 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.184295 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-140224697975120"] Aug 04 11:35:35.275903 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.275868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzsmd\" (UniqueName: \"kubernetes.io/projected/4e8ae40e-658e-400f-a8da-0ea1eea5a13f-kube-api-access-vzsmd\") pod \"test-curl-7401-140224697975120\" (UID: \"4e8ae40e-658e-400f-a8da-0ea1eea5a13f\") " pod="opendatahub/test-curl-7401-140224697975120" Aug 04 11:35:35.377072 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.377034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzsmd\" (UniqueName: \"kubernetes.io/projected/4e8ae40e-658e-400f-a8da-0ea1eea5a13f-kube-api-access-vzsmd\") pod \"test-curl-7401-140224697975120\" (UID: \"4e8ae40e-658e-400f-a8da-0ea1eea5a13f\") " pod="opendatahub/test-curl-7401-140224697975120" Aug 04 11:35:35.385358 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.385330 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzsmd\" (UniqueName: \"kubernetes.io/projected/4e8ae40e-658e-400f-a8da-0ea1eea5a13f-kube-api-access-vzsmd\") pod \"test-curl-7401-140224697975120\" (UID: \"4e8ae40e-658e-400f-a8da-0ea1eea5a13f\") " pod="opendatahub/test-curl-7401-140224697975120" Aug 04 11:35:35.438134 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.438054 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a3bc981-ec0a-413d-9c61-c477b45c34e2" path="/var/lib/kubelet/pods/9a3bc981-ec0a-413d-9c61-c477b45c34e2/volumes" Aug 04 11:35:35.484557 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.484513 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224697975120" Aug 04 11:35:35.621658 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.621630 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-140224697975120"] Aug 04 11:35:35.622894 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:35:35.622867 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e8ae40e_658e_400f_a8da_0ea1eea5a13f.slice/crio-23eb0ba6969e305878da10f65354fb7b68ec4a275e25ec69ba67ff439fd3d2f4 WatchSource:0}: Error finding container 23eb0ba6969e305878da10f65354fb7b68ec4a275e25ec69ba67ff439fd3d2f4: Status 404 returned error can't find the container with id 23eb0ba6969e305878da10f65354fb7b68ec4a275e25ec69ba67ff439fd3d2f4 Aug 04 11:35:35.879443 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.879411 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224698113312" Aug 04 11:35:35.879634 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.879463 2576 scope.go:117] "RemoveContainer" containerID="0ec1969e7f84a4d9763862dbb3e5ed45e5f961868dc748f0b0407d81bfbfd40c" Aug 04 11:35:35.880931 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:35.880905 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-140224697975120" event={"ID":"4e8ae40e-658e-400f-a8da-0ea1eea5a13f","Type":"ContainerStarted","Data":"23eb0ba6969e305878da10f65354fb7b68ec4a275e25ec69ba67ff439fd3d2f4"} Aug 04 11:35:36.887847 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:36.887811 2576 generic.go:356] "Generic (PLEG): container finished" podID="4e8ae40e-658e-400f-a8da-0ea1eea5a13f" containerID="f9ddef73bc8767541d7dfcc7e746ae3b333ee82860e9780e985fa4ef286edf5c" exitCode=0 Aug 04 11:35:36.888251 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:36.887903 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-140224697975120" event={"ID":"4e8ae40e-658e-400f-a8da-0ea1eea5a13f","Type":"ContainerDied","Data":"f9ddef73bc8767541d7dfcc7e746ae3b333ee82860e9780e985fa4ef286edf5c"} Aug 04 11:35:38.031332 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.031306 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224697975120" Aug 04 11:35:38.052450 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.052423 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7401-140224697975120_4e8ae40e-658e-400f-a8da-0ea1eea5a13f/test-curl-7401-140224697975120/0.log" Aug 04 11:35:38.076153 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.076112 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7401-140224697975120"] Aug 04 11:35:38.078632 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.078605 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7401-140224697975120"] Aug 04 11:35:38.099950 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.099910 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzsmd\" (UniqueName: \"kubernetes.io/projected/4e8ae40e-658e-400f-a8da-0ea1eea5a13f-kube-api-access-vzsmd\") pod \"4e8ae40e-658e-400f-a8da-0ea1eea5a13f\" (UID: \"4e8ae40e-658e-400f-a8da-0ea1eea5a13f\") " Aug 04 11:35:38.102050 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.102007 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8ae40e-658e-400f-a8da-0ea1eea5a13f-kube-api-access-vzsmd" (OuterVolumeSpecName: "kube-api-access-vzsmd") pod "4e8ae40e-658e-400f-a8da-0ea1eea5a13f" (UID: "4e8ae40e-658e-400f-a8da-0ea1eea5a13f"). InnerVolumeSpecName "kube-api-access-vzsmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:35:38.197791 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.197709 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7401-140224697975120"] Aug 04 11:35:38.198128 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.198116 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e8ae40e-658e-400f-a8da-0ea1eea5a13f" containerName="test-curl-7401-140224697975120" Aug 04 11:35:38.198176 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.198130 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e8ae40e-658e-400f-a8da-0ea1eea5a13f" containerName="test-curl-7401-140224697975120" Aug 04 11:35:38.198215 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.198206 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e8ae40e-658e-400f-a8da-0ea1eea5a13f" containerName="test-curl-7401-140224697975120" Aug 04 11:35:38.200911 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.200886 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzsmd\" (UniqueName: \"kubernetes.io/projected/4e8ae40e-658e-400f-a8da-0ea1eea5a13f-kube-api-access-vzsmd\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:35:38.201338 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.201322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224697975120" Aug 04 11:35:38.207059 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.207030 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-140224697975120"] Aug 04 11:35:38.301636 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.301596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgd22\" (UniqueName: \"kubernetes.io/projected/e6272d62-cfd7-4694-82b7-b371ac6bb9d1-kube-api-access-pgd22\") pod \"test-curl-7401-140224697975120\" (UID: \"e6272d62-cfd7-4694-82b7-b371ac6bb9d1\") " pod="opendatahub/test-curl-7401-140224697975120" Aug 04 11:35:38.402529 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.402462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgd22\" (UniqueName: \"kubernetes.io/projected/e6272d62-cfd7-4694-82b7-b371ac6bb9d1-kube-api-access-pgd22\") pod \"test-curl-7401-140224697975120\" (UID: \"e6272d62-cfd7-4694-82b7-b371ac6bb9d1\") " pod="opendatahub/test-curl-7401-140224697975120" Aug 04 11:35:38.411055 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.411020 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgd22\" (UniqueName: \"kubernetes.io/projected/e6272d62-cfd7-4694-82b7-b371ac6bb9d1-kube-api-access-pgd22\") pod \"test-curl-7401-140224697975120\" (UID: \"e6272d62-cfd7-4694-82b7-b371ac6bb9d1\") " pod="opendatahub/test-curl-7401-140224697975120" Aug 04 11:35:38.512559 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.512437 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224697975120" Aug 04 11:35:38.648734 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.648709 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-140224697975120"] Aug 04 11:35:38.649866 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:35:38.649843 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6272d62_cfd7_4694_82b7_b371ac6bb9d1.slice/crio-89968ca68be15696eb6e36e8f085a87facd8e3346ce5f22e23fbe542101e7ced WatchSource:0}: Error finding container 89968ca68be15696eb6e36e8f085a87facd8e3346ce5f22e23fbe542101e7ced: Status 404 returned error can't find the container with id 89968ca68be15696eb6e36e8f085a87facd8e3346ce5f22e23fbe542101e7ced Aug 04 11:35:38.901763 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.901720 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-140224697975120" event={"ID":"e6272d62-cfd7-4694-82b7-b371ac6bb9d1","Type":"ContainerStarted","Data":"89968ca68be15696eb6e36e8f085a87facd8e3346ce5f22e23fbe542101e7ced"} Aug 04 11:35:38.902884 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.902856 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224697975120" Aug 04 11:35:38.902982 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.902900 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23eb0ba6969e305878da10f65354fb7b68ec4a275e25ec69ba67ff439fd3d2f4" Aug 04 11:35:38.914724 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:38.914696 2576 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7401-140224697975120" oldPodUID="4e8ae40e-658e-400f-a8da-0ea1eea5a13f" podUID="e6272d62-cfd7-4694-82b7-b371ac6bb9d1" Aug 04 11:35:39.438792 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:39.438760 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e8ae40e-658e-400f-a8da-0ea1eea5a13f" path="/var/lib/kubelet/pods/4e8ae40e-658e-400f-a8da-0ea1eea5a13f/volumes" Aug 04 11:35:39.909857 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:39.909825 2576 generic.go:356] "Generic (PLEG): container finished" podID="e6272d62-cfd7-4694-82b7-b371ac6bb9d1" containerID="4fee967b2e8c515c12894be9fd2c00252835ced52f78afbf363fe3f21d0ceff7" exitCode=0 Aug 04 11:35:39.910028 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:39.909907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-140224697975120" event={"ID":"e6272d62-cfd7-4694-82b7-b371ac6bb9d1","Type":"ContainerDied","Data":"4fee967b2e8c515c12894be9fd2c00252835ced52f78afbf363fe3f21d0ceff7"} Aug 04 11:35:41.050047 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:41.050021 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224697975120" Aug 04 11:35:41.070944 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:41.070901 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7401-140224697975120_e6272d62-cfd7-4694-82b7-b371ac6bb9d1/test-curl-7401-140224697975120/0.log" Aug 04 11:35:41.094302 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:41.094266 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7401-140224697975120"] Aug 04 11:35:41.098469 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:41.098442 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7401-140224697975120"] Aug 04 11:35:41.126594 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:41.126569 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgd22\" (UniqueName: \"kubernetes.io/projected/e6272d62-cfd7-4694-82b7-b371ac6bb9d1-kube-api-access-pgd22\") pod \"e6272d62-cfd7-4694-82b7-b371ac6bb9d1\" (UID: \"e6272d62-cfd7-4694-82b7-b371ac6bb9d1\") " Aug 04 11:35:41.128663 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:41.128632 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6272d62-cfd7-4694-82b7-b371ac6bb9d1-kube-api-access-pgd22" (OuterVolumeSpecName: "kube-api-access-pgd22") pod "e6272d62-cfd7-4694-82b7-b371ac6bb9d1" (UID: "e6272d62-cfd7-4694-82b7-b371ac6bb9d1"). InnerVolumeSpecName "kube-api-access-pgd22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:35:41.227917 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:41.227818 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgd22\" (UniqueName: \"kubernetes.io/projected/e6272d62-cfd7-4694-82b7-b371ac6bb9d1-kube-api-access-pgd22\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:35:41.439068 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:41.439030 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6272d62-cfd7-4694-82b7-b371ac6bb9d1" path="/var/lib/kubelet/pods/e6272d62-cfd7-4694-82b7-b371ac6bb9d1/volumes" Aug 04 11:35:41.619474 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:41.619441 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:35:41.922655 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:41.922575 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-140224697975120" Aug 04 11:35:41.922655 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:41.922609 2576 scope.go:117] "RemoveContainer" containerID="4fee967b2e8c515c12894be9fd2c00252835ced52f78afbf363fe3f21d0ceff7" Aug 04 11:35:41.948779 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:41.948749 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:35:42.058754 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:42.058715 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:35:45.223368 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:45.223331 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:35:45.316017 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:45.315984 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:35:46.311525 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.311481 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95"] Aug 04 11:35:46.311922 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.311908 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6272d62-cfd7-4694-82b7-b371ac6bb9d1" containerName="test-curl-7401-140224697975120" Aug 04 11:35:46.311968 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.311923 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6272d62-cfd7-4694-82b7-b371ac6bb9d1" containerName="test-curl-7401-140224697975120" Aug 04 11:35:46.312036 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.312025 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6272d62-cfd7-4694-82b7-b371ac6bb9d1" containerName="test-curl-7401-140224697975120" Aug 04 11:35:46.316384 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.316366 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:35:46.318820 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.318799 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e44431-serving-cert\"" Aug 04 11:35:46.326731 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.326707 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95"] Aug 04 11:35:46.347336 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.347310 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:35:46.378791 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.378757 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bfab7d0-b7f0-438a-8774-99e577355c5b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95\" (UID: \"7bfab7d0-b7f0-438a-8774-99e577355c5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:35:46.378791 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.378792 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bfab7d0-b7f0-438a-8774-99e577355c5b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95\" (UID: \"7bfab7d0-b7f0-438a-8774-99e577355c5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:35:46.378994 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.378880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5874n\" (UniqueName: \"kubernetes.io/projected/7bfab7d0-b7f0-438a-8774-99e577355c5b-kube-api-access-5874n\") pod \"maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95\" (UID: \"7bfab7d0-b7f0-438a-8774-99e577355c5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:35:46.410408 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.410382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:35:46.480376 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.480329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5874n\" (UniqueName: \"kubernetes.io/projected/7bfab7d0-b7f0-438a-8774-99e577355c5b-kube-api-access-5874n\") pod \"maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95\" (UID: \"7bfab7d0-b7f0-438a-8774-99e577355c5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:35:46.480567 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.480419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bfab7d0-b7f0-438a-8774-99e577355c5b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95\" (UID: \"7bfab7d0-b7f0-438a-8774-99e577355c5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:35:46.480567 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.480439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bfab7d0-b7f0-438a-8774-99e577355c5b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95\" (UID: \"7bfab7d0-b7f0-438a-8774-99e577355c5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:35:46.481106 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.481084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bfab7d0-b7f0-438a-8774-99e577355c5b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95\" (UID: \"7bfab7d0-b7f0-438a-8774-99e577355c5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:35:46.482837 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.482815 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bfab7d0-b7f0-438a-8774-99e577355c5b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95\" (UID: \"7bfab7d0-b7f0-438a-8774-99e577355c5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:35:46.488480 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.488449 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5874n\" (UniqueName: \"kubernetes.io/projected/7bfab7d0-b7f0-438a-8774-99e577355c5b-kube-api-access-5874n\") pod \"maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95\" (UID: \"7bfab7d0-b7f0-438a-8774-99e577355c5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:35:46.628246 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.628192 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:35:46.770022 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.769993 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95"] Aug 04 11:35:46.772167 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:35:46.772138 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bfab7d0_b7f0_438a_8774_99e577355c5b.slice/crio-7e66760db038f6fd3e7b789400989c3d8dce928aa248478d25b154f0c7a14008 WatchSource:0}: Error finding container 7e66760db038f6fd3e7b789400989c3d8dce928aa248478d25b154f0c7a14008: Status 404 returned error can't find the container with id 7e66760db038f6fd3e7b789400989c3d8dce928aa248478d25b154f0c7a14008 Aug 04 11:35:46.899968 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.899877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:35:46.955463 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:46.955430 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" event={"ID":"7bfab7d0-b7f0-438a-8774-99e577355c5b","Type":"ContainerStarted","Data":"7e66760db038f6fd3e7b789400989c3d8dce928aa248478d25b154f0c7a14008"} Aug 04 11:35:47.078041 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:47.078012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:35:47.963098 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:47.963056 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" event={"ID":"7bfab7d0-b7f0-438a-8774-99e577355c5b","Type":"ContainerStarted","Data":"bf3dbb3d6a6f3eee1c70eeb4df65b627be7f54eefff8c06c4c9e279cf0549b1b"} Aug 04 11:35:47.963518 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:47.963166 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:35:47.995210 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:47.995155 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" podStartSLOduration=1.585440825 podStartE2EDuration="1.995137614s" totalImagesPullingTime="409.696789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:35:47.992158193 +0000 UTC m=+3537.089951586" watchObservedRunningTime="2026-08-04 11:35:47.995137614 +0000 UTC m=+3537.092931096" Aug 04 11:35:49.193331 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:49.193284 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:35:53.974469 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:53.974424 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:35:59.942717 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:35:59.942685 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:36:00.588206 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:00.588161 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:36:00.957408 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:00.957323 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:36:01.430932 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:01.430904 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:36:04.401485 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:04.401438 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:36:04.514315 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:04.514283 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:36:06.370030 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.369994 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs"] Aug 04 11:36:06.373734 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.373712 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:36:06.376707 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.376683 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-29b2ff-serving-cert\"" Aug 04 11:36:06.385950 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.385924 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs"] Aug 04 11:36:06.452834 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.452800 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:36:06.463160 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.463133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d006cef-9618-4a04-a178-2cbb35d4a20c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs\" (UID: \"7d006cef-9618-4a04-a178-2cbb35d4a20c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:36:06.463264 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.463247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2nf2\" (UniqueName: \"kubernetes.io/projected/7d006cef-9618-4a04-a178-2cbb35d4a20c-kube-api-access-t2nf2\") pod \"maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs\" (UID: \"7d006cef-9618-4a04-a178-2cbb35d4a20c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:36:06.463303 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.463274 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d006cef-9618-4a04-a178-2cbb35d4a20c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs\" (UID: \"7d006cef-9618-4a04-a178-2cbb35d4a20c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:36:06.563793 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.563754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2nf2\" (UniqueName: \"kubernetes.io/projected/7d006cef-9618-4a04-a178-2cbb35d4a20c-kube-api-access-t2nf2\") pod \"maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs\" (UID: \"7d006cef-9618-4a04-a178-2cbb35d4a20c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:36:06.563793 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.563794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d006cef-9618-4a04-a178-2cbb35d4a20c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs\" (UID: \"7d006cef-9618-4a04-a178-2cbb35d4a20c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:36:06.564009 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.563836 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d006cef-9618-4a04-a178-2cbb35d4a20c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs\" (UID: \"7d006cef-9618-4a04-a178-2cbb35d4a20c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:36:06.564569 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.564542 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d006cef-9618-4a04-a178-2cbb35d4a20c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs\" (UID: \"7d006cef-9618-4a04-a178-2cbb35d4a20c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:36:06.566307 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.566277 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d006cef-9618-4a04-a178-2cbb35d4a20c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs\" (UID: \"7d006cef-9618-4a04-a178-2cbb35d4a20c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:36:06.573079 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.573057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2nf2\" (UniqueName: \"kubernetes.io/projected/7d006cef-9618-4a04-a178-2cbb35d4a20c-kube-api-access-t2nf2\") pod \"maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs\" (UID: \"7d006cef-9618-4a04-a178-2cbb35d4a20c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:36:06.611753 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.611722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:36:06.685983 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.685891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:36:06.829926 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.829901 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs"] Aug 04 11:36:06.830877 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:36:06.830850 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d006cef_9618_4a04_a178_2cbb35d4a20c.slice/crio-4df740f3584b3e6e86f7ddb3f35c6b05239d86c8d23369a6abd5fcc7a636e458 WatchSource:0}: Error finding container 4df740f3584b3e6e86f7ddb3f35c6b05239d86c8d23369a6abd5fcc7a636e458: Status 404 returned error can't find the container with id 4df740f3584b3e6e86f7ddb3f35c6b05239d86c8d23369a6abd5fcc7a636e458 Aug 04 11:36:06.952636 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:06.952550 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:36:07.085276 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:07.085244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:36:07.086842 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:07.086817 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" event={"ID":"7d006cef-9618-4a04-a178-2cbb35d4a20c","Type":"ContainerStarted","Data":"4df740f3584b3e6e86f7ddb3f35c6b05239d86c8d23369a6abd5fcc7a636e458"} Aug 04 11:36:08.093641 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:08.093606 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" event={"ID":"7d006cef-9618-4a04-a178-2cbb35d4a20c","Type":"ContainerStarted","Data":"e805ac2621ec89922baa02a414ad55d086eeba2e9e0bc50f104e0451586e84de"} Aug 04 11:36:08.094051 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:08.093703 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:36:08.117171 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:08.117122 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" podStartSLOduration=1.6079244959999999 podStartE2EDuration="2.117105169s" totalImagesPullingTime="509.180673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:36:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:36:08.116315587 +0000 UTC m=+3557.214108983" watchObservedRunningTime="2026-08-04 11:36:08.117105169 +0000 UTC m=+3557.214898562" Aug 04 11:36:09.698242 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:09.698206 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:36:14.104826 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:14.104798 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:36:19.256076 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:19.256043 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:36:39.080160 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.080126 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd"] Aug 04 11:36:39.083741 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.083725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.086123 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.086102 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e44431\"/\"ipp-route-e44431-kserve-self-signed-certs\"" Aug 04 11:36:39.087071 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.087045 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e44431\"/\"kube-root-ca.crt\"" Aug 04 11:36:39.087176 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.087081 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e44431\"/\"default-dockercfg-b48k4\"" Aug 04 11:36:39.087176 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.087147 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e44431\"/\"openshift-service-ca.crt\"" Aug 04 11:36:39.102300 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.102273 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd"] Aug 04 11:36:39.129117 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.129070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:36:39.171897 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.171855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-tmp-dir\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.172067 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.171908 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-dshm\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.172067 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.172009 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c6b0b7-034f-4603-8e50-e12dd0689a32-tls-certs\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.172067 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.172040 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9k9v\" (UniqueName: \"kubernetes.io/projected/43c6b0b7-034f-4603-8e50-e12dd0689a32-kube-api-access-b9k9v\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.172178 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.172134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-home\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.172178 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.172156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-model-cache\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.272650 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.272615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-home\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.272650 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.272654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-model-cache\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.272902 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.272676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-tmp-dir\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.272902 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.272700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-dshm\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.272902 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.272745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c6b0b7-034f-4603-8e50-e12dd0689a32-tls-certs\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.272902 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.272772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9k9v\" (UniqueName: \"kubernetes.io/projected/43c6b0b7-034f-4603-8e50-e12dd0689a32-kube-api-access-b9k9v\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.273160 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.273121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-tmp-dir\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.273160 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.273111 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-home\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.273328 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.273308 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-model-cache\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.275294 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.275260 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c6b0b7-034f-4603-8e50-e12dd0689a32-tls-certs\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.275429 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.275307 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-dshm\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.281049 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.281022 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9k9v\" (UniqueName: \"kubernetes.io/projected/43c6b0b7-034f-4603-8e50-e12dd0689a32-kube-api-access-b9k9v\") pod \"ipp-route-e44431-kserve-7dbc4b4c6c-qknfd\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.393365 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.393326 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:39.533851 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:39.533820 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd"] Aug 04 11:36:39.534239 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:36:39.534215 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43c6b0b7_034f_4603_8e50_e12dd0689a32.slice/crio-7521fac98e8e5a85393fc8045f9a2ca8cb910bcf9b3734aeddc4e429bc71cba1 WatchSource:0}: Error finding container 7521fac98e8e5a85393fc8045f9a2ca8cb910bcf9b3734aeddc4e429bc71cba1: Status 404 returned error can't find the container with id 7521fac98e8e5a85393fc8045f9a2ca8cb910bcf9b3734aeddc4e429bc71cba1 Aug 04 11:36:40.285323 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:40.285284 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" event={"ID":"43c6b0b7-034f-4603-8e50-e12dd0689a32","Type":"ContainerStarted","Data":"87379c8ef268a5153c7ea7a9ce85eb945e16933628286de800f344f8bc1d4ce2"} Aug 04 11:36:40.285797 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:40.285330 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" event={"ID":"43c6b0b7-034f-4603-8e50-e12dd0689a32","Type":"ContainerStarted","Data":"7521fac98e8e5a85393fc8045f9a2ca8cb910bcf9b3734aeddc4e429bc71cba1"} Aug 04 11:36:40.307215 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:40.307154 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" podStartSLOduration=1.30713737 podStartE2EDuration="1.30713737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:36:40.303496837 +0000 UTC m=+3589.401290231" watchObservedRunningTime="2026-08-04 11:36:40.30713737 +0000 UTC m=+3589.404930764" Aug 04 11:36:41.885205 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:41.885166 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:36:49.394484 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:49.394386 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:49.394484 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:49.394441 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:49.406675 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:49.406648 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:36:50.359855 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:36:50.359824 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:37:05.192365 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:05.192325 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:37:15.731850 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:15.731817 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:16.450058 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:16.450028 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:17.264775 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:17.264722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:17.363135 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:17.363086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:20.602723 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:20.602688 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:20.702493 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:20.702435 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:21.820466 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:21.820433 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j"] Aug 04 11:37:21.824622 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:21.824599 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:21.827195 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:21.827174 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-20aab6-serving-cert\"" Aug 04 11:37:21.835938 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:21.835916 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j"] Aug 04 11:37:21.863790 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:21.863757 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:21.978803 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:21.978760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjk8c\" (UniqueName: \"kubernetes.io/projected/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-kube-api-access-cjk8c\") pod \"maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j\" (UID: \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:21.979011 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:21.978906 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j\" (UID: \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:21.979066 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:21.979024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j\" (UID: \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:22.079625 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:22.079527 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j\" (UID: \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:22.079789 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:22.079649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j\" (UID: \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:22.079789 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:22.079691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjk8c\" (UniqueName: \"kubernetes.io/projected/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-kube-api-access-cjk8c\") pod \"maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j\" (UID: \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:22.080260 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:22.080236 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j\" (UID: \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:22.082017 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:22.082001 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j\" (UID: \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:22.090781 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:22.090756 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjk8c\" (UniqueName: \"kubernetes.io/projected/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-kube-api-access-cjk8c\") pod \"maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j\" (UID: \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:22.136335 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:22.136294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:22.150689 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:22.150657 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:22.275156 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:22.275123 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j"] Aug 04 11:37:22.276419 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:37:22.276394 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4295a9_9b08_4e3a_8993_18cf4e603cdb.slice/crio-24805474110db748271a6582de6870fd99f258af09996332b3261f0b2f330f4d WatchSource:0}: Error finding container 24805474110db748271a6582de6870fd99f258af09996332b3261f0b2f330f4d: Status 404 returned error can't find the container with id 24805474110db748271a6582de6870fd99f258af09996332b3261f0b2f330f4d Aug 04 11:37:22.394159 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:22.394104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:22.538872 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:22.538839 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" event={"ID":"1f4295a9-9b08-4e3a-8993-18cf4e603cdb","Type":"ContainerStarted","Data":"24805474110db748271a6582de6870fd99f258af09996332b3261f0b2f330f4d"} Aug 04 11:37:23.148753 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:23.148661 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:23.545829 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:23.545735 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" event={"ID":"1f4295a9-9b08-4e3a-8993-18cf4e603cdb","Type":"ContainerStarted","Data":"e9ce84913be9473810349b7b0a53a0491cce1366a5981bd06e59690d4909523a"} Aug 04 11:37:23.545829 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:23.545799 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:23.568743 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:23.568691 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" podStartSLOduration=2.007935058 podStartE2EDuration="2.568675478s" totalImagesPullingTime="560.74042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:37:23.565662048 +0000 UTC m=+3632.663455442" watchObservedRunningTime="2026-08-04 11:37:23.568675478 +0000 UTC m=+3632.666468872" Aug 04 11:37:26.602988 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:26.602956 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:37:29.557083 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:29.557055 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:34.472023 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:34.471986 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:40.504084 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:40.504047 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm"] Aug 04 11:37:40.507722 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:40.507699 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm" Aug 04 11:37:40.514128 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:40.514101 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tgsgx\"" Aug 04 11:37:40.529921 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:40.529893 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm"] Aug 04 11:37:40.544435 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:40.544394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a222586f-e6e5-4e24-87b5-3bcd28be35b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm\" (UID: \"a222586f-e6e5-4e24-87b5-3bcd28be35b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm" Aug 04 11:37:40.645154 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:40.645118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a222586f-e6e5-4e24-87b5-3bcd28be35b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm\" (UID: \"a222586f-e6e5-4e24-87b5-3bcd28be35b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm" Aug 04 11:37:40.645820 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:40.645798 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a222586f-e6e5-4e24-87b5-3bcd28be35b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm\" (UID: \"a222586f-e6e5-4e24-87b5-3bcd28be35b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm" Aug 04 11:37:40.820012 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:40.819924 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm" Aug 04 11:37:40.969954 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:40.969921 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm"] Aug 04 11:37:40.972048 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:37:40.972012 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda222586f_e6e5_4e24_87b5_3bcd28be35b0.slice/crio-8423e443314b127a625901606fb164d0a7fc3e9f9b12b6960781c1b2cb2745ca WatchSource:0}: Error finding container 8423e443314b127a625901606fb164d0a7fc3e9f9b12b6960781c1b2cb2745ca: Status 404 returned error can't find the container with id 8423e443314b127a625901606fb164d0a7fc3e9f9b12b6960781c1b2cb2745ca Aug 04 11:37:41.657911 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:41.657870 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm" event={"ID":"a222586f-e6e5-4e24-87b5-3bcd28be35b0","Type":"ContainerStarted","Data":"488f01008f2a391c44753aa76b7b1d824ad18a7057bf60b041a949653cbd806c"} Aug 04 11:37:41.658365 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:41.657934 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm" event={"ID":"a222586f-e6e5-4e24-87b5-3bcd28be35b0","Type":"ContainerStarted","Data":"8423e443314b127a625901606fb164d0a7fc3e9f9b12b6960781c1b2cb2745ca"} Aug 04 11:37:41.676403 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:41.676346 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm" podStartSLOduration=1.676331416 podStartE2EDuration="1.676331416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:37:41.674990425 +0000 UTC m=+3650.772783820" watchObservedRunningTime="2026-08-04 11:37:41.676331416 +0000 UTC m=+3650.774124832" Aug 04 11:37:42.666295 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:42.666258 2576 generic.go:356] "Generic (PLEG): container finished" podID="a222586f-e6e5-4e24-87b5-3bcd28be35b0" containerID="488f01008f2a391c44753aa76b7b1d824ad18a7057bf60b041a949653cbd806c" exitCode=0 Aug 04 11:37:42.666734 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:42.666352 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm" event={"ID":"a222586f-e6e5-4e24-87b5-3bcd28be35b0","Type":"ContainerDied","Data":"488f01008f2a391c44753aa76b7b1d824ad18a7057bf60b041a949653cbd806c"} Aug 04 11:37:43.803457 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:43.803431 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm" Aug 04 11:37:43.878128 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:43.878090 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a222586f-e6e5-4e24-87b5-3bcd28be35b0-maas-api-service-ca\") pod \"a222586f-e6e5-4e24-87b5-3bcd28be35b0\" (UID: \"a222586f-e6e5-4e24-87b5-3bcd28be35b0\") " Aug 04 11:37:43.878469 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:43.878447 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a222586f-e6e5-4e24-87b5-3bcd28be35b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a222586f-e6e5-4e24-87b5-3bcd28be35b0" (UID: "a222586f-e6e5-4e24-87b5-3bcd28be35b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:37:43.979673 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:43.979590 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a222586f-e6e5-4e24-87b5-3bcd28be35b0-maas-api-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:37:44.679302 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:44.679267 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm" Aug 04 11:37:44.679476 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:44.679310 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm" event={"ID":"a222586f-e6e5-4e24-87b5-3bcd28be35b0","Type":"ContainerDied","Data":"8423e443314b127a625901606fb164d0a7fc3e9f9b12b6960781c1b2cb2745ca"} Aug 04 11:37:44.679476 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:44.679357 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8423e443314b127a625901606fb164d0a7fc3e9f9b12b6960781c1b2cb2745ca" Aug 04 11:37:49.481347 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.481313 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j"] Aug 04 11:37:49.481783 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.481565 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" podUID="1f4295a9-9b08-4e3a-8993-18cf4e603cdb" containerName="maas-api" containerID="cri-o://e9ce84913be9473810349b7b0a53a0491cce1366a5981bd06e59690d4909523a" gracePeriod=30 Aug 04 11:37:49.504711 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.504680 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:49.552345 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.552276 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" podUID="1f4295a9-9b08-4e3a-8993-18cf4e603cdb" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.76:8443/health\": dial tcp 10.133.0.76:8443: connect: connection refused" Aug 04 11:37:49.590905 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.590869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:49.661185 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.661149 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:49.711077 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.711050 2576 generic.go:356] "Generic (PLEG): container finished" podID="1f4295a9-9b08-4e3a-8993-18cf4e603cdb" containerID="e9ce84913be9473810349b7b0a53a0491cce1366a5981bd06e59690d4909523a" exitCode=0 Aug 04 11:37:49.711220 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.711124 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" event={"ID":"1f4295a9-9b08-4e3a-8993-18cf4e603cdb","Type":"ContainerDied","Data":"e9ce84913be9473810349b7b0a53a0491cce1366a5981bd06e59690d4909523a"} Aug 04 11:37:49.729513 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.729479 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:49.794495 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.794402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:49.841919 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.841881 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-openshift-service-ca\") pod \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\" (UID: \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\") " Aug 04 11:37:49.842118 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.841931 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-maas-api-tls\") pod \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\" (UID: \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\") " Aug 04 11:37:49.842118 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.841981 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjk8c\" (UniqueName: \"kubernetes.io/projected/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-kube-api-access-cjk8c\") pod \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\" (UID: \"1f4295a9-9b08-4e3a-8993-18cf4e603cdb\") " Aug 04 11:37:49.842365 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.842338 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f4295a9-9b08-4e3a-8993-18cf4e603cdb" (UID: "1f4295a9-9b08-4e3a-8993-18cf4e603cdb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:37:49.844187 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.844156 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-kube-api-access-cjk8c" (OuterVolumeSpecName: "kube-api-access-cjk8c") pod "1f4295a9-9b08-4e3a-8993-18cf4e603cdb" (UID: "1f4295a9-9b08-4e3a-8993-18cf4e603cdb"). InnerVolumeSpecName "kube-api-access-cjk8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:37:49.844413 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.844384 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f4295a9-9b08-4e3a-8993-18cf4e603cdb" (UID: "1f4295a9-9b08-4e3a-8993-18cf4e603cdb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:37:49.936353 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.936313 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm"] Aug 04 11:37:49.940307 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.940276 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-20aab6-hmcmm"] Aug 04 11:37:49.942925 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.942904 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-openshift-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:37:49.942984 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.942929 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-maas-api-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:37:49.942984 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:49.942940 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjk8c\" (UniqueName: \"kubernetes.io/projected/1f4295a9-9b08-4e3a-8993-18cf4e603cdb-kube-api-access-cjk8c\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:37:50.319899 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:50.319856 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:50.718224 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:50.718199 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" Aug 04 11:37:50.718653 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:50.718240 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j" event={"ID":"1f4295a9-9b08-4e3a-8993-18cf4e603cdb","Type":"ContainerDied","Data":"24805474110db748271a6582de6870fd99f258af09996332b3261f0b2f330f4d"} Aug 04 11:37:50.718653 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:50.718301 2576 scope.go:117] "RemoveContainer" containerID="e9ce84913be9473810349b7b0a53a0491cce1366a5981bd06e59690d4909523a" Aug 04 11:37:50.740732 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:50.740708 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j"] Aug 04 11:37:50.745744 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:50.745713 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-20aab6-57f95ff6fb-qk76j"] Aug 04 11:37:51.440001 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:51.439968 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f4295a9-9b08-4e3a-8993-18cf4e603cdb" path="/var/lib/kubelet/pods/1f4295a9-9b08-4e3a-8993-18cf4e603cdb/volumes" Aug 04 11:37:51.440385 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:51.440364 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a222586f-e6e5-4e24-87b5-3bcd28be35b0" path="/var/lib/kubelet/pods/a222586f-e6e5-4e24-87b5-3bcd28be35b0/volumes" Aug 04 11:37:52.593937 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:52.593899 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:37:57.004379 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:57.004347 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:57.102933 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:57.102883 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:57.282921 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:57.282833 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:37:57.495682 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:37:57.495650 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:38:00.815187 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:00.815151 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj"] Aug 04 11:38:00.815621 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:00.815608 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f4295a9-9b08-4e3a-8993-18cf4e603cdb" containerName="maas-api" Aug 04 11:38:00.815719 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:00.815622 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f4295a9-9b08-4e3a-8993-18cf4e603cdb" containerName="maas-api" Aug 04 11:38:00.815719 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:00.815644 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a222586f-e6e5-4e24-87b5-3bcd28be35b0" containerName="revoke-keys" Aug 04 11:38:00.815719 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:00.815650 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a222586f-e6e5-4e24-87b5-3bcd28be35b0" containerName="revoke-keys" Aug 04 11:38:00.815814 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:00.815724 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f4295a9-9b08-4e3a-8993-18cf4e603cdb" containerName="maas-api" Aug 04 11:38:00.815814 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:00.815732 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a222586f-e6e5-4e24-87b5-3bcd28be35b0" containerName="revoke-keys" Aug 04 11:38:00.820395 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:00.820376 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj" Aug 04 11:38:00.824184 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:00.824158 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tgsgx\"" Aug 04 11:38:00.828553 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:00.828490 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj"] Aug 04 11:38:00.948347 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:00.948299 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8516e379-814f-43e1-b6d1-0e7d74894968-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj\" (UID: \"8516e379-814f-43e1-b6d1-0e7d74894968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj" Aug 04 11:38:01.048917 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:01.048874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8516e379-814f-43e1-b6d1-0e7d74894968-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj\" (UID: \"8516e379-814f-43e1-b6d1-0e7d74894968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj" Aug 04 11:38:01.049536 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:01.049493 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8516e379-814f-43e1-b6d1-0e7d74894968-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj\" (UID: \"8516e379-814f-43e1-b6d1-0e7d74894968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj" Aug 04 11:38:01.133003 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:01.132971 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj" Aug 04 11:38:01.270763 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:01.270734 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj"] Aug 04 11:38:01.271976 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:38:01.271949 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8516e379_814f_43e1_b6d1_0e7d74894968.slice/crio-a73b4fd631dcf3be7a9325aa2734da07aea29c6dfc0fa5f6e5ae16f84af84ea1 WatchSource:0}: Error finding container a73b4fd631dcf3be7a9325aa2734da07aea29c6dfc0fa5f6e5ae16f84af84ea1: Status 404 returned error can't find the container with id a73b4fd631dcf3be7a9325aa2734da07aea29c6dfc0fa5f6e5ae16f84af84ea1 Aug 04 11:38:01.784456 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:01.784366 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj" event={"ID":"8516e379-814f-43e1-b6d1-0e7d74894968","Type":"ContainerStarted","Data":"97c6021b08b15ec2c77505071b9e44209471915f2cd504aec3a088b926b46dd1"} Aug 04 11:38:01.784456 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:01.784426 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj" event={"ID":"8516e379-814f-43e1-b6d1-0e7d74894968","Type":"ContainerStarted","Data":"a73b4fd631dcf3be7a9325aa2734da07aea29c6dfc0fa5f6e5ae16f84af84ea1"} Aug 04 11:38:01.801454 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:01.801405 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj" podStartSLOduration=1.801389246 podStartE2EDuration="1.801389246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:38:01.799705825 +0000 UTC m=+3670.897499219" watchObservedRunningTime="2026-08-04 11:38:01.801389246 +0000 UTC m=+3670.899182647" Aug 04 11:38:02.794625 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:02.794592 2576 generic.go:356] "Generic (PLEG): container finished" podID="8516e379-814f-43e1-b6d1-0e7d74894968" containerID="97c6021b08b15ec2c77505071b9e44209471915f2cd504aec3a088b926b46dd1" exitCode=0 Aug 04 11:38:02.795035 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:02.794673 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj" event={"ID":"8516e379-814f-43e1-b6d1-0e7d74894968","Type":"ContainerDied","Data":"97c6021b08b15ec2c77505071b9e44209471915f2cd504aec3a088b926b46dd1"} Aug 04 11:38:03.935608 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:03.935586 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj" Aug 04 11:38:04.076871 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:04.076785 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8516e379-814f-43e1-b6d1-0e7d74894968-maas-api-service-ca\") pod \"8516e379-814f-43e1-b6d1-0e7d74894968\" (UID: \"8516e379-814f-43e1-b6d1-0e7d74894968\") " Aug 04 11:38:04.077188 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:04.077169 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8516e379-814f-43e1-b6d1-0e7d74894968-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8516e379-814f-43e1-b6d1-0e7d74894968" (UID: "8516e379-814f-43e1-b6d1-0e7d74894968"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:38:04.178183 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:04.178147 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8516e379-814f-43e1-b6d1-0e7d74894968-maas-api-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:38:04.807640 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:04.807610 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj" Aug 04 11:38:04.807811 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:04.807612 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj" event={"ID":"8516e379-814f-43e1-b6d1-0e7d74894968","Type":"ContainerDied","Data":"a73b4fd631dcf3be7a9325aa2734da07aea29c6dfc0fa5f6e5ae16f84af84ea1"} Aug 04 11:38:04.807811 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:04.807731 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a73b4fd631dcf3be7a9325aa2734da07aea29c6dfc0fa5f6e5ae16f84af84ea1" Aug 04 11:38:10.810033 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:10.809992 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:38:14.812302 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:14.812214 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95"] Aug 04 11:38:14.812706 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:14.812455 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" podUID="7bfab7d0-b7f0-438a-8774-99e577355c5b" containerName="maas-api" containerID="cri-o://bf3dbb3d6a6f3eee1c70eeb4df65b627be7f54eefff8c06c4c9e279cf0549b1b" gracePeriod=30 Aug 04 11:38:14.858836 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:14.858805 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:38:15.062539 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.062458 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:38:15.075227 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.075199 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bfab7d0-b7f0-438a-8774-99e577355c5b-openshift-service-ca\") pod \"7bfab7d0-b7f0-438a-8774-99e577355c5b\" (UID: \"7bfab7d0-b7f0-438a-8774-99e577355c5b\") " Aug 04 11:38:15.075342 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.075242 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5874n\" (UniqueName: \"kubernetes.io/projected/7bfab7d0-b7f0-438a-8774-99e577355c5b-kube-api-access-5874n\") pod \"7bfab7d0-b7f0-438a-8774-99e577355c5b\" (UID: \"7bfab7d0-b7f0-438a-8774-99e577355c5b\") " Aug 04 11:38:15.075433 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.075416 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bfab7d0-b7f0-438a-8774-99e577355c5b-maas-api-tls\") pod \"7bfab7d0-b7f0-438a-8774-99e577355c5b\" (UID: \"7bfab7d0-b7f0-438a-8774-99e577355c5b\") " Aug 04 11:38:15.075580 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.075557 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bfab7d0-b7f0-438a-8774-99e577355c5b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7bfab7d0-b7f0-438a-8774-99e577355c5b" (UID: "7bfab7d0-b7f0-438a-8774-99e577355c5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:38:15.075735 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.075721 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bfab7d0-b7f0-438a-8774-99e577355c5b-openshift-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:38:15.077326 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.077298 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bfab7d0-b7f0-438a-8774-99e577355c5b-kube-api-access-5874n" (OuterVolumeSpecName: "kube-api-access-5874n") pod "7bfab7d0-b7f0-438a-8774-99e577355c5b" (UID: "7bfab7d0-b7f0-438a-8774-99e577355c5b"). InnerVolumeSpecName "kube-api-access-5874n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:38:15.077419 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.077358 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bfab7d0-b7f0-438a-8774-99e577355c5b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7bfab7d0-b7f0-438a-8774-99e577355c5b" (UID: "7bfab7d0-b7f0-438a-8774-99e577355c5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:38:15.132659 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.132619 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj"] Aug 04 11:38:15.135747 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.135725 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e44431-cgwxj"] Aug 04 11:38:15.176360 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.176327 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bfab7d0-b7f0-438a-8774-99e577355c5b-maas-api-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:38:15.176360 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.176357 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5874n\" (UniqueName: \"kubernetes.io/projected/7bfab7d0-b7f0-438a-8774-99e577355c5b-kube-api-access-5874n\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:38:15.437911 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.437877 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8516e379-814f-43e1-b6d1-0e7d74894968" path="/var/lib/kubelet/pods/8516e379-814f-43e1-b6d1-0e7d74894968/volumes" Aug 04 11:38:15.875133 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.875102 2576 generic.go:356] "Generic (PLEG): container finished" podID="7bfab7d0-b7f0-438a-8774-99e577355c5b" containerID="bf3dbb3d6a6f3eee1c70eeb4df65b627be7f54eefff8c06c4c9e279cf0549b1b" exitCode=0 Aug 04 11:38:15.875564 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.875147 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" Aug 04 11:38:15.875564 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.875159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" event={"ID":"7bfab7d0-b7f0-438a-8774-99e577355c5b","Type":"ContainerDied","Data":"bf3dbb3d6a6f3eee1c70eeb4df65b627be7f54eefff8c06c4c9e279cf0549b1b"} Aug 04 11:38:15.875564 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.875198 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95" event={"ID":"7bfab7d0-b7f0-438a-8774-99e577355c5b","Type":"ContainerDied","Data":"7e66760db038f6fd3e7b789400989c3d8dce928aa248478d25b154f0c7a14008"} Aug 04 11:38:15.875564 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.875220 2576 scope.go:117] "RemoveContainer" containerID="bf3dbb3d6a6f3eee1c70eeb4df65b627be7f54eefff8c06c4c9e279cf0549b1b" Aug 04 11:38:15.884937 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.884921 2576 scope.go:117] "RemoveContainer" containerID="bf3dbb3d6a6f3eee1c70eeb4df65b627be7f54eefff8c06c4c9e279cf0549b1b" Aug 04 11:38:15.885196 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:38:15.885161 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf3dbb3d6a6f3eee1c70eeb4df65b627be7f54eefff8c06c4c9e279cf0549b1b\": container with ID starting with bf3dbb3d6a6f3eee1c70eeb4df65b627be7f54eefff8c06c4c9e279cf0549b1b not found: ID does not exist" containerID="bf3dbb3d6a6f3eee1c70eeb4df65b627be7f54eefff8c06c4c9e279cf0549b1b" Aug 04 11:38:15.885244 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.885203 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3dbb3d6a6f3eee1c70eeb4df65b627be7f54eefff8c06c4c9e279cf0549b1b"} err="failed to get container status \"bf3dbb3d6a6f3eee1c70eeb4df65b627be7f54eefff8c06c4c9e279cf0549b1b\": rpc error: code = NotFound desc = could not find container \"bf3dbb3d6a6f3eee1c70eeb4df65b627be7f54eefff8c06c4c9e279cf0549b1b\": container with ID starting with bf3dbb3d6a6f3eee1c70eeb4df65b627be7f54eefff8c06c4c9e279cf0549b1b not found: ID does not exist" Aug 04 11:38:15.894928 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.894904 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95"] Aug 04 11:38:15.902742 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:15.902723 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e44431-5f848d4cb4-m8s95"] Aug 04 11:38:17.438696 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:17.438666 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bfab7d0-b7f0-438a-8774-99e577355c5b" path="/var/lib/kubelet/pods/7bfab7d0-b7f0-438a-8774-99e577355c5b/volumes" Aug 04 11:38:17.787702 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:17.787620 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:38:19.851113 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:19.851076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:38:20.417959 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:20.417925 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd"] Aug 04 11:38:20.462208 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:20.462165 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:38:35.437494 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.437434 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" podUID="43c6b0b7-034f-4603-8e50-e12dd0689a32" containerName="main" containerID="cri-o://87379c8ef268a5153c7ea7a9ce85eb945e16933628286de800f344f8bc1d4ce2" gracePeriod=45 Aug 04 11:38:35.686491 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.686465 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:38:35.767422 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.767333 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c6b0b7-034f-4603-8e50-e12dd0689a32-tls-certs\") pod \"43c6b0b7-034f-4603-8e50-e12dd0689a32\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " Aug 04 11:38:35.767422 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.767396 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-tmp-dir\") pod \"43c6b0b7-034f-4603-8e50-e12dd0689a32\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " Aug 04 11:38:35.767689 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.767430 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-home\") pod \"43c6b0b7-034f-4603-8e50-e12dd0689a32\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " Aug 04 11:38:35.767689 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.767475 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-model-cache\") pod \"43c6b0b7-034f-4603-8e50-e12dd0689a32\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " Aug 04 11:38:35.767689 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.767497 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-dshm\") pod \"43c6b0b7-034f-4603-8e50-e12dd0689a32\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " Aug 04 11:38:35.767689 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.767539 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9k9v\" (UniqueName: \"kubernetes.io/projected/43c6b0b7-034f-4603-8e50-e12dd0689a32-kube-api-access-b9k9v\") pod \"43c6b0b7-034f-4603-8e50-e12dd0689a32\" (UID: \"43c6b0b7-034f-4603-8e50-e12dd0689a32\") " Aug 04 11:38:35.767887 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.767686 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43c6b0b7-034f-4603-8e50-e12dd0689a32" (UID: "43c6b0b7-034f-4603-8e50-e12dd0689a32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:38:35.767887 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.767720 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-home" (OuterVolumeSpecName: "home") pod "43c6b0b7-034f-4603-8e50-e12dd0689a32" (UID: "43c6b0b7-034f-4603-8e50-e12dd0689a32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:38:35.767887 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.767807 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-model-cache" (OuterVolumeSpecName: "model-cache") pod "43c6b0b7-034f-4603-8e50-e12dd0689a32" (UID: "43c6b0b7-034f-4603-8e50-e12dd0689a32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:38:35.768027 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.767906 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-tmp-dir\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:38:35.768027 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.767918 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-home\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:38:35.768027 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.767926 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-model-cache\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:38:35.769681 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.769655 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c6b0b7-034f-4603-8e50-e12dd0689a32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43c6b0b7-034f-4603-8e50-e12dd0689a32" (UID: "43c6b0b7-034f-4603-8e50-e12dd0689a32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:38:35.769681 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.769670 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-dshm" (OuterVolumeSpecName: "dshm") pod "43c6b0b7-034f-4603-8e50-e12dd0689a32" (UID: "43c6b0b7-034f-4603-8e50-e12dd0689a32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:38:35.769681 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.769675 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c6b0b7-034f-4603-8e50-e12dd0689a32-kube-api-access-b9k9v" (OuterVolumeSpecName: "kube-api-access-b9k9v") pod "43c6b0b7-034f-4603-8e50-e12dd0689a32" (UID: "43c6b0b7-034f-4603-8e50-e12dd0689a32"). InnerVolumeSpecName "kube-api-access-b9k9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:38:35.869330 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.869292 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43c6b0b7-034f-4603-8e50-e12dd0689a32-dshm\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:38:35.869330 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.869328 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9k9v\" (UniqueName: \"kubernetes.io/projected/43c6b0b7-034f-4603-8e50-e12dd0689a32-kube-api-access-b9k9v\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:38:35.869614 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.869345 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c6b0b7-034f-4603-8e50-e12dd0689a32-tls-certs\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:38:35.995059 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.995024 2576 generic.go:356] "Generic (PLEG): container finished" podID="43c6b0b7-034f-4603-8e50-e12dd0689a32" containerID="87379c8ef268a5153c7ea7a9ce85eb945e16933628286de800f344f8bc1d4ce2" exitCode=0 Aug 04 11:38:35.995240 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.995065 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" Aug 04 11:38:35.995240 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.995104 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" event={"ID":"43c6b0b7-034f-4603-8e50-e12dd0689a32","Type":"ContainerDied","Data":"87379c8ef268a5153c7ea7a9ce85eb945e16933628286de800f344f8bc1d4ce2"} Aug 04 11:38:35.995240 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.995148 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd" event={"ID":"43c6b0b7-034f-4603-8e50-e12dd0689a32","Type":"ContainerDied","Data":"7521fac98e8e5a85393fc8045f9a2ca8cb910bcf9b3734aeddc4e429bc71cba1"} Aug 04 11:38:35.995240 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:35.995170 2576 scope.go:117] "RemoveContainer" containerID="87379c8ef268a5153c7ea7a9ce85eb945e16933628286de800f344f8bc1d4ce2" Aug 04 11:38:36.005050 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:36.005032 2576 scope.go:117] "RemoveContainer" containerID="87379c8ef268a5153c7ea7a9ce85eb945e16933628286de800f344f8bc1d4ce2" Aug 04 11:38:36.005364 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:38:36.005344 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87379c8ef268a5153c7ea7a9ce85eb945e16933628286de800f344f8bc1d4ce2\": container with ID starting with 87379c8ef268a5153c7ea7a9ce85eb945e16933628286de800f344f8bc1d4ce2 not found: ID does not exist" containerID="87379c8ef268a5153c7ea7a9ce85eb945e16933628286de800f344f8bc1d4ce2" Aug 04 11:38:36.005425 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:36.005377 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87379c8ef268a5153c7ea7a9ce85eb945e16933628286de800f344f8bc1d4ce2"} err="failed to get container status \"87379c8ef268a5153c7ea7a9ce85eb945e16933628286de800f344f8bc1d4ce2\": rpc error: code = NotFound desc = could not find container \"87379c8ef268a5153c7ea7a9ce85eb945e16933628286de800f344f8bc1d4ce2\": container with ID starting with 87379c8ef268a5153c7ea7a9ce85eb945e16933628286de800f344f8bc1d4ce2 not found: ID does not exist" Aug 04 11:38:36.017724 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:36.017642 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd"] Aug 04 11:38:36.021517 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:36.021474 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-e44431/ipp-route-e44431-kserve-7dbc4b4c6c-qknfd"] Aug 04 11:38:37.438294 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:37.438259 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43c6b0b7-034f-4603-8e50-e12dd0689a32" path="/var/lib/kubelet/pods/43c6b0b7-034f-4603-8e50-e12dd0689a32/volumes" Aug 04 11:38:57.998143 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:57.998103 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:38:58.378272 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:38:58.378240 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:04.920289 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:04.920252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:11.258929 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.258896 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:11.359590 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.359556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:11.563284 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.563193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:11.779415 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.779382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:11.889054 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.889013 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5"] Aug 04 11:39:11.889535 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.889487 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8516e379-814f-43e1-b6d1-0e7d74894968" containerName="revoke-keys" Aug 04 11:39:11.889535 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.889517 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8516e379-814f-43e1-b6d1-0e7d74894968" containerName="revoke-keys" Aug 04 11:39:11.889635 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.889543 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c6b0b7-034f-4603-8e50-e12dd0689a32" containerName="main" Aug 04 11:39:11.889635 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.889549 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c6b0b7-034f-4603-8e50-e12dd0689a32" containerName="main" Aug 04 11:39:11.889635 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.889560 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bfab7d0-b7f0-438a-8774-99e577355c5b" containerName="maas-api" Aug 04 11:39:11.889635 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.889566 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bfab7d0-b7f0-438a-8774-99e577355c5b" containerName="maas-api" Aug 04 11:39:11.889761 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.889651 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8516e379-814f-43e1-b6d1-0e7d74894968" containerName="revoke-keys" Aug 04 11:39:11.889761 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.889661 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="43c6b0b7-034f-4603-8e50-e12dd0689a32" containerName="main" Aug 04 11:39:11.889761 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.889668 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bfab7d0-b7f0-438a-8774-99e577355c5b" containerName="maas-api" Aug 04 11:39:11.892925 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.892903 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5" Aug 04 11:39:11.896440 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.896417 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tgsgx\"" Aug 04 11:39:11.903044 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.903022 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5"] Aug 04 11:39:11.995647 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:11.995608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7281adf4-e608-432f-a546-62f35e47246a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5\" (UID: \"7281adf4-e608-432f-a546-62f35e47246a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5" Aug 04 11:39:12.096957 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:12.096918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7281adf4-e608-432f-a546-62f35e47246a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5\" (UID: \"7281adf4-e608-432f-a546-62f35e47246a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5" Aug 04 11:39:12.097594 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:12.097573 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7281adf4-e608-432f-a546-62f35e47246a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5\" (UID: \"7281adf4-e608-432f-a546-62f35e47246a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5" Aug 04 11:39:12.204781 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:12.204691 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5" Aug 04 11:39:12.346733 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:12.346706 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5"] Aug 04 11:39:12.348326 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:39:12.348292 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7281adf4_e608_432f_a546_62f35e47246a.slice/crio-c47dbf919f5c9ee68c39d0758dba8989bf2cd53abaa9af95ab088f66c886fdd8 WatchSource:0}: Error finding container c47dbf919f5c9ee68c39d0758dba8989bf2cd53abaa9af95ab088f66c886fdd8: Status 404 returned error can't find the container with id c47dbf919f5c9ee68c39d0758dba8989bf2cd53abaa9af95ab088f66c886fdd8 Aug 04 11:39:13.219412 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:13.219320 2576 generic.go:356] "Generic (PLEG): container finished" podID="7281adf4-e608-432f-a546-62f35e47246a" containerID="f61abe1dcec2ddbb42f41a2aa361e8c95869ac5edbe7287317d1146b657e9b45" exitCode=0 Aug 04 11:39:13.219412 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:13.219383 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5" event={"ID":"7281adf4-e608-432f-a546-62f35e47246a","Type":"ContainerDied","Data":"f61abe1dcec2ddbb42f41a2aa361e8c95869ac5edbe7287317d1146b657e9b45"} Aug 04 11:39:13.219636 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:13.219431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5" event={"ID":"7281adf4-e608-432f-a546-62f35e47246a","Type":"ContainerStarted","Data":"c47dbf919f5c9ee68c39d0758dba8989bf2cd53abaa9af95ab088f66c886fdd8"} Aug 04 11:39:14.365923 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:14.365895 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5" Aug 04 11:39:14.419912 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:14.419880 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7281adf4-e608-432f-a546-62f35e47246a-maas-api-service-ca\") pod \"7281adf4-e608-432f-a546-62f35e47246a\" (UID: \"7281adf4-e608-432f-a546-62f35e47246a\") " Aug 04 11:39:14.420242 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:14.420220 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7281adf4-e608-432f-a546-62f35e47246a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7281adf4-e608-432f-a546-62f35e47246a" (UID: "7281adf4-e608-432f-a546-62f35e47246a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:39:14.521211 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:14.521118 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7281adf4-e608-432f-a546-62f35e47246a-maas-api-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:39:15.233010 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:15.232980 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5" Aug 04 11:39:15.233214 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:15.233013 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5" event={"ID":"7281adf4-e608-432f-a546-62f35e47246a","Type":"ContainerDied","Data":"c47dbf919f5c9ee68c39d0758dba8989bf2cd53abaa9af95ab088f66c886fdd8"} Aug 04 11:39:15.233214 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:15.233064 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c47dbf919f5c9ee68c39d0758dba8989bf2cd53abaa9af95ab088f66c886fdd8" Aug 04 11:39:15.961639 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:15.961607 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:17.592241 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:17.592207 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs"] Aug 04 11:39:17.592690 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:17.592436 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" podUID="7d006cef-9618-4a04-a178-2cbb35d4a20c" containerName="maas-api" containerID="cri-o://e805ac2621ec89922baa02a414ad55d086eeba2e9e0bc50f104e0451586e84de" gracePeriod=30 Aug 04 11:39:17.608467 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:17.608433 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:17.690817 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:17.690786 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:17.752428 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:17.752379 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:17.845019 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:17.844961 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:39:17.854021 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:17.853993 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:17.955002 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:17.954959 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2nf2\" (UniqueName: \"kubernetes.io/projected/7d006cef-9618-4a04-a178-2cbb35d4a20c-kube-api-access-t2nf2\") pod \"7d006cef-9618-4a04-a178-2cbb35d4a20c\" (UID: \"7d006cef-9618-4a04-a178-2cbb35d4a20c\") " Aug 04 11:39:17.955187 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:17.955025 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d006cef-9618-4a04-a178-2cbb35d4a20c-openshift-service-ca\") pod \"7d006cef-9618-4a04-a178-2cbb35d4a20c\" (UID: \"7d006cef-9618-4a04-a178-2cbb35d4a20c\") " Aug 04 11:39:17.955187 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:17.955077 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d006cef-9618-4a04-a178-2cbb35d4a20c-maas-api-tls\") pod \"7d006cef-9618-4a04-a178-2cbb35d4a20c\" (UID: \"7d006cef-9618-4a04-a178-2cbb35d4a20c\") " Aug 04 11:39:17.955395 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:17.955365 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d006cef-9618-4a04-a178-2cbb35d4a20c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7d006cef-9618-4a04-a178-2cbb35d4a20c" (UID: "7d006cef-9618-4a04-a178-2cbb35d4a20c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:39:17.957159 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:17.957125 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d006cef-9618-4a04-a178-2cbb35d4a20c-kube-api-access-t2nf2" (OuterVolumeSpecName: "kube-api-access-t2nf2") pod "7d006cef-9618-4a04-a178-2cbb35d4a20c" (UID: "7d006cef-9618-4a04-a178-2cbb35d4a20c"). InnerVolumeSpecName "kube-api-access-t2nf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:39:17.957270 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:17.957219 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d006cef-9618-4a04-a178-2cbb35d4a20c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7d006cef-9618-4a04-a178-2cbb35d4a20c" (UID: "7d006cef-9618-4a04-a178-2cbb35d4a20c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:39:17.999400 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:17.999360 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5"] Aug 04 11:39:18.002284 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:18.002249 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-29b2ff-6z9v5"] Aug 04 11:39:18.056398 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:18.056360 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2nf2\" (UniqueName: \"kubernetes.io/projected/7d006cef-9618-4a04-a178-2cbb35d4a20c-kube-api-access-t2nf2\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:39:18.056398 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:18.056391 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d006cef-9618-4a04-a178-2cbb35d4a20c-openshift-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:39:18.056398 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:18.056406 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d006cef-9618-4a04-a178-2cbb35d4a20c-maas-api-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:39:18.257525 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:18.257412 2576 generic.go:356] "Generic (PLEG): container finished" podID="7d006cef-9618-4a04-a178-2cbb35d4a20c" containerID="e805ac2621ec89922baa02a414ad55d086eeba2e9e0bc50f104e0451586e84de" exitCode=0 Aug 04 11:39:18.257525 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:18.257459 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" Aug 04 11:39:18.257525 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:18.257475 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" event={"ID":"7d006cef-9618-4a04-a178-2cbb35d4a20c","Type":"ContainerDied","Data":"e805ac2621ec89922baa02a414ad55d086eeba2e9e0bc50f104e0451586e84de"} Aug 04 11:39:18.257815 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:18.257548 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs" event={"ID":"7d006cef-9618-4a04-a178-2cbb35d4a20c","Type":"ContainerDied","Data":"4df740f3584b3e6e86f7ddb3f35c6b05239d86c8d23369a6abd5fcc7a636e458"} Aug 04 11:39:18.257815 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:18.257574 2576 scope.go:117] "RemoveContainer" containerID="e805ac2621ec89922baa02a414ad55d086eeba2e9e0bc50f104e0451586e84de" Aug 04 11:39:18.267276 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:18.267261 2576 scope.go:117] "RemoveContainer" containerID="e805ac2621ec89922baa02a414ad55d086eeba2e9e0bc50f104e0451586e84de" Aug 04 11:39:18.267538 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:39:18.267514 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e805ac2621ec89922baa02a414ad55d086eeba2e9e0bc50f104e0451586e84de\": container with ID starting with e805ac2621ec89922baa02a414ad55d086eeba2e9e0bc50f104e0451586e84de not found: ID does not exist" containerID="e805ac2621ec89922baa02a414ad55d086eeba2e9e0bc50f104e0451586e84de" Aug 04 11:39:18.267594 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:18.267546 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e805ac2621ec89922baa02a414ad55d086eeba2e9e0bc50f104e0451586e84de"} err="failed to get container status \"e805ac2621ec89922baa02a414ad55d086eeba2e9e0bc50f104e0451586e84de\": rpc error: code = NotFound desc = could not find container \"e805ac2621ec89922baa02a414ad55d086eeba2e9e0bc50f104e0451586e84de\": container with ID starting with e805ac2621ec89922baa02a414ad55d086eeba2e9e0bc50f104e0451586e84de not found: ID does not exist" Aug 04 11:39:18.290662 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:18.290632 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs"] Aug 04 11:39:18.303612 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:18.303589 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-29b2ff-5cc87479f9-2jtcs"] Aug 04 11:39:19.058590 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:19.058551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:19.437867 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:19.437822 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7281adf4-e608-432f-a546-62f35e47246a" path="/var/lib/kubelet/pods/7281adf4-e608-432f-a546-62f35e47246a/volumes" Aug 04 11:39:19.438328 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:19.438308 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d006cef-9618-4a04-a178-2cbb35d4a20c" path="/var/lib/kubelet/pods/7d006cef-9618-4a04-a178-2cbb35d4a20c/volumes" Aug 04 11:39:19.988751 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:19.988712 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:39:25.226380 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.226340 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:25.329167 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.329129 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:25.487809 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.487727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:25.717336 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.717303 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:25.861857 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.861824 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42"] Aug 04 11:39:25.862229 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.862217 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7281adf4-e608-432f-a546-62f35e47246a" containerName="revoke-keys" Aug 04 11:39:25.862272 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.862230 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7281adf4-e608-432f-a546-62f35e47246a" containerName="revoke-keys" Aug 04 11:39:25.862272 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.862243 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d006cef-9618-4a04-a178-2cbb35d4a20c" containerName="maas-api" Aug 04 11:39:25.862272 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.862249 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d006cef-9618-4a04-a178-2cbb35d4a20c" containerName="maas-api" Aug 04 11:39:25.862369 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.862315 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7281adf4-e608-432f-a546-62f35e47246a" containerName="revoke-keys" Aug 04 11:39:25.862369 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.862324 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d006cef-9618-4a04-a178-2cbb35d4a20c" containerName="maas-api" Aug 04 11:39:25.866867 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.866848 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42" Aug 04 11:39:25.869508 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.869478 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tgsgx\"" Aug 04 11:39:25.873762 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.873734 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42"] Aug 04 11:39:25.927902 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:25.927866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2611f3b6-e38f-471a-9db8-ac11a1c31837-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42\" (UID: \"2611f3b6-e38f-471a-9db8-ac11a1c31837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42" Aug 04 11:39:26.029379 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:26.029337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2611f3b6-e38f-471a-9db8-ac11a1c31837-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42\" (UID: \"2611f3b6-e38f-471a-9db8-ac11a1c31837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42" Aug 04 11:39:26.030047 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:26.030027 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2611f3b6-e38f-471a-9db8-ac11a1c31837-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42\" (UID: \"2611f3b6-e38f-471a-9db8-ac11a1c31837\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42" Aug 04 11:39:26.179763 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:26.179676 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42" Aug 04 11:39:26.320177 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:26.320144 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42"] Aug 04 11:39:26.321774 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:39:26.321750 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2611f3b6_e38f_471a_9db8_ac11a1c31837.slice/crio-345c3ff7628b1b618279748e415dd0439cc3df3a60d9fc105b48e2013327905b WatchSource:0}: Error finding container 345c3ff7628b1b618279748e415dd0439cc3df3a60d9fc105b48e2013327905b: Status 404 returned error can't find the container with id 345c3ff7628b1b618279748e415dd0439cc3df3a60d9fc105b48e2013327905b Aug 04 11:39:27.313968 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:27.313873 2576 generic.go:356] "Generic (PLEG): container finished" podID="2611f3b6-e38f-471a-9db8-ac11a1c31837" containerID="b91a7c6cb64c55481297984398a59191e964ae3779d3db8adf5f8b0a232a7733" exitCode=0 Aug 04 11:39:27.314121 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:27.313967 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42" event={"ID":"2611f3b6-e38f-471a-9db8-ac11a1c31837","Type":"ContainerDied","Data":"b91a7c6cb64c55481297984398a59191e964ae3779d3db8adf5f8b0a232a7733"} Aug 04 11:39:27.314121 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:27.314063 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42" event={"ID":"2611f3b6-e38f-471a-9db8-ac11a1c31837","Type":"ContainerStarted","Data":"345c3ff7628b1b618279748e415dd0439cc3df3a60d9fc105b48e2013327905b"} Aug 04 11:39:28.457604 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:28.457575 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42" Aug 04 11:39:28.551810 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:28.551780 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2611f3b6-e38f-471a-9db8-ac11a1c31837-maas-api-service-ca\") pod \"2611f3b6-e38f-471a-9db8-ac11a1c31837\" (UID: \"2611f3b6-e38f-471a-9db8-ac11a1c31837\") " Aug 04 11:39:28.552146 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:28.552115 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2611f3b6-e38f-471a-9db8-ac11a1c31837-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2611f3b6-e38f-471a-9db8-ac11a1c31837" (UID: "2611f3b6-e38f-471a-9db8-ac11a1c31837"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:39:28.652518 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:28.652483 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2611f3b6-e38f-471a-9db8-ac11a1c31837-maas-api-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:39:29.328218 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:29.328192 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42" Aug 04 11:39:29.328366 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:29.328233 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42" event={"ID":"2611f3b6-e38f-471a-9db8-ac11a1c31837","Type":"ContainerDied","Data":"345c3ff7628b1b618279748e415dd0439cc3df3a60d9fc105b48e2013327905b"} Aug 04 11:39:29.328366 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:29.328282 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="345c3ff7628b1b618279748e415dd0439cc3df3a60d9fc105b48e2013327905b" Aug 04 11:39:30.934289 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:30.934257 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5"] Aug 04 11:39:30.934775 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:30.934467 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" podUID="dbe00718-f1a6-4f52-8aee-125b654ed662" containerName="maas-api" containerID="cri-o://f00b4cc06297521fc79906ca6b1340d27656784d5d180e639c23f9c4bbc73047" gracePeriod=30 Aug 04 11:39:30.970929 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:30.970896 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ed8ca8-29764050-7vxtf"] Aug 04 11:39:30.975096 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:30.975065 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ed8ca8-29764050-7vxtf"] Aug 04 11:39:30.979367 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:30.979344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:31.049000 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.048972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:31.182845 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.182823 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:39:31.190351 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.190294 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:31.228412 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.228370 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42"] Aug 04 11:39:31.231548 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.231518 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ed8ca8-mmt42"] Aug 04 11:39:31.274574 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.274491 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe00718-f1a6-4f52-8aee-125b654ed662-openshift-service-ca\") pod \"dbe00718-f1a6-4f52-8aee-125b654ed662\" (UID: \"dbe00718-f1a6-4f52-8aee-125b654ed662\") " Aug 04 11:39:31.274574 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.274579 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmw4x\" (UniqueName: \"kubernetes.io/projected/dbe00718-f1a6-4f52-8aee-125b654ed662-kube-api-access-bmw4x\") pod \"dbe00718-f1a6-4f52-8aee-125b654ed662\" (UID: \"dbe00718-f1a6-4f52-8aee-125b654ed662\") " Aug 04 11:39:31.274829 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.274641 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbe00718-f1a6-4f52-8aee-125b654ed662-maas-api-tls\") pod \"dbe00718-f1a6-4f52-8aee-125b654ed662\" (UID: \"dbe00718-f1a6-4f52-8aee-125b654ed662\") " Aug 04 11:39:31.274890 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.274839 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe00718-f1a6-4f52-8aee-125b654ed662-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dbe00718-f1a6-4f52-8aee-125b654ed662" (UID: "dbe00718-f1a6-4f52-8aee-125b654ed662"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:39:31.276698 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.276664 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe00718-f1a6-4f52-8aee-125b654ed662-kube-api-access-bmw4x" (OuterVolumeSpecName: "kube-api-access-bmw4x") pod "dbe00718-f1a6-4f52-8aee-125b654ed662" (UID: "dbe00718-f1a6-4f52-8aee-125b654ed662"). InnerVolumeSpecName "kube-api-access-bmw4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:39:31.276698 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.276665 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbe00718-f1a6-4f52-8aee-125b654ed662-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dbe00718-f1a6-4f52-8aee-125b654ed662" (UID: "dbe00718-f1a6-4f52-8aee-125b654ed662"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:39:31.342006 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.341964 2576 generic.go:356] "Generic (PLEG): container finished" podID="dbe00718-f1a6-4f52-8aee-125b654ed662" containerID="f00b4cc06297521fc79906ca6b1340d27656784d5d180e639c23f9c4bbc73047" exitCode=0 Aug 04 11:39:31.342213 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.342014 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" Aug 04 11:39:31.342213 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.342048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" event={"ID":"dbe00718-f1a6-4f52-8aee-125b654ed662","Type":"ContainerDied","Data":"f00b4cc06297521fc79906ca6b1340d27656784d5d180e639c23f9c4bbc73047"} Aug 04 11:39:31.342213 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.342108 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5" event={"ID":"dbe00718-f1a6-4f52-8aee-125b654ed662","Type":"ContainerDied","Data":"0d6f8ab109792be77adfe097bbaae050b88ada5b7151ef1145c054bff1bbea19"} Aug 04 11:39:31.342213 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.342134 2576 scope.go:117] "RemoveContainer" containerID="f00b4cc06297521fc79906ca6b1340d27656784d5d180e639c23f9c4bbc73047" Aug 04 11:39:31.352136 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.352120 2576 scope.go:117] "RemoveContainer" containerID="f00b4cc06297521fc79906ca6b1340d27656784d5d180e639c23f9c4bbc73047" Aug 04 11:39:31.352395 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:39:31.352380 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00b4cc06297521fc79906ca6b1340d27656784d5d180e639c23f9c4bbc73047\": container with ID starting with f00b4cc06297521fc79906ca6b1340d27656784d5d180e639c23f9c4bbc73047 not found: ID does not exist" containerID="f00b4cc06297521fc79906ca6b1340d27656784d5d180e639c23f9c4bbc73047" Aug 04 11:39:31.352462 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.352404 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00b4cc06297521fc79906ca6b1340d27656784d5d180e639c23f9c4bbc73047"} err="failed to get container status \"f00b4cc06297521fc79906ca6b1340d27656784d5d180e639c23f9c4bbc73047\": rpc error: code = NotFound desc = could not find container \"f00b4cc06297521fc79906ca6b1340d27656784d5d180e639c23f9c4bbc73047\": container with ID starting with f00b4cc06297521fc79906ca6b1340d27656784d5d180e639c23f9c4bbc73047 not found: ID does not exist" Aug 04 11:39:31.365292 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.365272 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5"] Aug 04 11:39:31.372697 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.372678 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed8ca8-84c896574d-skrj5"] Aug 04 11:39:31.375640 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.375619 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe00718-f1a6-4f52-8aee-125b654ed662-openshift-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:39:31.375702 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.375643 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmw4x\" (UniqueName: \"kubernetes.io/projected/dbe00718-f1a6-4f52-8aee-125b654ed662-kube-api-access-bmw4x\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:39:31.375702 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.375655 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbe00718-f1a6-4f52-8aee-125b654ed662-maas-api-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:39:31.438327 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.438292 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2611f3b6-e38f-471a-9db8-ac11a1c31837" path="/var/lib/kubelet/pods/2611f3b6-e38f-471a-9db8-ac11a1c31837/volumes" Aug 04 11:39:31.438732 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.438715 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd3d8bf7-954d-44b3-b08e-f250ddeed083" path="/var/lib/kubelet/pods/cd3d8bf7-954d-44b3-b08e-f250ddeed083/volumes" Aug 04 11:39:31.439044 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.439032 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbe00718-f1a6-4f52-8aee-125b654ed662" path="/var/lib/kubelet/pods/dbe00718-f1a6-4f52-8aee-125b654ed662/volumes" Aug 04 11:39:31.555029 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:31.554944 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:33.293663 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:33.293625 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:39:37.902091 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:37.902053 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:37.963029 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:37.962999 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:38.154228 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.154117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:38.396629 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.396592 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:38.503400 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.503318 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr"] Aug 04 11:39:38.503731 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.503718 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe00718-f1a6-4f52-8aee-125b654ed662" containerName="maas-api" Aug 04 11:39:38.503731 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.503732 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe00718-f1a6-4f52-8aee-125b654ed662" containerName="maas-api" Aug 04 11:39:38.503826 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.503750 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2611f3b6-e38f-471a-9db8-ac11a1c31837" containerName="revoke-keys" Aug 04 11:39:38.503826 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.503756 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2611f3b6-e38f-471a-9db8-ac11a1c31837" containerName="revoke-keys" Aug 04 11:39:38.503826 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.503824 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe00718-f1a6-4f52-8aee-125b654ed662" containerName="maas-api" Aug 04 11:39:38.503928 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.503831 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2611f3b6-e38f-471a-9db8-ac11a1c31837" containerName="revoke-keys" Aug 04 11:39:38.507022 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.507005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr" Aug 04 11:39:38.509574 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.509552 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tgsgx\"" Aug 04 11:39:38.515077 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.514547 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr"] Aug 04 11:39:38.640694 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.640655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0844e4-2130-4f8a-a637-9c4f55498f75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr\" (UID: \"9f0844e4-2130-4f8a-a637-9c4f55498f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr" Aug 04 11:39:38.742028 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.741990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0844e4-2130-4f8a-a637-9c4f55498f75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr\" (UID: \"9f0844e4-2130-4f8a-a637-9c4f55498f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr" Aug 04 11:39:38.742702 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.742681 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0844e4-2130-4f8a-a637-9c4f55498f75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr\" (UID: \"9f0844e4-2130-4f8a-a637-9c4f55498f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr" Aug 04 11:39:38.819140 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.819053 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr" Aug 04 11:39:38.981798 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:38.981772 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr"] Aug 04 11:39:38.983637 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:39:38.983608 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0844e4_2130_4f8a_a637_9c4f55498f75.slice/crio-a3d8cd7f86bef50d7ea870e72fde10c47858329980c27c8ffc1aa1458790b736 WatchSource:0}: Error finding container a3d8cd7f86bef50d7ea870e72fde10c47858329980c27c8ffc1aa1458790b736: Status 404 returned error can't find the container with id a3d8cd7f86bef50d7ea870e72fde10c47858329980c27c8ffc1aa1458790b736 Aug 04 11:39:39.391538 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:39.391438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr" event={"ID":"9f0844e4-2130-4f8a-a637-9c4f55498f75","Type":"ContainerStarted","Data":"b2ed006c20aeb3ea06d157d137a2e51693301c973f7716486d8cae5e0067975c"} Aug 04 11:39:39.391538 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:39.391492 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr" event={"ID":"9f0844e4-2130-4f8a-a637-9c4f55498f75","Type":"ContainerStarted","Data":"a3d8cd7f86bef50d7ea870e72fde10c47858329980c27c8ffc1aa1458790b736"} Aug 04 11:39:39.408280 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:39.408228 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr" podStartSLOduration=1.408211782 podStartE2EDuration="1.408211782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:39:39.405762256 +0000 UTC m=+3768.503555651" watchObservedRunningTime="2026-08-04 11:39:39.408211782 +0000 UTC m=+3768.506005198" Aug 04 11:39:39.639231 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:39:39.639200 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0844e4_2130_4f8a_a637_9c4f55498f75.slice/crio-b2ed006c20aeb3ea06d157d137a2e51693301c973f7716486d8cae5e0067975c.scope\": RecentStats: unable to find data in memory cache]" Aug 04 11:39:40.399054 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:40.399021 2576 generic.go:356] "Generic (PLEG): container finished" podID="9f0844e4-2130-4f8a-a637-9c4f55498f75" containerID="b2ed006c20aeb3ea06d157d137a2e51693301c973f7716486d8cae5e0067975c" exitCode=0 Aug 04 11:39:40.399450 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:40.399112 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr" event={"ID":"9f0844e4-2130-4f8a-a637-9c4f55498f75","Type":"ContainerDied","Data":"b2ed006c20aeb3ea06d157d137a2e51693301c973f7716486d8cae5e0067975c"} Aug 04 11:39:41.543174 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:41.543137 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr" Aug 04 11:39:41.666355 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:41.666246 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0844e4-2130-4f8a-a637-9c4f55498f75-maas-api-service-ca\") pod \"9f0844e4-2130-4f8a-a637-9c4f55498f75\" (UID: \"9f0844e4-2130-4f8a-a637-9c4f55498f75\") " Aug 04 11:39:41.666683 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:41.666663 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0844e4-2130-4f8a-a637-9c4f55498f75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f0844e4-2130-4f8a-a637-9c4f55498f75" (UID: "9f0844e4-2130-4f8a-a637-9c4f55498f75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:39:41.767920 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:41.767884 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0844e4-2130-4f8a-a637-9c4f55498f75-maas-api-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:39:42.412352 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:42.412321 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr" Aug 04 11:39:42.412539 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:42.412358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr" event={"ID":"9f0844e4-2130-4f8a-a637-9c4f55498f75","Type":"ContainerDied","Data":"a3d8cd7f86bef50d7ea870e72fde10c47858329980c27c8ffc1aa1458790b736"} Aug 04 11:39:42.412539 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:42.412402 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3d8cd7f86bef50d7ea870e72fde10c47858329980c27c8ffc1aa1458790b736" Aug 04 11:39:49.064712 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.064639 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56"] Aug 04 11:39:49.065079 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.064838 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" podUID="f6d67ddb-79d6-44bf-9e44-a7aef71c393e" containerName="maas-api" containerID="cri-o://22f5222a16df153869fcab808eb73d41999ec1863c2195098513d702a13387bf" gracePeriod=30 Aug 04 11:39:49.083058 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.082967 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:49.085763 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.085740 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c74755-29764050-8msxn"] Aug 04 11:39:49.094070 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.094036 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c74755-29764050-8msxn"] Aug 04 11:39:49.166146 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.166116 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:49.222757 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.222716 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:49.283325 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.283288 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr"] Aug 04 11:39:49.287832 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.287806 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c74755-c7fxr"] Aug 04 11:39:49.291743 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.291721 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:49.316238 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.316177 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:39:49.435097 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.435062 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-openshift-service-ca\") pod \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\" (UID: \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\") " Aug 04 11:39:49.435294 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.435125 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpvmk\" (UniqueName: \"kubernetes.io/projected/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-kube-api-access-dpvmk\") pod \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\" (UID: \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\") " Aug 04 11:39:49.435294 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.435196 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-maas-api-tls\") pod \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\" (UID: \"f6d67ddb-79d6-44bf-9e44-a7aef71c393e\") " Aug 04 11:39:49.435448 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.435426 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6d67ddb-79d6-44bf-9e44-a7aef71c393e" (UID: "f6d67ddb-79d6-44bf-9e44-a7aef71c393e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:39:49.437341 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.437316 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6d67ddb-79d6-44bf-9e44-a7aef71c393e" (UID: "f6d67ddb-79d6-44bf-9e44-a7aef71c393e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:39:49.437627 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.437597 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-kube-api-access-dpvmk" (OuterVolumeSpecName: "kube-api-access-dpvmk") pod "f6d67ddb-79d6-44bf-9e44-a7aef71c393e" (UID: "f6d67ddb-79d6-44bf-9e44-a7aef71c393e"). InnerVolumeSpecName "kube-api-access-dpvmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:39:49.439254 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.439232 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f0844e4-2130-4f8a-a637-9c4f55498f75" path="/var/lib/kubelet/pods/9f0844e4-2130-4f8a-a637-9c4f55498f75/volumes" Aug 04 11:39:49.439533 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.439518 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8189562-2ec7-41f6-a500-b720098b0759" path="/var/lib/kubelet/pods/c8189562-2ec7-41f6-a500-b720098b0759/volumes" Aug 04 11:39:49.458684 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.458651 2576 generic.go:356] "Generic (PLEG): container finished" podID="f6d67ddb-79d6-44bf-9e44-a7aef71c393e" containerID="22f5222a16df153869fcab808eb73d41999ec1863c2195098513d702a13387bf" exitCode=0 Aug 04 11:39:49.458855 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.458710 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" Aug 04 11:39:49.458855 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.458731 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" event={"ID":"f6d67ddb-79d6-44bf-9e44-a7aef71c393e","Type":"ContainerDied","Data":"22f5222a16df153869fcab808eb73d41999ec1863c2195098513d702a13387bf"} Aug 04 11:39:49.458855 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.458822 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56" event={"ID":"f6d67ddb-79d6-44bf-9e44-a7aef71c393e","Type":"ContainerDied","Data":"2aff506c674faa26e505263a8f514ee4b91d19707284cd99eb6b46053bac2ff4"} Aug 04 11:39:49.458855 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.458852 2576 scope.go:117] "RemoveContainer" containerID="22f5222a16df153869fcab808eb73d41999ec1863c2195098513d702a13387bf" Aug 04 11:39:49.469237 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.469212 2576 scope.go:117] "RemoveContainer" containerID="22f5222a16df153869fcab808eb73d41999ec1863c2195098513d702a13387bf" Aug 04 11:39:49.469607 ip-10-0-131-159 kubenswrapper[2576]: E0804 11:39:49.469579 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f5222a16df153869fcab808eb73d41999ec1863c2195098513d702a13387bf\": container with ID starting with 22f5222a16df153869fcab808eb73d41999ec1863c2195098513d702a13387bf not found: ID does not exist" containerID="22f5222a16df153869fcab808eb73d41999ec1863c2195098513d702a13387bf" Aug 04 11:39:49.469688 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.469619 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f5222a16df153869fcab808eb73d41999ec1863c2195098513d702a13387bf"} err="failed to get container status \"22f5222a16df153869fcab808eb73d41999ec1863c2195098513d702a13387bf\": rpc error: code = NotFound desc = could not find container \"22f5222a16df153869fcab808eb73d41999ec1863c2195098513d702a13387bf\": container with ID starting with 22f5222a16df153869fcab808eb73d41999ec1863c2195098513d702a13387bf not found: ID does not exist" Aug 04 11:39:49.478633 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.478607 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56"] Aug 04 11:39:49.484373 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.484349 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c74755-86b9d458dc-tfk56"] Aug 04 11:39:49.537448 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.537408 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpvmk\" (UniqueName: \"kubernetes.io/projected/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-kube-api-access-dpvmk\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:39:49.537448 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.537447 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-maas-api-tls\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:39:49.537709 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.537462 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6d67ddb-79d6-44bf-9e44-a7aef71c393e-openshift-service-ca\") on node \"ip-10-0-131-159.ec2.internal\" DevicePath \"\"" Aug 04 11:39:49.651871 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:49.651835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:51.293607 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:51.293571 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-x2gkr"] Aug 04 11:39:51.437387 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:51.437356 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6d67ddb-79d6-44bf-9e44-a7aef71c393e" path="/var/lib/kubelet/pods/f6d67ddb-79d6-44bf-9e44-a7aef71c393e/volumes" Aug 04 11:39:52.021441 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:52.021410 2576 scope.go:117] "RemoveContainer" containerID="aac725c9cc8cc164acb723ae08c72902d4d1b464ad2c6fc215db8e6c44cbcff9" Aug 04 11:39:52.030283 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:52.030261 2576 scope.go:117] "RemoveContainer" containerID="2e7a254ee6af4a7a5d50e6fb0704e2bd1eb199761e38d261d0281916f07d9661" Aug 04 11:39:55.965469 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:55.965428 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:56.029907 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:56.029865 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:56.232255 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:56.232139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:56.460861 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:56.460831 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:39:56.861551 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:39:56.861510 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-854cb55995-l5hbl_43b4b83d-a497-4006-a752-6d5fa2de3f3a/authorino/0.log" Aug 04 11:40:02.607799 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:02.607756 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hvdmm_e7aef772-361d-453a-af5c-03e4b3a87619/manager/0.log" Aug 04 11:40:02.973272 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:02.973180 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79895c7c95-vk94g_554f951a-7a34-4831-b714-3e4a5d9f2528/copy-manifests/0.log" Aug 04 11:40:02.983308 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:02.983273 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79895c7c95-vk94g_554f951a-7a34-4831-b714-3e4a5d9f2528/manager/0.log" Aug 04 11:40:03.435867 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:03.435825 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764020-m8p6r_ef3116da-8cc0-40ad-8795-40cdb70741a7/cleanup/0.log" Aug 04 11:40:03.540418 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:03.540388 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764035-85sd4_892fea7e-b7bf-4516-aac2-2b7520c2c927/cleanup/0.log" Aug 04 11:40:03.647406 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:03.647381 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764050-xv6m5_d7380efc-86ca-45c0-98ea-21fb4961d7fc/cleanup/0.log" Aug 04 11:40:04.483890 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:04.483854 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-854cb55995-l5hbl_43b4b83d-a497-4006-a752-6d5fa2de3f3a/authorino/0.log" Aug 04 11:40:04.700234 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:04.700198 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-zl75f_8c56e508-b1b2-4591-9442-cc5568f3fc0e/manager/0.log" Aug 04 11:40:05.045487 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:05.045455 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-x2gkr_8ed8ffdb-754b-4bed-9124-cc309f57cf04/limitador/0.log" Aug 04 11:40:05.722220 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:05.722184 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bc4f665b9-dmdks_0267bd4e-d9ea-4efb-b847-5b0c9d8ae507/kube-auth-proxy/0.log" Aug 04 11:40:06.289175 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:06.289138 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-798f494f76-94bvh_50e09e74-c462-4d60-a61a-b281bf126d83/router/0.log" Aug 04 11:40:12.814587 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.814552 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lm9vx/must-gather-llljr"] Aug 04 11:40:12.815141 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.814957 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6d67ddb-79d6-44bf-9e44-a7aef71c393e" containerName="maas-api" Aug 04 11:40:12.815141 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.814969 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6d67ddb-79d6-44bf-9e44-a7aef71c393e" containerName="maas-api" Aug 04 11:40:12.815141 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.814998 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0844e4-2130-4f8a-a637-9c4f55498f75" containerName="revoke-keys" Aug 04 11:40:12.815141 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.815005 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0844e4-2130-4f8a-a637-9c4f55498f75" containerName="revoke-keys" Aug 04 11:40:12.815141 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.815061 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6d67ddb-79d6-44bf-9e44-a7aef71c393e" containerName="maas-api" Aug 04 11:40:12.815141 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.815068 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f0844e4-2130-4f8a-a637-9c4f55498f75" containerName="revoke-keys" Aug 04 11:40:12.818107 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.818088 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lm9vx/must-gather-llljr" Aug 04 11:40:12.820591 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.820567 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lm9vx\"/\"kube-root-ca.crt\"" Aug 04 11:40:12.820721 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.820624 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lm9vx\"/\"default-dockercfg-jmtxh\"" Aug 04 11:40:12.820771 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.820745 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lm9vx\"/\"openshift-service-ca.crt\"" Aug 04 11:40:12.833576 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.833539 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lm9vx/must-gather-llljr"] Aug 04 11:40:12.843447 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.843401 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:40:12.942642 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.942591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:40:12.948888 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.948844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f5554c27-4399-41b6-9747-ea4717cca6d0-must-gather-output\") pod \"must-gather-llljr\" (UID: \"f5554c27-4399-41b6-9747-ea4717cca6d0\") " pod="openshift-must-gather-lm9vx/must-gather-llljr" Aug 04 11:40:12.949053 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:12.948913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdmnm\" (UniqueName: \"kubernetes.io/projected/f5554c27-4399-41b6-9747-ea4717cca6d0-kube-api-access-qdmnm\") pod \"must-gather-llljr\" (UID: \"f5554c27-4399-41b6-9747-ea4717cca6d0\") " pod="openshift-must-gather-lm9vx/must-gather-llljr" Aug 04 11:40:13.049815 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:13.049776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f5554c27-4399-41b6-9747-ea4717cca6d0-must-gather-output\") pod \"must-gather-llljr\" (UID: \"f5554c27-4399-41b6-9747-ea4717cca6d0\") " pod="openshift-must-gather-lm9vx/must-gather-llljr" Aug 04 11:40:13.049985 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:13.049824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdmnm\" (UniqueName: \"kubernetes.io/projected/f5554c27-4399-41b6-9747-ea4717cca6d0-kube-api-access-qdmnm\") pod \"must-gather-llljr\" (UID: \"f5554c27-4399-41b6-9747-ea4717cca6d0\") " pod="openshift-must-gather-lm9vx/must-gather-llljr" Aug 04 11:40:13.050143 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:13.050124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f5554c27-4399-41b6-9747-ea4717cca6d0-must-gather-output\") pod \"must-gather-llljr\" (UID: \"f5554c27-4399-41b6-9747-ea4717cca6d0\") " pod="openshift-must-gather-lm9vx/must-gather-llljr" Aug 04 11:40:13.059756 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:13.059720 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdmnm\" (UniqueName: \"kubernetes.io/projected/f5554c27-4399-41b6-9747-ea4717cca6d0-kube-api-access-qdmnm\") pod \"must-gather-llljr\" (UID: \"f5554c27-4399-41b6-9747-ea4717cca6d0\") " pod="openshift-must-gather-lm9vx/must-gather-llljr" Aug 04 11:40:13.127320 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:13.127286 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lm9vx/must-gather-llljr" Aug 04 11:40:13.273301 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:13.273274 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lm9vx/must-gather-llljr"] Aug 04 11:40:13.275654 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:40:13.275624 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5554c27_4399_41b6_9747_ea4717cca6d0.slice/crio-e7e865b0353da8033524e6950de198d51512fab475aa55e96ec33d763de073dd WatchSource:0}: Error finding container e7e865b0353da8033524e6950de198d51512fab475aa55e96ec33d763de073dd: Status 404 returned error can't find the container with id e7e865b0353da8033524e6950de198d51512fab475aa55e96ec33d763de073dd Aug 04 11:40:13.599397 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:13.599358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lm9vx/must-gather-llljr" event={"ID":"f5554c27-4399-41b6-9747-ea4717cca6d0","Type":"ContainerStarted","Data":"e7e865b0353da8033524e6950de198d51512fab475aa55e96ec33d763de073dd"} Aug 04 11:40:14.608150 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:14.608109 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lm9vx/must-gather-llljr" event={"ID":"f5554c27-4399-41b6-9747-ea4717cca6d0","Type":"ContainerStarted","Data":"e7f72b987b25b878d2ee7b9d70e17a20bc112cabe93de01b63c40cbe3c956e9c"} Aug 04 11:40:14.608598 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:14.608164 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lm9vx/must-gather-llljr" event={"ID":"f5554c27-4399-41b6-9747-ea4717cca6d0","Type":"ContainerStarted","Data":"dc98700cdc515f5528c3ef9c1445b0322ecdb2816626436034282cff7b9689b6"} Aug 04 11:40:15.762859 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:15.762825 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gr6kd_0d71b33c-089f-4d23-be91-d48f28a9c333/global-pull-secret-syncer/0.log" Aug 04 11:40:15.869005 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:15.868975 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" Aug 04 11:40:15.982332 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:15.982300 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-159.ec2.internal_57e7e3404a3ff3539a1dc706d1378691/haproxy/0.log" Aug 04 11:40:20.456817 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:20.456494 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-854cb55995-l5hbl_43b4b83d-a497-4006-a752-6d5fa2de3f3a/authorino/0.log" Aug 04 11:40:20.532360 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:20.532333 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-zl75f_8c56e508-b1b2-4591-9442-cc5568f3fc0e/manager/0.log" Aug 04 11:40:20.736981 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:20.736899 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-x2gkr_8ed8ffdb-754b-4bed-9124-cc309f57cf04/limitador/0.log" Aug 04 11:40:22.311591 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:22.311554 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5bedc61f-96d3-4f61-b87c-5db98c787dea/alertmanager/0.log" Aug 04 11:40:22.338468 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:22.338439 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5bedc61f-96d3-4f61-b87c-5db98c787dea/config-reloader/0.log" Aug 04 11:40:22.366821 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:22.366784 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5bedc61f-96d3-4f61-b87c-5db98c787dea/kube-rbac-proxy-web/0.log" Aug 04 11:40:22.394586 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:22.394554 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5bedc61f-96d3-4f61-b87c-5db98c787dea/kube-rbac-proxy/0.log" Aug 04 11:40:22.428326 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:22.428227 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5bedc61f-96d3-4f61-b87c-5db98c787dea/kube-rbac-proxy-metric/0.log" Aug 04 11:40:22.459753 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:22.459718 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5bedc61f-96d3-4f61-b87c-5db98c787dea/prom-label-proxy/0.log" Aug 04 11:40:22.488996 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:22.488935 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5bedc61f-96d3-4f61-b87c-5db98c787dea/init-config-reloader/0.log" Aug 04 11:40:22.543560 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:22.543519 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-bfjqd_97e701ea-3b55-4c01-8e14-695a759a7ad8/cluster-monitoring-operator/0.log" Aug 04 11:40:22.691422 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:22.691393 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-762hd_e4af0c2f-759e-47b4-a24e-a2771c9434d9/monitoring-plugin/0.log" Aug 04 11:40:22.742394 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:22.742342 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blsdj_380264cd-6db8-4b2a-bd74-a78aed669c40/node-exporter/0.log" Aug 04 11:40:22.769584 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:22.769557 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blsdj_380264cd-6db8-4b2a-bd74-a78aed669c40/kube-rbac-proxy/0.log" Aug 04 11:40:22.798957 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:22.798931 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blsdj_380264cd-6db8-4b2a-bd74-a78aed669c40/init-textfile/0.log" Aug 04 11:40:23.377762 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:23.377682 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-hm4tg_5be3683e-4567-4763-9475-960ebec0e2ec/prometheus-operator/0.log" Aug 04 11:40:23.398401 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:23.398365 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-hm4tg_5be3683e-4567-4763-9475-960ebec0e2ec/kube-rbac-proxy/0.log" Aug 04 11:40:23.426433 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:23.426371 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-7t7t5_763d8cd2-4479-4fb4-8107-af81b4bdc56b/prometheus-operator-admission-webhook/0.log" Aug 04 11:40:23.460732 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:23.460695 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6576c8665-c8j66_4a3898d6-385b-4066-b9fb-3054f2d4c29d/telemeter-client/0.log" Aug 04 11:40:23.486482 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:23.486453 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6576c8665-c8j66_4a3898d6-385b-4066-b9fb-3054f2d4c29d/reload/0.log" Aug 04 11:40:23.511790 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:23.511757 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6576c8665-c8j66_4a3898d6-385b-4066-b9fb-3054f2d4c29d/kube-rbac-proxy/0.log" Aug 04 11:40:23.555278 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:23.555249 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78df79b496-ssxqq_a0183d10-099c-4b35-bfd5-4729f172fffd/thanos-query/0.log" Aug 04 11:40:23.583362 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:23.583309 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78df79b496-ssxqq_a0183d10-099c-4b35-bfd5-4729f172fffd/kube-rbac-proxy-web/0.log" Aug 04 11:40:23.613064 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:23.613034 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78df79b496-ssxqq_a0183d10-099c-4b35-bfd5-4729f172fffd/kube-rbac-proxy/0.log" Aug 04 11:40:23.640561 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:23.640361 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78df79b496-ssxqq_a0183d10-099c-4b35-bfd5-4729f172fffd/prom-label-proxy/0.log" Aug 04 11:40:23.669666 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:23.669631 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78df79b496-ssxqq_a0183d10-099c-4b35-bfd5-4729f172fffd/kube-rbac-proxy-rules/0.log" Aug 04 11:40:23.697523 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:23.697205 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78df79b496-ssxqq_a0183d10-099c-4b35-bfd5-4729f172fffd/kube-rbac-proxy-metrics/0.log" Aug 04 11:40:24.046933 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:24.046837 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:40:24.126194 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:24.126151 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8qppj_4d7923f6-df0e-4f83-b6da-34d93041a405/konnectivity-agent/0.log" limit=1024 Aug 04 11:40:25.125870 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:25.125831 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-9l97v_6737f3c7-d910-4cb5-a73e-3eea0bab64e6/networking-console-plugin/0.log" Aug 04 11:40:25.969671 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:25.969605 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lm9vx/must-gather-llljr" podStartSLOduration=13.154579898 podStartE2EDuration="13.969580779s" totalImagesPullingTime="815.000881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:40:14.630961812 +0000 UTC m=+3803.728755207" watchObservedRunningTime="2026-08-04 11:40:25.969580779 +0000 UTC m=+3815.067374173" Aug 04 11:40:25.973417 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:25.973385 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l"] Aug 04 11:40:25.980100 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:25.980066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:25.985989 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:25.985956 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l"] Aug 04 11:40:26.088987 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.088939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg2v8\" (UniqueName: \"kubernetes.io/projected/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-kube-api-access-bg2v8\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.089178 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.089050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-sys\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.089178 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.089101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-lib-modules\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.089178 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.089126 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-proc\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.089178 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.089178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-podres\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.190211 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.190180 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-lib-modules\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.190623 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.190271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-proc\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.190623 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.190348 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-proc\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.190623 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.190360 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-lib-modules\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.190623 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.190348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-podres\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.190623 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.190426 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-podres\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.190623 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.190433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg2v8\" (UniqueName: \"kubernetes.io/projected/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-kube-api-access-bg2v8\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.190623 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.190577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-sys\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.190937 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.190660 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-sys\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.201638 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.201601 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg2v8\" (UniqueName: \"kubernetes.io/projected/4f9f27c9-9a33-41b5-be8c-8b4b458581fe-kube-api-access-bg2v8\") pod \"perf-node-gather-daemonset-rv22l\" (UID: \"4f9f27c9-9a33-41b5-be8c-8b4b458581fe\") " pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.291677 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.291590 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.470015 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.469988 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l"] Aug 04 11:40:26.471594 ip-10-0-131-159 kubenswrapper[2576]: W0804 11:40:26.471558 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4f9f27c9_9a33_41b5_be8c_8b4b458581fe.slice/crio-194a60e5a1dae14c39151b214d0e2bcfe41dfe293afc09b7b958a74299819653 WatchSource:0}: Error finding container 194a60e5a1dae14c39151b214d0e2bcfe41dfe293afc09b7b958a74299819653: Status 404 returned error can't find the container with id 194a60e5a1dae14c39151b214d0e2bcfe41dfe293afc09b7b958a74299819653 Aug 04 11:40:26.720112 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.720073 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" event={"ID":"4f9f27c9-9a33-41b5-be8c-8b4b458581fe","Type":"ContainerStarted","Data":"a08dc80de2010c4a0d11802395c22575dcdad0da9cb8682250a652c6f649653a"} Aug 04 11:40:26.720310 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.720139 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" event={"ID":"4f9f27c9-9a33-41b5-be8c-8b4b458581fe","Type":"ContainerStarted","Data":"194a60e5a1dae14c39151b214d0e2bcfe41dfe293afc09b7b958a74299819653"} Aug 04 11:40:26.720310 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.720168 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:26.739940 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:26.739879 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" podStartSLOduration=1.739861508 podStartE2EDuration="1.739861508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 11:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:40:26.737091156 +0000 UTC m=+3815.834884551" watchObservedRunningTime="2026-08-04 11:40:26.739861508 +0000 UTC m=+3815.837654880" Aug 04 11:40:28.401359 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:28.401325 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vlqv6_683c4f2b-d66c-4b5c-b6ee-96d4007f49dc/dns/0.log" Aug 04 11:40:28.425831 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:28.425800 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vlqv6_683c4f2b-d66c-4b5c-b6ee-96d4007f49dc/kube-rbac-proxy/0.log" Aug 04 11:40:28.513450 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:28.513425 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r9kvv_7b19bc6d-13f7-4db1-bc59-34eba05608ff/dns-node-resolver/0.log" Aug 04 11:40:29.086251 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:29.086214 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84bbdfb5b5-grjg4_853f18a6-1400-45ea-964a-fbb87c0a67bc/registry/0.log" Aug 04 11:40:29.111040 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:29.111010 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c9kzp_fa44517a-0e03-4dad-b3b0-79ee2001d9cd/node-ca/0.log" Aug 04 11:40:30.536972 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:30.536943 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bc4f665b9-dmdks_0267bd4e-d9ea-4efb-b847-5b0c9d8ae507/kube-auth-proxy/0.log" Aug 04 11:40:30.726137 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:30.726113 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-798f494f76-94bvh_50e09e74-c462-4d60-a61a-b281bf126d83/router/0.log" Aug 04 11:40:31.637426 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:31.637396 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xwpcd_b86857b2-2acf-4829-b4fb-2d5ac4bf5236/serve-healthcheck-canary/0.log" Aug 04 11:40:32.358354 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:32.358329 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ntr5t_04538f9b-1297-4c97-a89c-d68686229e69/kube-rbac-proxy/0.log" Aug 04 11:40:32.382796 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:32.382761 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ntr5t_04538f9b-1297-4c97-a89c-d68686229e69/exporter/0.log" Aug 04 11:40:32.410339 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:32.410311 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ntr5t_04538f9b-1297-4c97-a89c-d68686229e69/extractor/0.log" Aug 04 11:40:32.740011 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:32.739924 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lm9vx/perf-node-gather-daemonset-rv22l" Aug 04 11:40:35.463709 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:35.463670 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hvdmm_e7aef772-361d-453a-af5c-03e4b3a87619/manager/0.log" Aug 04 11:40:35.751955 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:35.751873 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79895c7c95-vk94g_554f951a-7a34-4831-b714-3e4a5d9f2528/manager/0.log" Aug 04 11:40:35.779071 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:35.779044 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79895c7c95-vk94g_554f951a-7a34-4831-b714-3e4a5d9f2528/copy-manifests/0.log" Aug 04 11:40:37.028569 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:37.028543 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b459fcc7c-k9r7q_2178a1c3-6a02-4bb9-9a67-c59e239888c8/manager/0.log" Aug 04 11:40:37.086215 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:37.086184 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-stpb8_911f00c5-a0a1-4b20-9d65-146bddbf69f3/openshift-lws-operator/0.log" Aug 04 11:40:37.319417 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:37.319329 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bc4f665b9-dmdks_0267bd4e-d9ea-4efb-b847-5b0c9d8ae507/kube-auth-proxy/0.log" Aug 04 11:40:37.446624 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:37.446594 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-798f494f76-94bvh_50e09e74-c462-4d60-a61a-b281bf126d83/router/0.log" Aug 04 11:40:39.204665 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:39.204625 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bc4f665b9-dmdks_0267bd4e-d9ea-4efb-b847-5b0c9d8ae507/kube-auth-proxy/0.log" Aug 04 11:40:39.373770 ip-10-0-131-159 kubenswrapper[2576]: I0804 11:40:39.373740 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-798f494f76-94bvh_50e09e74-c462-4d60-a61a-b281bf126d83/router/0.log"