Jul 27 23:16:16.941690 ip-10-0-142-53 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 23:16:17.320215 ip-10-0-142-53 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 23:16:17.320215 ip-10-0-142-53 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 23:16:17.320215 ip-10-0-142-53 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 23:16:17.320215 ip-10-0-142-53 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 23:16:17.320215 ip-10-0-142-53 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 23:16:17.321163 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.321045    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 23:16:17.325644 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325630    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 23:16:17.325644 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325644    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325648    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325651    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325654    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325658    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325661    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325663    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325666    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325668    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325671    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325673    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325676    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325678    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325681    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325684    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325687    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325689    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325691    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325694    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325697    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 23:16:17.325704 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325699    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325702    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325704    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325712    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325715    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325718    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325721    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325723    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325726    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325728    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325730    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325733    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325737    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325739    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325742    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325745    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325747    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325749    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325752    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325754    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 23:16:17.326166 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325756    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325759    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325761    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325763    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325766    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325768    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325772    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325776    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325779    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325781    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325784    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325786    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325788    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325791    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325794    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325797    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325800    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325802    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325805    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325807    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 23:16:17.326649 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325810    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325812    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325815    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325817    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325820    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325823    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325825    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325828    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325830    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325833    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325835    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325838    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325840    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325842    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325845    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325848    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325850    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325852    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325855    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 23:16:17.327118 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325859    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325863    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325866    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325869    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325871    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325874    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325877    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325881    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325884    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325886    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325889    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325891    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325894    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325896    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325899    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325901    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325904    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325906    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.325910    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 23:16:17.327580 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326627    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326635    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326638    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326641    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326644    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326648    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326651    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326654    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326656    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326659    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326662    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326665    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326668    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326670    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326672    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326675    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326678    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326680    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326683    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326685    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 23:16:17.328020 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326688    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326691    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326693    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326696    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326698    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326701    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326703    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326706    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326708    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326710    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326713    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326715    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326717    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326720    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326722    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326724    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326727    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326729    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326732    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326734    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 23:16:17.328510 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326737    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326739    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326742    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326744    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326746    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326749    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326751    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326754    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326758    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326761    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326763    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326765    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326769    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326772    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326774    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326776    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326779    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326781    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326784    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326786    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 23:16:17.328975 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326789    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326792    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326794    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326796    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326800    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326804    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326807    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326809    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326812    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326814    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326817    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326820    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326822    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326825    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326827    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326829    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326833    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326835    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326838    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 23:16:17.329467 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326840    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326842    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326845    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326847    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326850    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326853    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326855    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326858    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326860    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326862    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326864    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326867    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326869    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326871    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326874    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326877    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326879    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326882    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326886    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.326888    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 23:16:17.329909 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327246    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327251    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327254    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327257    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327260    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327263    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327265    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327268    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327270    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327272    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327275    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327277    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327280    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327282    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327284    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327287    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327289    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327294    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327297    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 23:16:17.330429 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327299    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327302    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327305    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327307    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327310    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327312    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327315    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327317    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327320    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327322    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327324    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327327    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327330    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327332    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327334    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327337    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327339    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327342    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327345    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327347    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 23:16:17.330882 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327350    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327353    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327355    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327357    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327360    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327362    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327365    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327385    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327388    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327390    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327394    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327397    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327399    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327403    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327406    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327409    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327411    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327414    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327430    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327435    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 23:16:17.331344 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327440    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327443    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327448    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327452    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327455    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327458    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327460    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327464    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327468    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327471    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327473    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327477    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327479    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327482    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327484    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327487    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327489    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327492    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327494    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327497    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 23:16:17.331888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327499    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327502    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327507    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327510    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327512    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327515    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327520    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327523    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327526    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327528    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327531    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327534    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327536    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327538    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327541    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327544    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327548    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327551    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327553    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 23:16:17.332353 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.327556    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327629    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327640    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327649    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327655    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327661    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327666    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327673    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327678    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327681    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327684    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327687    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327690    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327693    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327696    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327701    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327704    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327707    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.327710    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328467    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328471    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328474    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328478    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328481    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 23:16:17.332811 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328485    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328488    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328491    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328494    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328497    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328500    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328503    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328506    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328509    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328513    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328516    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328519    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328522    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328525    2559 flags.go:64] FLAG: --enable-server="true"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328529    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328533    2559 flags.go:64] FLAG: --event-burst="100"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328536    2559 flags.go:64] FLAG: --event-qps="50"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328539    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328542    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328544    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328548    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328550    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328553    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328557    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328561    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 23:16:17.333429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328564    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328567    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328569    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328572    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328575    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328578    2559 flags.go:64] FLAG: --feature-gates=""
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328581    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328584    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328587    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328590    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328593    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328596    2559 flags.go:64] FLAG: --help="false"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328598    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328601    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328604    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328607    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328610    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328613    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328616    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328619    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328621    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328625    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328628    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328631    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 23:16:17.334062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328634    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328637    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328639    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328642    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328645    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328648    2559 flags.go:64] FLAG: --lock-file=""
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328652    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328655    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328657    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328662    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328665    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328668    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328671    2559 flags.go:64] FLAG: --logging-format="text"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328673    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328677    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328679    2559 flags.go:64] FLAG: --manifest-url=""
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328682    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328686    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328690    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328694    2559 flags.go:64] FLAG: --max-pods="110"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328696    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328700    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328702    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328705    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328708    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 23:16:17.334659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328711    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328714    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328720    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328723    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328726    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328729    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328732    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328737    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328740    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328743    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328745    2559 flags.go:64] FLAG: --port="10250"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328748    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328751    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04fd0d303edf5e4ab"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328754    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328759    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328761    2559 flags.go:64] FLAG: --register-node="true"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328764    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328768    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328771    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328773    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328776    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328779    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328782    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328785    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328788    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 23:16:17.335276 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328791    2559 flags.go:64] FLAG: --runonce="false"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328793    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328796    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328800    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328803    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328806    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328809    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328812    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328815    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328817    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328821    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328823    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328826    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328829    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328833    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328835    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328841    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328843    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328846    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328851    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328853    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328857    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328859    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328862    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328865    2559 flags.go:64] FLAG: --v="2"
Jul 27 23:16:17.335896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328869    2559 flags.go:64] FLAG: --version="false"
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328872    2559 flags.go:64] FLAG: --vmodule=""
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328879    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.328882    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.328998    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329003    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329008    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329011    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329014    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329017    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329022    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329025    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329027    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329030    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329032    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329035    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329038    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329042    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329045    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 23:16:17.336537 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329048    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329051    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329054    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329057    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329060    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329062    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329065    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329067    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329070    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329073    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329075    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329078    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329081    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329083    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329085    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329088    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329090    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329092    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329095    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329097    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 23:16:17.336972 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329099    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329101    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329104    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329107    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329110    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329112    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329114    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329117    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329119    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329121    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329124    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329126    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329128    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329131    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329134    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329136    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329139    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329141    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329144    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 23:16:17.337502 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329146    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329148    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329151    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329153    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329156    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329158    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329161    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329163    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329166    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329168    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329171    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329173    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329176    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329178    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329180    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329183    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329186    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329189    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329191    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329194    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 23:16:17.337959 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329196    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329198    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329201    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329203    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329206    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329208    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329210    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329213    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329215    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329217    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329220    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329222    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329225    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329227    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329230    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329232    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329234    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329237    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329239    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329242    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 23:16:17.338460 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329244    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329247    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329249    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329251    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.329254    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.329737    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.336939    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.336954    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.336972    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.336978    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337043    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337048    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337051    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337054    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337057    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 23:16:17.338943 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337060    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337063    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337066    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337069    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337071    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337074    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337077    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337079    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337082    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337084    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337087    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337089    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337092    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337094    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337097    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337099    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337102    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337104    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337106    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337109    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 23:16:17.339311 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337111    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337114    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337116    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337119    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337123    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337125    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337128    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337130    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337133    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337135    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337138    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337140    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337144    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337147    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337150    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337152    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337155    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337157    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337160    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337162    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 23:16:17.339825 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337165    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337167    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337170    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337172    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337175    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337177    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337180    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337182    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337184    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337187    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337189    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337192    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337194    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337196    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337199    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337202    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337205    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337208    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337211    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337213    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 23:16:17.340335 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337216    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337218    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337221    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337223    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337225    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337228    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337231    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337233    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337236    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337238    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337240    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337243    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337245    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337248    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337250    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337252    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337255    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337257    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337259    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 23:16:17.340873 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337263    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 23:16:17.341350 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337265    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 23:16:17.341350 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337267    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 23:16:17.341350 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337270    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 23:16:17.341350 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337272    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 23:16:17.341350 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337276    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 23:16:17.341350 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337279    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 23:16:17.341350 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337282    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 23:16:17.341350 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337285    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 23:16:17.341350 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337288    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 23:16:17.341350 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337291    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 23:16:17.341350 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337294    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 23:16:17.341350 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337297    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 23:16:17.341350 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337301    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 23:16:17.341350 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337303    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 23:16:17.341770 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.337308    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 23:16:17.341770 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337452    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 23:16:17.341770 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337457    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 23:16:17.341770 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337460    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 23:16:17.341770 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337463    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 23:16:17.341770 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337466    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 23:16:17.341770 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337468    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 23:16:17.341770 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337470    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 23:16:17.341770 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337473    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 23:16:17.341770 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337477    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 23:16:17.341770 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337481    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 23:16:17.341770 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337484    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 23:16:17.341770 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337486    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 23:16:17.341770 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337489    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337491    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337494    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337496    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337498    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337501    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337503    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337506    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337508    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337511    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337514    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337516    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337519    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337522    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337524    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337527    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337530    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337533    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337535    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337537    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 23:16:17.342123 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337539    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337542    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337544    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337547    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337549    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337551    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337555    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337559    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337562    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337564    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337567    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337570    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337572    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337575    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337578    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337580    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337583    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337585    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337587    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 23:16:17.342618 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337590    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337592    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337595    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337597    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337599    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337602    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337604    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337607    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337610    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337612    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337615    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337617    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337620    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337622    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337624    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337627    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337629    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337631    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337634    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337636    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 23:16:17.343155 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337638    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337641    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337643    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337646    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337648    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337650    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337653    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337655    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337657    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337660    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337662    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337665    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337667    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337669    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337672    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337674    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337676    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337679    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337681    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337684    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 23:16:17.343974 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337687    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 23:16:17.344615 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337689    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 23:16:17.344615 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337692    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 23:16:17.344615 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337696    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 23:16:17.344615 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337698    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 23:16:17.344615 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337701    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 23:16:17.344615 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337703    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 23:16:17.344615 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:17.337706    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 23:16:17.344615 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.337711    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 23:16:17.344615 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.337817    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 23:16:17.344615 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.340681    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 23:16:17.344615 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.342119    2559 server.go:1019] "Starting client certificate rotation"
Jul 27 23:16:17.344615 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.342217    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 23:16:17.344615 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.342248    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 23:16:17.345884 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.345867    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 23:16:17.348600 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.348584    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 23:16:17.357543 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.357526    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 27 23:16:17.361736 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.361718    2559 log.go:25] "Validated CRI v1 image API"
Jul 27 23:16:17.362620 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.362601    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 23:16:17.366652 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.366634    2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8b9084f0-1c9f-4b3f-898a-8927ee22f757:/dev/nvme0n1p4 b183dcf9-801f-401d-86ed-13ccc953f4fe:/dev/nvme0n1p3]
Jul 27 23:16:17.366703 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.366653    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 23:16:17.372264 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.372143    2559 manager.go:216] Machine: {Timestamp:2026-07-27 23:16:17.37088825 +0000 UTC m=+0.315118989 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3196598 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29f74ed57934bb8c3033da2dd7f33b SystemUUID:ec29f74e-d579-34bb-8c30-33da2dd7f33b BootID:d87e6ab1-2be4-423f-89ee-c45dcafddbe6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a2:fd:eb:df:79 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a2:fd:eb:df:79 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:59:6f:c7:24:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 23:16:17.372264 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.372258    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 23:16:17.372397 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.372337    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 23:16:17.373566 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.373551    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 23:16:17.375408 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.375364    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 23:16:17.375567 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.375413    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-53.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 23:16:17.376042 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.376030    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 23:16:17.376097 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.376049    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 23:16:17.376097 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.376059    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 23:16:17.376713 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.376700    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 23:16:17.377830 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.377819    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 23:16:17.380631 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.380618    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 23:16:17.380796 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.380785    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 23:16:17.380862 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.380823    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 23:16:17.380862 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.380838    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 23:16:17.380862 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.380853    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 23:16:17.380976 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.380865    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 27 23:16:17.380976 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.380878    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 23:16:17.381236 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.381221    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 23:16:17.381292 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.381245    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 23:16:17.384413 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.384393    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 23:16:17.386647 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.386634    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 23:16:17.386700 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.386663    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 23:16:17.388712 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.388699    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 23:16:17.388794 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.388717    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 23:16:17.388794 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.388724    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 23:16:17.388794 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.388731    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 23:16:17.388794 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.388741    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 23:16:17.388794 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.388747    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 23:16:17.388794 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.388756    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 23:16:17.388794 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.388763    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 23:16:17.388794 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.388771    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 23:16:17.388794 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.388777    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 23:16:17.388794 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.388787    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 23:16:17.388794 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.388793    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 23:16:17.389332 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.389322    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 23:16:17.389332 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.389332    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 23:16:17.390542 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.390524    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5dpj"
Jul 27 23:16:17.390744 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.390711    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 23:16:17.390834 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.390711    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-53.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 23:16:17.392699 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.392682    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-53.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 23:16:17.392803 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.392793    2559 server.go:1268] "Started kubelet"
Jul 27 23:16:17.392878 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.392856    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 23:16:17.392964 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.392918    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 23:16:17.393011 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.392982    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 23:16:17.393613 ip-10-0-142-53 systemd[1]: Started Kubernetes Kubelet.
Jul 27 23:16:17.394824 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.394806    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 23:16:17.394910 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.394843    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 23:16:17.397404 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.397387    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5dpj"
Jul 27 23:16:17.398078 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.398059    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 23:16:17.399189 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.398078    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-53.ec2.internal.18c648a55e97675e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-53.ec2.internal,UID:ip-10-0-142-53.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-53.ec2.internal,},FirstTimestamp:2026-07-27 23:16:17.392764766 +0000 UTC m=+0.336995503,LastTimestamp:2026-07-27 23:16:17.392764766 +0000 UTC m=+0.336995503,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-53.ec2.internal,}"
Jul 27 23:16:17.405124 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.405105    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 23:16:17.405210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.405135    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 23:16:17.405324 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.405298    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
Jul 27 23:16:17.405889 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.405849    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 23:16:17.406197 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.406174    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 23:16:17.406197 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.406195    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 23:16:17.406360 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.406345    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 23:16:17.406360 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.406359    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 23:16:17.406470 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.406359    2559 factory.go:55] Registering systemd factory
Jul 27 23:16:17.406470 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.406401    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 27 23:16:17.406641 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.406626    2559 factory.go:153] Registering CRI-O factory
Jul 27 23:16:17.406641 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.406643    2559 factory.go:223] Registration of the crio container factory successfully
Jul 27 23:16:17.406759 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.406704    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 23:16:17.406759 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.406745    2559 factory.go:103] Registering Raw factory
Jul 27 23:16:17.406759 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.406755    2559 manager.go:1196] Started watching for new ooms in manager
Jul 27 23:16:17.407486 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.407447    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-53.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 23:16:17.407616 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.407539    2559 manager.go:349] Starting recovery of all containers
Jul 27 23:16:17.419558 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.419542    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 23:16:17.420617 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.420457    2559 manager.go:354] Recovery completed
Jul 27 23:16:17.424564 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.424551    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 23:16:17.426365 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.426349    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 23:16:17.426456 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.426428    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 23:16:17.426456 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.426440    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientPID"
Jul 27 23:16:17.426933 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.426919    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 23:16:17.426998 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.426934    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 23:16:17.426998 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.426953    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 23:16:17.427057 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.426925    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 23:16:17.427731 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.427718    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 23:16:17.427784 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.427743    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 23:16:17.427784 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.427754    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientPID"
Jul 27 23:16:17.429711 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.429700    2559 policy_none.go:49] "None policy: Start"
Jul 27 23:16:17.429745 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.429716    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 23:16:17.429745 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.429726    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 23:16:17.431855 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.431842    2559 policy_none.go:47] "Start"
Jul 27 23:16:17.472517 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.472504    2559 manager.go:337] "Starting Device Plugin manager"
Jul 27 23:16:17.472582 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.472533    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 23:16:17.472582 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.472543    2559 server.go:87] "Starting device plugin registration server"
Jul 27 23:16:17.472814 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.472801    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 23:16:17.472877 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.472822    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 23:16:17.472936 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.472897    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 23:16:17.473027 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.473010    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 23:16:17.473027 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.473025    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 23:16:17.473541 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.473510    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 23:16:17.473610 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.473566    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-53.ec2.internal\" not found"
Jul 27 23:16:17.555136 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.555102    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 23:16:17.557468 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.556358    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 23:16:17.557468 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.556407    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 23:16:17.557468 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.556430    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 23:16:17.557468 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.556466    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 23:16:17.558725 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.558704    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 23:16:17.573843 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.573790    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 23:16:17.574662 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.574639    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 23:16:17.574757 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.574673    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 23:16:17.574757 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.574687    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientPID"
Jul 27 23:16:17.574757 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.574710    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.580291 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.580274    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.580392 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.580296    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-53.ec2.internal\": node \"ip-10-0-142-53.ec2.internal\" not found"
Jul 27 23:16:17.590913 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.590895    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
Jul 27 23:16:17.657110 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.657066    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"]
Jul 27 23:16:17.657271 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.657147    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 23:16:17.658993 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.658975    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 23:16:17.659104 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.659004    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 23:16:17.659104 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.659014    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientPID"
Jul 27 23:16:17.661305 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.661293    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 23:16:17.661441 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.661413    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.661441 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.661440    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 23:16:17.662907 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.662887    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 23:16:17.663028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.662922    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 23:16:17.663028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.662888    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 23:16:17.663028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.662935    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientPID"
Jul 27 23:16:17.663028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.662976    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 23:16:17.663028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.662996    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientPID"
Jul 27 23:16:17.665209 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.665194    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.665259 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.665222    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 23:16:17.665892 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.665874    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 23:16:17.665965 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.665902    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 23:16:17.665965 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.665914    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientPID"
Jul 27 23:16:17.689015 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.688989    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-53.ec2.internal\" not found" node="ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.690978 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.690962    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
Jul 27 23:16:17.693498 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.693482    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-53.ec2.internal\" not found" node="ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.708517 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.708501    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f826d5b4e5f69d2c5ec64974c7fe045-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal\" (UID: \"2f826d5b4e5f69d2c5ec64974c7fe045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.708603 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.708524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a64776cf4968745c952331caeb9a6baf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-53.ec2.internal\" (UID: \"a64776cf4968745c952331caeb9a6baf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.708603 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.708558    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f826d5b4e5f69d2c5ec64974c7fe045-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal\" (UID: \"2f826d5b4e5f69d2c5ec64974c7fe045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.791886 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.791846    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
Jul 27 23:16:17.809228 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.809203    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a64776cf4968745c952331caeb9a6baf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-53.ec2.internal\" (UID: \"a64776cf4968745c952331caeb9a6baf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.809306 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.809245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a64776cf4968745c952331caeb9a6baf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-53.ec2.internal\" (UID: \"a64776cf4968745c952331caeb9a6baf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.809338 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.809290    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f826d5b4e5f69d2c5ec64974c7fe045-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal\" (UID: \"2f826d5b4e5f69d2c5ec64974c7fe045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.809338 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.809330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f826d5b4e5f69d2c5ec64974c7fe045-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal\" (UID: \"2f826d5b4e5f69d2c5ec64974c7fe045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.809425 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.809385    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f826d5b4e5f69d2c5ec64974c7fe045-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal\" (UID: \"2f826d5b4e5f69d2c5ec64974c7fe045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.809425 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.809396    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f826d5b4e5f69d2c5ec64974c7fe045-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal\" (UID: \"2f826d5b4e5f69d2c5ec64974c7fe045\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.892681 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.892612    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
Jul 27 23:16:17.992240 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.992193    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:17.993266 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:17.993247    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
Jul 27 23:16:17.997477 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:17.997266    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:18.094128 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:18.094085    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
Jul 27 23:16:18.195757 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:18.195720    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
Jul 27 23:16:18.296345 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:18.296312    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
Jul 27 23:16:18.341959 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:18.341925    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 23:16:18.342476 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:18.342084    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 23:16:18.342476 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:18.342102    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 23:16:18.396967 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:18.396932    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
Jul 27 23:16:18.402979 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:18.402948    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 23:11:18 +0000 UTC" deadline="2028-03-16 11:48:54.753630408 +0000 UTC"
Jul 27 23:16:18.402979 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:18.402976    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14340h32m36.350658032s"
Jul 27 23:16:18.417578 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:18.417517    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 23:16:18.506059 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:18.506041    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:18.515937 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:18.515912    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 23:16:18.517775 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:18.517756    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal"
Jul 27 23:16:18.528061 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:18.528032    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 23:16:18.562473 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:18.562447    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 23:16:18.629674 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:18.629634    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f826d5b4e5f69d2c5ec64974c7fe045.slice/crio-69e56dca6d34c25ac9a93e9a3b98192dfe00afdca2ef3d667cebc582194eedfa WatchSource:0}: Error finding container 69e56dca6d34c25ac9a93e9a3b98192dfe00afdca2ef3d667cebc582194eedfa: Status 404 returned error can't find the container with id 69e56dca6d34c25ac9a93e9a3b98192dfe00afdca2ef3d667cebc582194eedfa
Jul 27 23:16:18.636925 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:18.636907    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 23:16:18.640436 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:18.640410    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64776cf4968745c952331caeb9a6baf.slice/crio-d410175aa713572e5142d27ec400dc5c5cdeb8b3efc3bc152c8ae0064acd616c WatchSource:0}: Error finding container d410175aa713572e5142d27ec400dc5c5cdeb8b3efc3bc152c8ae0064acd616c: Status 404 returned error can't find the container with id d410175aa713572e5142d27ec400dc5c5cdeb8b3efc3bc152c8ae0064acd616c
Jul 27 23:16:19.331690 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.331653    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 23:16:19.383949 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.383910    2559 apiserver.go:52] "Watching apiserver"
Jul 27 23:16:19.392143 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.392113    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 23:16:19.392753 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.392727    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-fhqmk","kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal","openshift-cluster-node-tuning-operator/tuned-ztqx2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal","openshift-multus/multus-additional-cni-plugins-nxt99","openshift-multus/multus-n75xf","openshift-multus/network-metrics-daemon-hrrmv","openshift-ovn-kubernetes/ovnkube-node-c27p5","kube-system/konnectivity-agent-bcgpc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs","openshift-dns/node-resolver-h8v75","openshift-image-registry/node-ca-mzw9d","openshift-network-diagnostics/network-check-target-s8znn","openshift-network-operator/iptables-alerter-bxkjp"]
Jul 27 23:16:19.395599 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.395575    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.397926 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.397900    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:19.398021 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:19.397982    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:19.399129 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.398869    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 23:16:19.399129 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.398911    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 23:16:19.399129 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.399105    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 23:16:19.399334 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.399105    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x8hlr\""
Jul 27 23:16:19.400263 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.400241    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.402267 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.402244    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 23:16:19.402411 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.402386    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 23:16:19.402717 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.402548    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v7j7m\""
Jul 27 23:16:19.403475 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.403102    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bxkjp"
Jul 27 23:16:19.404968 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.404950    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 23:16:19.405121 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.405045    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 23:16:19.405405 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.405385    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 23:16:19.405659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.405391    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 23:16:19.405659 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.405546    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s6qlb\""
Jul 27 23:16:19.405853 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.405832    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.407800 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.407767    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 23:16:19.407904 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.407805    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zwrvv\""
Jul 27 23:16:19.408155 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.408137    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 23:16:19.408226 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.408155    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 23:16:19.408373 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.408355    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 23:16:19.408461 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.408414    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 23:16:19.408594 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.408576    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.410865 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.410848    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bcgpc"
Jul 27 23:16:19.411070 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.411048    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-98zpz\""
Jul 27 23:16:19.411209 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.411193    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 23:16:19.413038 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.413017    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 23:16:19.413134 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.413088    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-st9mz\""
Jul 27 23:16:19.413344 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.413329    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 23:16:19.413727 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.413690    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:19.413815 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:19.413759    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:19.417134 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.417108    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.420845 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.420646    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 23:16:19.422214 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.421154    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 23:16:19.422214 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.421515    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 23:16:19.422214 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.421692    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 23:16:19.422214 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.421979    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 23:16:19.422462 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.422239    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.422539    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h8v75"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.422717    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/549494f8-ca88-4fde-84cd-06f3af312efc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.422762    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-cnibin\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.422794    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-os-release\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.422826    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-var-lib-kubelet\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.422851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-systemd\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.422898    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-sys\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.422930    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.422934    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-host\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.422964    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4284f262-15b8-4855-ac5e-3cac44c66357-etc-tuned\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:19.423005    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423074    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c298f75-ae76-4cb5-9f69-2f0bf4d41758-host-slash\") pod \"iptables-alerter-bxkjp\" (UID: \"3c298f75-ae76-4cb5-9f69-2f0bf4d41758\") " pod="openshift-network-operator/iptables-alerter-bxkjp"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25x62\" (UniqueName: \"kubernetes.io/projected/549494f8-ca88-4fde-84cd-06f3af312efc-kube-api-access-25x62\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423246    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-cni-binary-copy\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423278    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-run-k8s-cni-cncf-io\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423307    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-etc-selinux\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423338    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-sysconfig\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.424372 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423368    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-lib-modules\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423398    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/549494f8-ca88-4fde-84cd-06f3af312efc-os-release\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423527    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m5n7w\""
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423428    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-system-cni-dir\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-var-lib-cni-bin\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423680    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-var-lib-kubelet\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423708    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4284f262-15b8-4855-ac5e-3cac44c66357-tmp\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423734    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/549494f8-ca88-4fde-84cd-06f3af312efc-cnibin\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423764    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-run-multus-certs\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423793    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423825    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-registration-dir\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423855    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-sys-fs\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423979    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-kubernetes\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.424007    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-sysctl-conf\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.424036    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-var-lib-cni-multus\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.424065    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-device-dir\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.424231    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-run\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.425283 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.423829    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 23:16:19.426131 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.424538    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9srf\" (UniqueName: \"kubernetes.io/projected/3c298f75-ae76-4cb5-9f69-2f0bf4d41758-kube-api-access-x9srf\") pod \"iptables-alerter-bxkjp\" (UID: \"3c298f75-ae76-4cb5-9f69-2f0bf4d41758\") " pod="openshift-network-operator/iptables-alerter-bxkjp"
Jul 27 23:16:19.426131 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.424603    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/549494f8-ca88-4fde-84cd-06f3af312efc-system-cni-dir\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.426131 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.425344    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 23:16:19.426867 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.424674    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/549494f8-ca88-4fde-84cd-06f3af312efc-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.426975 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.426854    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 23:16:19.426975 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.426892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-run-netns\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.426975 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.426926    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-etc-kubernetes\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.426975 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.426951    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-socket-dir\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.427172 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.426975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c298f75-ae76-4cb5-9f69-2f0bf4d41758-iptables-alerter-script\") pod \"iptables-alerter-bxkjp\" (UID: \"3c298f75-ae76-4cb5-9f69-2f0bf4d41758\") " pod="openshift-network-operator/iptables-alerter-bxkjp"
Jul 27 23:16:19.427172 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427000    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/549494f8-ca88-4fde-84cd-06f3af312efc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.427172 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427024    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-multus-cni-dir\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.427172 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427047    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-multus-socket-dir-parent\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.427172 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427102    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-multus-conf-dir\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.427172 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427117    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mzw9d"
Jul 27 23:16:19.427172 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427135    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv754\" (UniqueName: \"kubernetes.io/projected/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-kube-api-access-vv754\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.427172 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427166    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4xz\" (UniqueName: \"kubernetes.io/projected/635f8680-ff02-492b-a3c3-2e8306d53894-kube-api-access-rw4xz\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.427591 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427220    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-modprobe-d\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.427591 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427228    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9fxkk\""
Jul 27 23:16:19.427591 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427249    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/549494f8-ca88-4fde-84cd-06f3af312efc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.427591 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427274    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-hostroot\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.427591 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427301    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-multus-daemon-config\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.427591 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427326    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8x87\" (UniqueName: \"kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87\") pod \"network-check-target-s8znn\" (UID: \"03ff5096-6d95-4ad0-8fc6-c1591c1da37e\") " pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:19.427591 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427351    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-sysctl-d\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.427591 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.427435    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48c7p\" (UniqueName: \"kubernetes.io/projected/4284f262-15b8-4855-ac5e-3cac44c66357-kube-api-access-48c7p\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.428816 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.428792    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 23:16:19.429241 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.429222    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 23:16:19.429341 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.429302    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z4xtv\""
Jul 27 23:16:19.429495 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.429451    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 23:16:19.497301 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.497271    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-knpzj"
Jul 27 23:16:19.505946 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.505911    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-knpzj"
Jul 27 23:16:19.507440 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.507416    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 23:16:19.528538 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.528514    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/549494f8-ca88-4fde-84cd-06f3af312efc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.528673 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.528548    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-multus-cni-dir\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.528673 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.528575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4xz\" (UniqueName: \"kubernetes.io/projected/635f8680-ff02-492b-a3c3-2e8306d53894-kube-api-access-rw4xz\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.528673 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.528604    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-node-log\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.528673 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.528628    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.528673 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.528652    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ec18511-c498-4407-93eb-d0edabcd4311-hosts-file\") pod \"node-resolver-h8v75\" (UID: \"7ec18511-c498-4407-93eb-d0edabcd4311\") " pod="openshift-dns/node-resolver-h8v75"
Jul 27 23:16:19.528896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.528694    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-hostroot\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.528896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.528762    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-multus-daemon-config\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.528896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.528793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x87\" (UniqueName: \"kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87\") pod \"network-check-target-s8znn\" (UID: \"03ff5096-6d95-4ad0-8fc6-c1591c1da37e\") " pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:19.528896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.528827    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-multus-cni-dir\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.528896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.528836    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-kubelet\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.528896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.528863    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-systemd-units\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.529196 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/549494f8-ca88-4fde-84cd-06f3af312efc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.529196 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529044    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-cnibin\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.529196 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-os-release\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.529196 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529110    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-cnibin\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.529196 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529125    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/549494f8-ca88-4fde-84cd-06f3af312efc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.529196 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529152    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-hostroot\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.529196 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529146    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-var-lib-kubelet\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.529196 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529197    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-systemd\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529204    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-os-release\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529224    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4284f262-15b8-4855-ac5e-3cac44c66357-etc-tuned\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529254    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-var-lib-kubelet\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529297    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-systemd\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-var-lib-openvswitch\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529386    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25x62\" (UniqueName: \"kubernetes.io/projected/549494f8-ca88-4fde-84cd-06f3af312efc-kube-api-access-25x62\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-cni-binary-copy\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529482    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-multus-daemon-config\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529490    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-run-k8s-cni-cncf-io\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529526    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-run-k8s-cni-cncf-io\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529546    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-etc-selinux\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529580    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-var-lib-kubelet\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529585    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/549494f8-ca88-4fde-84cd-06f3af312efc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529606    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-etc-openvswitch\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.529626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529614    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/549494f8-ca88-4fde-84cd-06f3af312efc-os-release\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-system-cni-dir\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529690    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-var-lib-cni-bin\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529691    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-var-lib-kubelet\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529720    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-etc-selinux\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529732    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/549494f8-ca88-4fde-84cd-06f3af312efc-os-release\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529736    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-system-cni-dir\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529773    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-var-lib-cni-bin\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529809    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/549494f8-ca88-4fde-84cd-06f3af312efc-cnibin\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529875    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-run-multus-certs\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529885    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/549494f8-ca88-4fde-84cd-06f3af312efc-cnibin\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529904    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-sys-fs\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529910    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-run-multus-certs\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529927    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-sysctl-conf\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.529985    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-sys-fs\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.530290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530016    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz87r\" (UniqueName: \"kubernetes.io/projected/ee785ab1-83ca-4254-b144-c99f9114660f-kube-api-access-tz87r\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530037    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/906d2765-8007-4aed-8c0d-2aa12031fa5a-host\") pod \"node-ca-mzw9d\" (UID: \"906d2765-8007-4aed-8c0d-2aa12031fa5a\") " pod="openshift-image-registry/node-ca-mzw9d"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530037    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-sysctl-conf\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530055    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-slash\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530040    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-cni-binary-copy\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530074    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-cni-netd\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530119    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-var-lib-cni-multus\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-device-dir\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530157    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-env-overrides\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530183    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9srf\" (UniqueName: \"kubernetes.io/projected/3c298f75-ae76-4cb5-9f69-2f0bf4d41758-kube-api-access-x9srf\") pod \"iptables-alerter-bxkjp\" (UID: \"3c298f75-ae76-4cb5-9f69-2f0bf4d41758\") " pod="openshift-network-operator/iptables-alerter-bxkjp"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/549494f8-ca88-4fde-84cd-06f3af312efc-system-cni-dir\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530210    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-device-dir\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530243    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-var-lib-cni-multus\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530272    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/549494f8-ca88-4fde-84cd-06f3af312efc-system-cni-dir\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530322    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/549494f8-ca88-4fde-84cd-06f3af312efc-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530346    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03e4d553-7ad4-4072-acf6-1d8424cbff5b-dbus\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530365    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec18511-c498-4407-93eb-d0edabcd4311-tmp-dir\") pod \"node-resolver-h8v75\" (UID: \"7ec18511-c498-4407-93eb-d0edabcd4311\") " pod="openshift-dns/node-resolver-h8v75"
Jul 27 23:16:19.531261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c298f75-ae76-4cb5-9f69-2f0bf4d41758-iptables-alerter-script\") pod \"iptables-alerter-bxkjp\" (UID: \"3c298f75-ae76-4cb5-9f69-2f0bf4d41758\") " pod="openshift-network-operator/iptables-alerter-bxkjp"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530510    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-multus-socket-dir-parent\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530610    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-multus-socket-dir-parent\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-multus-conf-dir\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530708    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv754\" (UniqueName: \"kubernetes.io/projected/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-kube-api-access-vv754\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530731    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-modprobe-d\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530840    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/549494f8-ca88-4fde-84cd-06f3af312efc-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530900    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-sysctl-d\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530923    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-multus-conf-dir\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-modprobe-d\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530961    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c298f75-ae76-4cb5-9f69-2f0bf4d41758-iptables-alerter-script\") pod \"iptables-alerter-bxkjp\" (UID: \"3c298f75-ae76-4cb5-9f69-2f0bf4d41758\") " pod="openshift-network-operator/iptables-alerter-bxkjp"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.530965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5bsz\" (UniqueName: \"kubernetes.io/projected/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-kube-api-access-c5bsz\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/549494f8-ca88-4fde-84cd-06f3af312efc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531046    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48c7p\" (UniqueName: \"kubernetes.io/projected/4284f262-15b8-4855-ac5e-3cac44c66357-kube-api-access-48c7p\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531067    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-sys\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-sysctl-d\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531090    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-host\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.532076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531117    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/906d2765-8007-4aed-8c0d-2aa12031fa5a-serviceca\") pod \"node-ca-mzw9d\" (UID: \"906d2765-8007-4aed-8c0d-2aa12031fa5a\") " pod="openshift-image-registry/node-ca-mzw9d"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531124    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-sys\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531137    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-host\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531143    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c298f75-ae76-4cb5-9f69-2f0bf4d41758-host-slash\") pod \"iptables-alerter-bxkjp\" (UID: \"3c298f75-ae76-4cb5-9f69-2f0bf4d41758\") " pod="openshift-network-operator/iptables-alerter-bxkjp"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531185    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c298f75-ae76-4cb5-9f69-2f0bf4d41758-host-slash\") pod \"iptables-alerter-bxkjp\" (UID: \"3c298f75-ae76-4cb5-9f69-2f0bf4d41758\") " pod="openshift-network-operator/iptables-alerter-bxkjp"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531188    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/549494f8-ca88-4fde-84cd-06f3af312efc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531189    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-sysconfig\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531227    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-sysconfig\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531229    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-lib-modules\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531259    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4cf21fbf-dc73-405a-836a-f5cf4a7c64d8-agent-certs\") pod \"konnectivity-agent-bcgpc\" (UID: \"4cf21fbf-dc73-405a-836a-f5cf4a7c64d8\") " pod="kube-system/konnectivity-agent-bcgpc"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531285    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-ovnkube-config\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531312    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4284f262-15b8-4855-ac5e-3cac44c66357-tmp\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531337    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-ovn-node-metrics-cert\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531314    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-lib-modules\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531370    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03e4d553-7ad4-4072-acf6-1d8424cbff5b-kubelet-config\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531448    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531476    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-registration-dir\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.532868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531503    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-kubernetes\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531522    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-registration-dir\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531531    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xffq9\" (UniqueName: \"kubernetes.io/projected/906d2765-8007-4aed-8c0d-2aa12031fa5a-kube-api-access-xffq9\") pod \"node-ca-mzw9d\" (UID: \"906d2765-8007-4aed-8c0d-2aa12031fa5a\") " pod="openshift-image-registry/node-ca-mzw9d"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531588    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531592    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-run-netns\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531623    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-run-openvswitch\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531644    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-run\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531682    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-etc-kubernetes\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531715    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4cf21fbf-dc73-405a-836a-f5cf4a7c64d8-konnectivity-ca\") pod \"konnectivity-agent-bcgpc\" (UID: \"4cf21fbf-dc73-405a-836a-f5cf4a7c64d8\") " pod="kube-system/konnectivity-agent-bcgpc"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531736    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-run-systemd\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531759    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-run-ovn\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531788    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4284f262-15b8-4855-ac5e-3cac44c66357-run\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531793    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-log-socket\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531817    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqpp8\" (UniqueName: \"kubernetes.io/projected/7ec18511-c498-4407-93eb-d0edabcd4311-kube-api-access-xqpp8\") pod \"node-resolver-h8v75\" (UID: \"7ec18511-c498-4407-93eb-d0edabcd4311\") " pod="openshift-dns/node-resolver-h8v75"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531839    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-run-netns\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531857    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-etc-kubernetes\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.533680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531880    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-socket-dir\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.534271 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531906    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.534271 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531919    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-etc-kubernetes\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.534271 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-cni-bin\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.534271 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531910    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-host-run-netns\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.534271 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.531958    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-ovnkube-script-lib\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.534271 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.532200    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/635f8680-ff02-492b-a3c3-2e8306d53894-socket-dir\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.534271 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.533577    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4284f262-15b8-4855-ac5e-3cac44c66357-etc-tuned\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.534271 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.533750    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4284f262-15b8-4855-ac5e-3cac44c66357-tmp\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.538996 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:19.538796    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 23:16:19.538996 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:19.538823    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 23:16:19.538996 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:19.538838    2559 projected.go:196] Error preparing data for projected volume kube-api-access-b8x87 for pod openshift-network-diagnostics/network-check-target-s8znn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:19.538996 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:19.538900    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87 podName:03ff5096-6d95-4ad0-8fc6-c1591c1da37e nodeName:}" failed. No retries permitted until 2026-07-27 23:16:20.038878975 +0000 UTC m=+2.983109707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b8x87" (UniqueName: "kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87") pod "network-check-target-s8znn" (UID: "03ff5096-6d95-4ad0-8fc6-c1591c1da37e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:19.542748 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.542723    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25x62\" (UniqueName: \"kubernetes.io/projected/549494f8-ca88-4fde-84cd-06f3af312efc-kube-api-access-25x62\") pod \"multus-additional-cni-plugins-nxt99\" (UID: \"549494f8-ca88-4fde-84cd-06f3af312efc\") " pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.543167 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.543144    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9srf\" (UniqueName: \"kubernetes.io/projected/3c298f75-ae76-4cb5-9f69-2f0bf4d41758-kube-api-access-x9srf\") pod \"iptables-alerter-bxkjp\" (UID: \"3c298f75-ae76-4cb5-9f69-2f0bf4d41758\") " pod="openshift-network-operator/iptables-alerter-bxkjp"
Jul 27 23:16:19.543528 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.543505    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4xz\" (UniqueName: \"kubernetes.io/projected/635f8680-ff02-492b-a3c3-2e8306d53894-kube-api-access-rw4xz\") pod \"aws-ebs-csi-driver-node-bs7qs\" (UID: \"635f8680-ff02-492b-a3c3-2e8306d53894\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.544669 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.544647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48c7p\" (UniqueName: \"kubernetes.io/projected/4284f262-15b8-4855-ac5e-3cac44c66357-kube-api-access-48c7p\") pod \"tuned-ztqx2\" (UID: \"4284f262-15b8-4855-ac5e-3cac44c66357\") " pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.544866 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.544842    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv754\" (UniqueName: \"kubernetes.io/projected/01afb354-a5aa-41d8-ae0a-5028e75ae3e9-kube-api-access-vv754\") pod \"multus-n75xf\" (UID: \"01afb354-a5aa-41d8-ae0a-5028e75ae3e9\") " pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.565224 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.565178    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal" event={"ID":"a64776cf4968745c952331caeb9a6baf","Type":"ContainerStarted","Data":"d410175aa713572e5142d27ec400dc5c5cdeb8b3efc3bc152c8ae0064acd616c"}
Jul 27 23:16:19.566535 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.566506    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal" event={"ID":"2f826d5b4e5f69d2c5ec64974c7fe045","Type":"ContainerStarted","Data":"69e56dca6d34c25ac9a93e9a3b98192dfe00afdca2ef3d667cebc582194eedfa"}
Jul 27 23:16:19.632570 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632486    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 23:16:19.632738 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632569    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-node-log\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.632738 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.632738 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632633    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ec18511-c498-4407-93eb-d0edabcd4311-hosts-file\") pod \"node-resolver-h8v75\" (UID: \"7ec18511-c498-4407-93eb-d0edabcd4311\") " pod="openshift-dns/node-resolver-h8v75"
Jul 27 23:16:19.632738 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632665    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-node-log\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.632738 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632674    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-kubelet\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.632738 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633052 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632755    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ec18511-c498-4407-93eb-d0edabcd4311-hosts-file\") pod \"node-resolver-h8v75\" (UID: \"7ec18511-c498-4407-93eb-d0edabcd4311\") " pod="openshift-dns/node-resolver-h8v75"
Jul 27 23:16:19.633052 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632795    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-systemd-units\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633052 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632836    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-var-lib-openvswitch\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633052 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632860    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-kubelet\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633052 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632891    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-var-lib-openvswitch\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633052 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632892    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-systemd-units\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633052 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-etc-openvswitch\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633052 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:19.633052 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz87r\" (UniqueName: \"kubernetes.io/projected/ee785ab1-83ca-4254-b144-c99f9114660f-kube-api-access-tz87r\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:19.633052 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-etc-openvswitch\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633052 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.632996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/906d2765-8007-4aed-8c0d-2aa12031fa5a-host\") pod \"node-ca-mzw9d\" (UID: \"906d2765-8007-4aed-8c0d-2aa12031fa5a\") " pod="openshift-image-registry/node-ca-mzw9d"
Jul 27 23:16:19.633052 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/906d2765-8007-4aed-8c0d-2aa12031fa5a-host\") pod \"node-ca-mzw9d\" (UID: \"906d2765-8007-4aed-8c0d-2aa12031fa5a\") " pod="openshift-image-registry/node-ca-mzw9d"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-slash\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:19.633086    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633107    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-cni-netd\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633138    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-slash\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:19.633148    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret podName:03e4d553-7ad4-4072-acf6-1d8424cbff5b nodeName:}" failed. No retries permitted until 2026-07-27 23:16:20.133127572 +0000 UTC m=+3.077358516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret") pod "global-pull-secret-syncer-fhqmk" (UID: "03e4d553-7ad4-4072-acf6-1d8424cbff5b") : object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633161    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-cni-netd\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633166    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-env-overrides\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633203    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03e4d553-7ad4-4072-acf6-1d8424cbff5b-dbus\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633233    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec18511-c498-4407-93eb-d0edabcd4311-tmp-dir\") pod \"node-resolver-h8v75\" (UID: \"7ec18511-c498-4407-93eb-d0edabcd4311\") " pod="openshift-dns/node-resolver-h8v75"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633262    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03e4d553-7ad4-4072-acf6-1d8424cbff5b-dbus\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633265    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5bsz\" (UniqueName: \"kubernetes.io/projected/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-kube-api-access-c5bsz\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633308    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/906d2765-8007-4aed-8c0d-2aa12031fa5a-serviceca\") pod \"node-ca-mzw9d\" (UID: \"906d2765-8007-4aed-8c0d-2aa12031fa5a\") " pod="openshift-image-registry/node-ca-mzw9d"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633332    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4cf21fbf-dc73-405a-836a-f5cf4a7c64d8-agent-certs\") pod \"konnectivity-agent-bcgpc\" (UID: \"4cf21fbf-dc73-405a-836a-f5cf4a7c64d8\") " pod="kube-system/konnectivity-agent-bcgpc"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633349    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-ovnkube-config\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-ovn-node-metrics-cert\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633392    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03e4d553-7ad4-4072-acf6-1d8424cbff5b-kubelet-config\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:19.633550 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633440    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xffq9\" (UniqueName: \"kubernetes.io/projected/906d2765-8007-4aed-8c0d-2aa12031fa5a-kube-api-access-xffq9\") pod \"node-ca-mzw9d\" (UID: \"906d2765-8007-4aed-8c0d-2aa12031fa5a\") " pod="openshift-image-registry/node-ca-mzw9d"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633488    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-run-netns\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633515    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-run-openvswitch\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633541    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4cf21fbf-dc73-405a-836a-f5cf4a7c64d8-konnectivity-ca\") pod \"konnectivity-agent-bcgpc\" (UID: \"4cf21fbf-dc73-405a-836a-f5cf4a7c64d8\") " pod="kube-system/konnectivity-agent-bcgpc"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633566    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-run-systemd\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633589    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-run-ovn\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec18511-c498-4407-93eb-d0edabcd4311-tmp-dir\") pod \"node-resolver-h8v75\" (UID: \"7ec18511-c498-4407-93eb-d0edabcd4311\") " pod="openshift-dns/node-resolver-h8v75"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633611    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-log-socket\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqpp8\" (UniqueName: \"kubernetes.io/projected/7ec18511-c498-4407-93eb-d0edabcd4311-kube-api-access-xqpp8\") pod \"node-resolver-h8v75\" (UID: \"7ec18511-c498-4407-93eb-d0edabcd4311\") " pod="openshift-dns/node-resolver-h8v75"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633689    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-cni-bin\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633708    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-run-openvswitch\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633719    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-env-overrides\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-ovnkube-script-lib\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633777    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03e4d553-7ad4-4072-acf6-1d8424cbff5b-kubelet-config\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633822    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-log-socket\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:19.633849    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:19.634210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633883    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/906d2765-8007-4aed-8c0d-2aa12031fa5a-serviceca\") pod \"node-ca-mzw9d\" (UID: \"906d2765-8007-4aed-8c0d-2aa12031fa5a\") " pod="openshift-image-registry/node-ca-mzw9d"
Jul 27 23:16:19.634914 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:19.633913    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs podName:ee785ab1-83ca-4254-b144-c99f9114660f nodeName:}" failed. No retries permitted until 2026-07-27 23:16:20.133896213 +0000 UTC m=+3.078126958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs") pod "network-metrics-daemon-hrrmv" (UID: "ee785ab1-83ca-4254-b144-c99f9114660f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:19.634914 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.633949    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-run-systemd\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634914 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.634067    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-run-netns\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634914 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.634122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634914 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.634127    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-host-cni-bin\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634914 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.634168    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-run-ovn\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634914 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.634326    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4cf21fbf-dc73-405a-836a-f5cf4a7c64d8-konnectivity-ca\") pod \"konnectivity-agent-bcgpc\" (UID: \"4cf21fbf-dc73-405a-836a-f5cf4a7c64d8\") " pod="kube-system/konnectivity-agent-bcgpc"
Jul 27 23:16:19.634914 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.634537    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-ovnkube-config\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.634914 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.634882    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-ovnkube-script-lib\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.636184 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.636158    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4cf21fbf-dc73-405a-836a-f5cf4a7c64d8-agent-certs\") pod \"konnectivity-agent-bcgpc\" (UID: \"4cf21fbf-dc73-405a-836a-f5cf4a7c64d8\") " pod="kube-system/konnectivity-agent-bcgpc"
Jul 27 23:16:19.636755 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.636732    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-ovn-node-metrics-cert\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.641689 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.641660    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5bsz\" (UniqueName: \"kubernetes.io/projected/5eedceb6-d520-4108-8b3d-5c6b933d1fb1-kube-api-access-c5bsz\") pod \"ovnkube-node-c27p5\" (UID: \"5eedceb6-d520-4108-8b3d-5c6b933d1fb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.642028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.641997    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz87r\" (UniqueName: \"kubernetes.io/projected/ee785ab1-83ca-4254-b144-c99f9114660f-kube-api-access-tz87r\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:19.642310 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.642289    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqpp8\" (UniqueName: \"kubernetes.io/projected/7ec18511-c498-4407-93eb-d0edabcd4311-kube-api-access-xqpp8\") pod \"node-resolver-h8v75\" (UID: \"7ec18511-c498-4407-93eb-d0edabcd4311\") " pod="openshift-dns/node-resolver-h8v75"
Jul 27 23:16:19.643693 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.643656    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xffq9\" (UniqueName: \"kubernetes.io/projected/906d2765-8007-4aed-8c0d-2aa12031fa5a-kube-api-access-xffq9\") pod \"node-ca-mzw9d\" (UID: \"906d2765-8007-4aed-8c0d-2aa12031fa5a\") " pod="openshift-image-registry/node-ca-mzw9d"
Jul 27 23:16:19.712296 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.712261    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs"
Jul 27 23:16:19.719442 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.719419    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ztqx2"
Jul 27 23:16:19.734253 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.734232    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bxkjp"
Jul 27 23:16:19.739103 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.739070    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nxt99"
Jul 27 23:16:19.747194 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.747174    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n75xf"
Jul 27 23:16:19.756017 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.755997    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bcgpc"
Jul 27 23:16:19.765034 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.765013    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:19.778931 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.778905    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h8v75"
Jul 27 23:16:19.785771 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:19.785754    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mzw9d"
Jul 27 23:16:20.136716 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:20.136681    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:20.136867 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:20.136730    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x87\" (UniqueName: \"kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87\") pod \"network-check-target-s8znn\" (UID: \"03ff5096-6d95-4ad0-8fc6-c1591c1da37e\") " pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:20.136867 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:20.136773    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:20.136982 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:20.136869    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:20.136982 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:20.136932    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs podName:ee785ab1-83ca-4254-b144-c99f9114660f nodeName:}" failed. No retries permitted until 2026-07-27 23:16:21.136916449 +0000 UTC m=+4.081147179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs") pod "network-metrics-daemon-hrrmv" (UID: "ee785ab1-83ca-4254-b144-c99f9114660f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:20.136982 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:20.136868    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:20.137127 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:20.136982    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 23:16:20.137127 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:20.137007    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 23:16:20.137127 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:20.137023    2559 projected.go:196] Error preparing data for projected volume kube-api-access-b8x87 for pod openshift-network-diagnostics/network-check-target-s8znn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:20.137127 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:20.137051    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret podName:03e4d553-7ad4-4072-acf6-1d8424cbff5b nodeName:}" failed. No retries permitted until 2026-07-27 23:16:21.137029637 +0000 UTC m=+4.081260380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret") pod "global-pull-secret-syncer-fhqmk" (UID: "03e4d553-7ad4-4072-acf6-1d8424cbff5b") : object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:20.137127 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:20.137080    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87 podName:03ff5096-6d95-4ad0-8fc6-c1591c1da37e nodeName:}" failed. No retries permitted until 2026-07-27 23:16:21.137063262 +0000 UTC m=+4.081294002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8x87" (UniqueName: "kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87") pod "network-check-target-s8znn" (UID: "03ff5096-6d95-4ad0-8fc6-c1591c1da37e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:20.510005 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:20.509924    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 23:11:20 +0000 UTC" deadline="2028-03-08 13:37:41.137326151 +0000 UTC"
Jul 27 23:16:20.510517 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:20.510009    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14150h21m20.627329059s"
Jul 27 23:16:20.557536 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:20.557503    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:20.557732 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:20.557503    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:20.557732 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:20.557651    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:20.557862 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:20.557755    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:20.648120 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:20.648044    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01afb354_a5aa_41d8_ae0a_5028e75ae3e9.slice/crio-529bcee391311bce6ff59ae6f054c20d61e0be759b7b5103d6dd7d5549224373 WatchSource:0}: Error finding container 529bcee391311bce6ff59ae6f054c20d61e0be759b7b5103d6dd7d5549224373: Status 404 returned error can't find the container with id 529bcee391311bce6ff59ae6f054c20d61e0be759b7b5103d6dd7d5549224373
Jul 27 23:16:20.650305 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:20.650279    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec18511_c498_4407_93eb_d0edabcd4311.slice/crio-c2d0dfd61163533ad59adf1fc28f51e5044bad8f1e5069ece62bc111a9d59703 WatchSource:0}: Error finding container c2d0dfd61163533ad59adf1fc28f51e5044bad8f1e5069ece62bc111a9d59703: Status 404 returned error can't find the container with id c2d0dfd61163533ad59adf1fc28f51e5044bad8f1e5069ece62bc111a9d59703
Jul 27 23:16:20.656789 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:20.656759    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4284f262_15b8_4855_ac5e_3cac44c66357.slice/crio-6a7a4883a120bf9d8703bf7cc90af8f07fce0813f3b074f90bdb0f29692b2d6b WatchSource:0}: Error finding container 6a7a4883a120bf9d8703bf7cc90af8f07fce0813f3b074f90bdb0f29692b2d6b: Status 404 returned error can't find the container with id 6a7a4883a120bf9d8703bf7cc90af8f07fce0813f3b074f90bdb0f29692b2d6b
Jul 27 23:16:20.660824 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:20.660796    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c298f75_ae76_4cb5_9f69_2f0bf4d41758.slice/crio-5a208250ef508658ebafab4dfe505ad459b667e9b22d906711229910b3ce89d7 WatchSource:0}: Error finding container 5a208250ef508658ebafab4dfe505ad459b667e9b22d906711229910b3ce89d7: Status 404 returned error can't find the container with id 5a208250ef508658ebafab4dfe505ad459b667e9b22d906711229910b3ce89d7
Jul 27 23:16:20.662611 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:20.662493    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635f8680_ff02_492b_a3c3_2e8306d53894.slice/crio-c275e8bc65acbf34a0a07e3ad3f94f231c36540a1ac3a2b38e4d5dd5aaad3cee WatchSource:0}: Error finding container c275e8bc65acbf34a0a07e3ad3f94f231c36540a1ac3a2b38e4d5dd5aaad3cee: Status 404 returned error can't find the container with id c275e8bc65acbf34a0a07e3ad3f94f231c36540a1ac3a2b38e4d5dd5aaad3cee
Jul 27 23:16:20.664428 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:20.664027    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eedceb6_d520_4108_8b3d_5c6b933d1fb1.slice/crio-1f1577b8d606d737e3e9856629c5614438f6fad6eb52ca84d987033501a0e21a WatchSource:0}: Error finding container 1f1577b8d606d737e3e9856629c5614438f6fad6eb52ca84d987033501a0e21a: Status 404 returned error can't find the container with id 1f1577b8d606d737e3e9856629c5614438f6fad6eb52ca84d987033501a0e21a
Jul 27 23:16:20.664849 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:20.664824    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906d2765_8007_4aed_8c0d_2aa12031fa5a.slice/crio-ff5fdbdfdc2c9568172ea23ab90cea03e7db8df556a77dbc246f20414208ec63 WatchSource:0}: Error finding container ff5fdbdfdc2c9568172ea23ab90cea03e7db8df556a77dbc246f20414208ec63: Status 404 returned error can't find the container with id ff5fdbdfdc2c9568172ea23ab90cea03e7db8df556a77dbc246f20414208ec63
Jul 27 23:16:20.665623 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:20.665559    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549494f8_ca88_4fde_84cd_06f3af312efc.slice/crio-c657764cdb36bd9f49aee34ae9ffc3bb494527bd2869b96c858ec630905bc034 WatchSource:0}: Error finding container c657764cdb36bd9f49aee34ae9ffc3bb494527bd2869b96c858ec630905bc034: Status 404 returned error can't find the container with id c657764cdb36bd9f49aee34ae9ffc3bb494527bd2869b96c858ec630905bc034
Jul 27 23:16:20.667164 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:20.667114    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf21fbf_dc73_405a_836a_f5cf4a7c64d8.slice/crio-bb307a3a1b9bdbea0fff831b36bfdcb75ff607d1527d31710e5fc4c1b849a27f WatchSource:0}: Error finding container bb307a3a1b9bdbea0fff831b36bfdcb75ff607d1527d31710e5fc4c1b849a27f: Status 404 returned error can't find the container with id bb307a3a1b9bdbea0fff831b36bfdcb75ff607d1527d31710e5fc4c1b849a27f
Jul 27 23:16:21.146314 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.145376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x87\" (UniqueName: \"kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87\") pod \"network-check-target-s8znn\" (UID: \"03ff5096-6d95-4ad0-8fc6-c1591c1da37e\") " pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:21.146314 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:21.145668    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 23:16:21.146314 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:21.145693    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 23:16:21.146314 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:21.145707    2559 projected.go:196] Error preparing data for projected volume kube-api-access-b8x87 for pod openshift-network-diagnostics/network-check-target-s8znn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:21.146314 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.145705    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:21.146314 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:21.145765    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87 podName:03ff5096-6d95-4ad0-8fc6-c1591c1da37e nodeName:}" failed. No retries permitted until 2026-07-27 23:16:23.145748374 +0000 UTC m=+6.089979110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8x87" (UniqueName: "kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87") pod "network-check-target-s8znn" (UID: "03ff5096-6d95-4ad0-8fc6-c1591c1da37e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:21.146314 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.145799    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:21.146314 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:21.145814    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:21.146314 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:21.145852    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret podName:03e4d553-7ad4-4072-acf6-1d8424cbff5b nodeName:}" failed. No retries permitted until 2026-07-27 23:16:23.145839874 +0000 UTC m=+6.090070602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret") pod "global-pull-secret-syncer-fhqmk" (UID: "03e4d553-7ad4-4072-acf6-1d8424cbff5b") : object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:21.146314 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:21.145882    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:21.146314 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:21.145913    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs podName:ee785ab1-83ca-4254-b144-c99f9114660f nodeName:}" failed. No retries permitted until 2026-07-27 23:16:23.145902284 +0000 UTC m=+6.090133015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs") pod "network-metrics-daemon-hrrmv" (UID: "ee785ab1-83ca-4254-b144-c99f9114660f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:21.510694 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.510571    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 23:11:20 +0000 UTC" deadline="2028-01-19 08:00:22.954503842 +0000 UTC"
Jul 27 23:16:21.510694 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.510643    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12968h44m1.443866598s"
Jul 27 23:16:21.559669 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.559634    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:21.559856 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:21.559779    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:21.576612 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.576575    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal" event={"ID":"a64776cf4968745c952331caeb9a6baf","Type":"ContainerStarted","Data":"a8b2028eddf5bc0aaae4c799db73eb04610b703948872a4a22b204f67567be75"}
Jul 27 23:16:21.580510 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.580480    2559 generic.go:356] "Generic (PLEG): container finished" podID="2f826d5b4e5f69d2c5ec64974c7fe045" containerID="7c13ff4647c38c0413edb5f15b16cecb05df152cc3ce886e4ab6f8464070084c" exitCode=0
Jul 27 23:16:21.580650 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.580607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal" event={"ID":"2f826d5b4e5f69d2c5ec64974c7fe045","Type":"ContainerDied","Data":"7c13ff4647c38c0413edb5f15b16cecb05df152cc3ce886e4ab6f8464070084c"}
Jul 27 23:16:21.594396 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.594363    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5" event={"ID":"5eedceb6-d520-4108-8b3d-5c6b933d1fb1","Type":"ContainerStarted","Data":"1f1577b8d606d737e3e9856629c5614438f6fad6eb52ca84d987033501a0e21a"}
Jul 27 23:16:21.604031 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.603807    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bxkjp" event={"ID":"3c298f75-ae76-4cb5-9f69-2f0bf4d41758","Type":"ContainerStarted","Data":"5a208250ef508658ebafab4dfe505ad459b667e9b22d906711229910b3ce89d7"}
Jul 27 23:16:21.604511 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.604461    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal" podStartSLOduration=2.604422934 podStartE2EDuration="2.604422934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:21.589883762 +0000 UTC m=+4.534114512" watchObservedRunningTime="2026-07-27 23:16:21.604422934 +0000 UTC m=+4.548653683"
Jul 27 23:16:21.621592 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.621553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ztqx2" event={"ID":"4284f262-15b8-4855-ac5e-3cac44c66357","Type":"ContainerStarted","Data":"6a7a4883a120bf9d8703bf7cc90af8f07fce0813f3b074f90bdb0f29692b2d6b"}
Jul 27 23:16:21.632180 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.632147    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h8v75" event={"ID":"7ec18511-c498-4407-93eb-d0edabcd4311","Type":"ContainerStarted","Data":"c2d0dfd61163533ad59adf1fc28f51e5044bad8f1e5069ece62bc111a9d59703"}
Jul 27 23:16:21.638517 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.638486    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bcgpc" event={"ID":"4cf21fbf-dc73-405a-836a-f5cf4a7c64d8","Type":"ContainerStarted","Data":"bb307a3a1b9bdbea0fff831b36bfdcb75ff607d1527d31710e5fc4c1b849a27f"}
Jul 27 23:16:21.658100 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.658061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxt99" event={"ID":"549494f8-ca88-4fde-84cd-06f3af312efc","Type":"ContainerStarted","Data":"c657764cdb36bd9f49aee34ae9ffc3bb494527bd2869b96c858ec630905bc034"}
Jul 27 23:16:21.663901 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.663801    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mzw9d" event={"ID":"906d2765-8007-4aed-8c0d-2aa12031fa5a","Type":"ContainerStarted","Data":"ff5fdbdfdc2c9568172ea23ab90cea03e7db8df556a77dbc246f20414208ec63"}
Jul 27 23:16:21.676560 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.676524    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs" event={"ID":"635f8680-ff02-492b-a3c3-2e8306d53894","Type":"ContainerStarted","Data":"c275e8bc65acbf34a0a07e3ad3f94f231c36540a1ac3a2b38e4d5dd5aaad3cee"}
Jul 27 23:16:21.685076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:21.685020    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n75xf" event={"ID":"01afb354-a5aa-41d8-ae0a-5028e75ae3e9","Type":"ContainerStarted","Data":"529bcee391311bce6ff59ae6f054c20d61e0be759b7b5103d6dd7d5549224373"}
Jul 27 23:16:22.556871 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:22.556836    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:22.557370 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:22.556985    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:22.557472 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:22.557420    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:22.557564 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:22.557544    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:22.696022 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:22.695951    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal" event={"ID":"2f826d5b4e5f69d2c5ec64974c7fe045","Type":"ContainerStarted","Data":"38a02a7e5eaec4ef8052f4855c82a2384bdd00171a06794d81915d56a0761dec"}
Jul 27 23:16:22.709483 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:22.708310    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal" podStartSLOduration=3.708288358 podStartE2EDuration="3.708288358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:22.707982272 +0000 UTC m=+5.652213022" watchObservedRunningTime="2026-07-27 23:16:22.708288358 +0000 UTC m=+5.652519108"
Jul 27 23:16:23.166964 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:23.166262    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x87\" (UniqueName: \"kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87\") pod \"network-check-target-s8znn\" (UID: \"03ff5096-6d95-4ad0-8fc6-c1591c1da37e\") " pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:23.166964 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:23.166327    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:23.166964 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:23.166383    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:23.166964 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:23.166491    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 23:16:23.166964 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:23.166514    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:23.166964 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:23.166524    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 23:16:23.166964 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:23.166538    2559 projected.go:196] Error preparing data for projected volume kube-api-access-b8x87 for pod openshift-network-diagnostics/network-check-target-s8znn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:23.166964 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:23.166585    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs podName:ee785ab1-83ca-4254-b144-c99f9114660f nodeName:}" failed. No retries permitted until 2026-07-27 23:16:27.16655834 +0000 UTC m=+10.110789082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs") pod "network-metrics-daemon-hrrmv" (UID: "ee785ab1-83ca-4254-b144-c99f9114660f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:23.166964 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:23.166604    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87 podName:03ff5096-6d95-4ad0-8fc6-c1591c1da37e nodeName:}" failed. No retries permitted until 2026-07-27 23:16:27.166594262 +0000 UTC m=+10.110824995 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8x87" (UniqueName: "kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87") pod "network-check-target-s8znn" (UID: "03ff5096-6d95-4ad0-8fc6-c1591c1da37e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:23.166964 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:23.166651    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:23.166964 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:23.166697    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret podName:03e4d553-7ad4-4072-acf6-1d8424cbff5b nodeName:}" failed. No retries permitted until 2026-07-27 23:16:27.166679391 +0000 UTC m=+10.110910123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret") pod "global-pull-secret-syncer-fhqmk" (UID: "03e4d553-7ad4-4072-acf6-1d8424cbff5b") : object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:23.557682 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:23.557520    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:23.557682 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:23.557664    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:24.557614 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:24.557581    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:24.557795 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:24.557699    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:24.558083 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:24.558020    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:24.558171 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:24.558113    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:25.557174 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:25.557133    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:25.557370 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:25.557282    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:26.557512 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:26.557477    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:26.557939 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:26.557479    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:26.557939 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:26.557613    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:26.557939 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:26.557740    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:27.201297 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:27.200332    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:27.201297 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:27.200391    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x87\" (UniqueName: \"kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87\") pod \"network-check-target-s8znn\" (UID: \"03ff5096-6d95-4ad0-8fc6-c1591c1da37e\") " pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:27.201297 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:27.200432    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:27.201297 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:27.200624    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:27.201297 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:27.200688    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret podName:03e4d553-7ad4-4072-acf6-1d8424cbff5b nodeName:}" failed. No retries permitted until 2026-07-27 23:16:35.200666431 +0000 UTC m=+18.144897171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret") pod "global-pull-secret-syncer-fhqmk" (UID: "03e4d553-7ad4-4072-acf6-1d8424cbff5b") : object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:27.201297 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:27.201117    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:27.201297 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:27.201153    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs podName:ee785ab1-83ca-4254-b144-c99f9114660f nodeName:}" failed. No retries permitted until 2026-07-27 23:16:35.2011432 +0000 UTC m=+18.145373929 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs") pod "network-metrics-daemon-hrrmv" (UID: "ee785ab1-83ca-4254-b144-c99f9114660f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:27.201297 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:27.201212    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 23:16:27.201297 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:27.201226    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 23:16:27.201297 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:27.201235    2559 projected.go:196] Error preparing data for projected volume kube-api-access-b8x87 for pod openshift-network-diagnostics/network-check-target-s8znn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:27.201297 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:27.201258    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87 podName:03ff5096-6d95-4ad0-8fc6-c1591c1da37e nodeName:}" failed. No retries permitted until 2026-07-27 23:16:35.201251014 +0000 UTC m=+18.145481742 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8x87" (UniqueName: "kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87") pod "network-check-target-s8znn" (UID: "03ff5096-6d95-4ad0-8fc6-c1591c1da37e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:27.569678 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:27.567993    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:27.569678 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:27.568149    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:28.557482 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:28.557363    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:28.558306 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:28.557788    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:28.558306 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:28.557465    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:28.558306 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:28.558267    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:29.557537 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:29.557504    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:29.558026 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:29.557651    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:30.557404 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:30.557048    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:30.557404 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:30.557089    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:30.557404 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:30.557186    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:30.557404 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:30.557329    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:31.556866 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:31.556832    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:31.557057 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:31.556966    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:32.557502 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:32.557462    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:32.557502 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:32.557494    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:32.557974 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:32.557595    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:32.557974 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:32.557724    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:33.556783 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:33.556746    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:33.556959 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:33.556877    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:34.556995 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:34.556959    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:34.557502 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:34.557095    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:34.557502 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:34.557141    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:34.557502 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:34.557235    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:35.267510 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:35.267482    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:35.267676 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:35.267521    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x87\" (UniqueName: \"kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87\") pod \"network-check-target-s8znn\" (UID: \"03ff5096-6d95-4ad0-8fc6-c1591c1da37e\") " pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:35.267676 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:35.267543    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:35.267676 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:35.267633    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:35.267676 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:35.267633    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:35.267676 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:35.267652    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 23:16:35.267676 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:35.267671    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 23:16:35.267676 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:35.267681    2559 projected.go:196] Error preparing data for projected volume kube-api-access-b8x87 for pod openshift-network-diagnostics/network-check-target-s8znn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:35.267890 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:35.267688    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs podName:ee785ab1-83ca-4254-b144-c99f9114660f nodeName:}" failed. No retries permitted until 2026-07-27 23:16:51.267672971 +0000 UTC m=+34.211903698 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs") pod "network-metrics-daemon-hrrmv" (UID: "ee785ab1-83ca-4254-b144-c99f9114660f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:35.267890 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:35.267701    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret podName:03e4d553-7ad4-4072-acf6-1d8424cbff5b nodeName:}" failed. No retries permitted until 2026-07-27 23:16:51.267695316 +0000 UTC m=+34.211926040 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret") pod "global-pull-secret-syncer-fhqmk" (UID: "03e4d553-7ad4-4072-acf6-1d8424cbff5b") : object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:35.267890 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:35.267729    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87 podName:03ff5096-6d95-4ad0-8fc6-c1591c1da37e nodeName:}" failed. No retries permitted until 2026-07-27 23:16:51.267713663 +0000 UTC m=+34.211944398 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8x87" (UniqueName: "kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87") pod "network-check-target-s8znn" (UID: "03ff5096-6d95-4ad0-8fc6-c1591c1da37e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:35.557035 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:35.556950    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:35.557588 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:35.557078    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:36.557070 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:36.557026    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:36.557070 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:36.557059    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:36.557626 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:36.557167    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:36.557626 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:36.557336    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:37.557716 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:37.557683    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:37.558253 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:37.557799    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:38.557541 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:38.557054    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:38.557541 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:38.557106    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:38.557541 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:38.557199    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:38.557541 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:38.557316    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:39.556704 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:39.556674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:39.556835 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:39.556811    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:39.733147 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:39.733120    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n75xf" event={"ID":"01afb354-a5aa-41d8-ae0a-5028e75ae3e9","Type":"ContainerStarted","Data":"507a9694ce6e63e9b1e77096376093afd3493ea3006fc9a6a6307336b5b9b568"}
Jul 27 23:16:39.734511 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:39.734494    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5" event={"ID":"5eedceb6-d520-4108-8b3d-5c6b933d1fb1","Type":"ContainerStarted","Data":"8c285837c30de28cfc31cd7970f0ea7ac99cd25b3b8c281654cd28c8ebe4600e"}
Jul 27 23:16:39.735763 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:39.735745    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ztqx2" event={"ID":"4284f262-15b8-4855-ac5e-3cac44c66357","Type":"ContainerStarted","Data":"4e561a19f5db086745147454962d8a75e4f5b477849cbf26d99f944e07e3e72c"}
Jul 27 23:16:39.737060 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:39.737041    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h8v75" event={"ID":"7ec18511-c498-4407-93eb-d0edabcd4311","Type":"ContainerStarted","Data":"910dca468981423194de99065a4188589663c3760899c58a9c8763f3a417509b"}
Jul 27 23:16:39.738723 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:39.738571    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bcgpc" event={"ID":"4cf21fbf-dc73-405a-836a-f5cf4a7c64d8","Type":"ContainerStarted","Data":"3e743d976982dd3b18d32731781ca537b9c2fcb592e54fb57cfd57a4f43361ec"}
Jul 27 23:16:39.739869 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:39.739850    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxt99" event={"ID":"549494f8-ca88-4fde-84cd-06f3af312efc","Type":"ContainerStarted","Data":"736e53f4769aec2286d0cb401728f82d28225ed5f08261f29d2e6e474e27355c"}
Jul 27 23:16:39.741639 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:39.741620    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mzw9d" event={"ID":"906d2765-8007-4aed-8c0d-2aa12031fa5a","Type":"ContainerStarted","Data":"c25162fe2c090d949e3cf0fd0902695439f3241418421c5616811f63c3933be5"}
Jul 27 23:16:39.772155 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:39.772111    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ztqx2" podStartSLOduration=2.958999367 podStartE2EDuration="21.77209306s" totalImagesPullingTime="18.813093693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:39.75085666 +0000 UTC m=+22.695087406" watchObservedRunningTime="2026-07-27 23:16:39.77209306 +0000 UTC m=+22.716323807"
Jul 27 23:16:39.785217 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:39.785167    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bcgpc" podStartSLOduration=2.979611972 podStartE2EDuration="21.785146849s" totalImagesPullingTime="18.805534877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:39.784366222 +0000 UTC m=+22.728596991" watchObservedRunningTime="2026-07-27 23:16:39.785146849 +0000 UTC m=+22.729377598"
Jul 27 23:16:40.557839 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.557621    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:40.557966 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.557666    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:40.557966 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:40.557885    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:40.557966 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:40.557947    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:40.746341 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.746292    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:16:40.746993 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.746617    2559 generic.go:356] "Generic (PLEG): container finished" podID="5eedceb6-d520-4108-8b3d-5c6b933d1fb1" containerID="8fd08555d18b7f74dea3368adeb930494547df4305717661a147b6ef57687b5e" exitCode=1
Jul 27 23:16:40.746993 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.746690    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5" event={"ID":"5eedceb6-d520-4108-8b3d-5c6b933d1fb1","Type":"ContainerStarted","Data":"f4ef469dcb4aa5c966cceb31a997ce44a760dafafebbaa8d2f52471064d9476c"}
Jul 27 23:16:40.746993 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.746764    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5" event={"ID":"5eedceb6-d520-4108-8b3d-5c6b933d1fb1","Type":"ContainerStarted","Data":"6ce6c5fb8ff9b38f2a0972096ca9435d645be3536b1a131cff63385ce2cb1adc"}
Jul 27 23:16:40.746993 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.746803    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5" event={"ID":"5eedceb6-d520-4108-8b3d-5c6b933d1fb1","Type":"ContainerStarted","Data":"2e026292428f8ad0e5709e1091c0cbfabc799906fd78afc1c1cc42839988dd81"}
Jul 27 23:16:40.746993 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.746831    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5" event={"ID":"5eedceb6-d520-4108-8b3d-5c6b933d1fb1","Type":"ContainerStarted","Data":"a5bac3eb3552be47ac3578fbe4f8a1a7741410f2ac1735773d1075c1e92f1e2b"}
Jul 27 23:16:40.746993 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.746875    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5" event={"ID":"5eedceb6-d520-4108-8b3d-5c6b933d1fb1","Type":"ContainerDied","Data":"8fd08555d18b7f74dea3368adeb930494547df4305717661a147b6ef57687b5e"}
Jul 27 23:16:40.747931 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.747908    2559 generic.go:356] "Generic (PLEG): container finished" podID="549494f8-ca88-4fde-84cd-06f3af312efc" containerID="736e53f4769aec2286d0cb401728f82d28225ed5f08261f29d2e6e474e27355c" exitCode=0
Jul 27 23:16:40.748051 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.747978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxt99" event={"ID":"549494f8-ca88-4fde-84cd-06f3af312efc","Type":"ContainerDied","Data":"736e53f4769aec2286d0cb401728f82d28225ed5f08261f29d2e6e474e27355c"}
Jul 27 23:16:40.749363 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.749346    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs" event={"ID":"635f8680-ff02-492b-a3c3-2e8306d53894","Type":"ContainerStarted","Data":"ee12624239f98e52c2d5fdd04db1dcda047b6003786744193df783d8a7876f12"}
Jul 27 23:16:40.766236 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.766186    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mzw9d" podStartSLOduration=3.958346818 podStartE2EDuration="22.766174804s" totalImagesPullingTime="18.807827986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:39.798908739 +0000 UTC m=+22.743139485" watchObservedRunningTime="2026-07-27 23:16:40.766174804 +0000 UTC m=+23.710405551"
Jul 27 23:16:40.780027 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.779989    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n75xf" podStartSLOduration=3.916459959 podStartE2EDuration="22.779978335s" totalImagesPullingTime="18.863518376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:40.779906801 +0000 UTC m=+23.724137547" watchObservedRunningTime="2026-07-27 23:16:40.779978335 +0000 UTC m=+23.724209080"
Jul 27 23:16:40.791774 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:40.791737    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h8v75" podStartSLOduration=3.968772538 podStartE2EDuration="22.79172427s" totalImagesPullingTime="18.822951732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:40.79131664 +0000 UTC m=+23.735547385" watchObservedRunningTime="2026-07-27 23:16:40.79172427 +0000 UTC m=+23.735955015"
Jul 27 23:16:41.015524 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:41.015500    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 23:16:41.485341 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:41.485239    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T23:16:41.015518538Z","UUID":"4eadc426-4388-44e3-957c-5211e1999178","Handler":null,"Name":"","Endpoint":""}
Jul 27 23:16:41.487147 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:41.487110    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 23:16:41.487282 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:41.487155    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 23:16:41.559935 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:41.559908    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:41.560095 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:41.560024    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:41.756497 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:41.756464    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bxkjp" event={"ID":"3c298f75-ae76-4cb5-9f69-2f0bf4d41758","Type":"ContainerStarted","Data":"a7b267726ffce1724c1ff54919c91cbf1854d6b91fb41af834680b2896d9d7c7"}
Jul 27 23:16:41.759533 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:41.759505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs" event={"ID":"635f8680-ff02-492b-a3c3-2e8306d53894","Type":"ContainerStarted","Data":"9bb30288609560de9880642de188bf8a4b42d0f0d9d7d33c8fd6da1fcc35e3ff"}
Jul 27 23:16:42.557900 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:42.557678    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:42.558049 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:42.558021    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:42.558248 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:42.557711    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:42.558248 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:42.558212    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:42.695658 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:42.695566    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bcgpc"
Jul 27 23:16:42.696289 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:42.696264    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bcgpc"
Jul 27 23:16:42.710814 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:42.710768    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bxkjp" podStartSLOduration=5.867768914 podStartE2EDuration="24.71075494s" totalImagesPullingTime="18.842986026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:41.770291183 +0000 UTC m=+24.714521929" watchObservedRunningTime="2026-07-27 23:16:42.71075494 +0000 UTC m=+25.654985739"
Jul 27 23:16:42.764263 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:42.764223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs" event={"ID":"635f8680-ff02-492b-a3c3-2e8306d53894","Type":"ContainerStarted","Data":"014a872b5d49de336ce130bfc4e1b6e0a17fafcddf42c3787bf9f1047df43bc8"}
Jul 27 23:16:42.767625 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:42.767600    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:16:42.768000 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:42.767966    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5" event={"ID":"5eedceb6-d520-4108-8b3d-5c6b933d1fb1","Type":"ContainerStarted","Data":"6746cce47d4f9202ae475bf47ae51b0105141cd93ff40230cef7ad120f71955e"}
Jul 27 23:16:42.768185 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:42.768166    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bcgpc"
Jul 27 23:16:42.768758 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:42.768742    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bcgpc"
Jul 27 23:16:42.780499 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:42.780463    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs7qs" podStartSLOduration=4.156981226 podStartE2EDuration="24.780449299s" totalImagesPullingTime="20.623468073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:42.779782548 +0000 UTC m=+25.724013294" watchObservedRunningTime="2026-07-27 23:16:42.780449299 +0000 UTC m=+25.724680044"
Jul 27 23:16:43.556841 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:43.556804    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:43.557045 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:43.556943    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:44.557423 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:44.557383    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:44.557917 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:44.557434    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:44.557917 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:44.557518    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:44.557917 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:44.557652    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:45.557291 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:45.557058    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:45.557434 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:45.557314    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:45.776936 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:45.776900    2559 generic.go:356] "Generic (PLEG): container finished" podID="549494f8-ca88-4fde-84cd-06f3af312efc" containerID="695fc22f7d68425bb5cccab132c9361c7316dd4e51a09997ab57b4de525e7d6e" exitCode=0
Jul 27 23:16:45.777664 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:45.776992    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxt99" event={"ID":"549494f8-ca88-4fde-84cd-06f3af312efc","Type":"ContainerDied","Data":"695fc22f7d68425bb5cccab132c9361c7316dd4e51a09997ab57b4de525e7d6e"}
Jul 27 23:16:45.780709 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:45.780687    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:16:45.781094 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:45.781067    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5" event={"ID":"5eedceb6-d520-4108-8b3d-5c6b933d1fb1","Type":"ContainerStarted","Data":"583675a25d05b16653e77282fc8a2771db9f65ae5cbf4e96ded3878359430801"}
Jul 27 23:16:45.781407 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:45.781386    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:45.781466 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:45.781449    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:45.781501 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:45.781489    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:45.781548 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:45.781532    2559 scope.go:117] "RemoveContainer" containerID="8fd08555d18b7f74dea3368adeb930494547df4305717661a147b6ef57687b5e"
Jul 27 23:16:45.797120 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:45.797100    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:45.797435 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:45.797422    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:16:46.557474 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:46.557447    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:46.557597 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:46.557443    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:46.557597 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:46.557572    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:46.557695 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:46.557646    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:46.787129 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:46.787098    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:16:46.787546 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:46.787524    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5" event={"ID":"5eedceb6-d520-4108-8b3d-5c6b933d1fb1","Type":"ContainerStarted","Data":"01882ea833f69c0f67ad634c5e761c53d71639e14c19e27de297dcd5bfa95942"}
Jul 27 23:16:46.789663 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:46.789640    2559 generic.go:356] "Generic (PLEG): container finished" podID="549494f8-ca88-4fde-84cd-06f3af312efc" containerID="182c9835135de273eb22d2acbdee118f1cb516464e925f086687ec5559e6d878" exitCode=0
Jul 27 23:16:46.789663 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:46.789674    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxt99" event={"ID":"549494f8-ca88-4fde-84cd-06f3af312efc","Type":"ContainerDied","Data":"182c9835135de273eb22d2acbdee118f1cb516464e925f086687ec5559e6d878"}
Jul 27 23:16:46.811245 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:46.811076    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5" podStartSLOduration=9.921888148 podStartE2EDuration="28.811064448s" totalImagesPullingTime="18.8891763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:46.810303219 +0000 UTC m=+29.754533964" watchObservedRunningTime="2026-07-27 23:16:46.811064448 +0000 UTC m=+29.755295192"
Jul 27 23:16:46.944511 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:46.944480    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s8znn"]
Jul 27 23:16:46.944670 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:46.944649    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:46.944784 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:46.944759    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:46.947964 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:46.947941    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fhqmk"]
Jul 27 23:16:46.948077 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:46.948033    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:46.948156 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:46.948130    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:46.948747 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:46.948691    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hrrmv"]
Jul 27 23:16:46.948830 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:46.948818    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:46.948941 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:46.948920    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:47.794581 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:47.794497    2559 generic.go:356] "Generic (PLEG): container finished" podID="549494f8-ca88-4fde-84cd-06f3af312efc" containerID="6f54564398f84f9a428444783f58d3e3930d253d0574cbafb78bd49ff914d22a" exitCode=0
Jul 27 23:16:47.794960 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:47.794580    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxt99" event={"ID":"549494f8-ca88-4fde-84cd-06f3af312efc","Type":"ContainerDied","Data":"6f54564398f84f9a428444783f58d3e3930d253d0574cbafb78bd49ff914d22a"}
Jul 27 23:16:48.557185 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:48.557144    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:48.557185 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:48.557177    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:48.557453 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:48.557147    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:48.557453 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:48.557317    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:48.557565 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:48.557497    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:48.557617 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:48.557602    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:50.557177 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:50.557140    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:50.557921 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:50.557140    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:50.557921 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:50.557252    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8znn" podUID="03ff5096-6d95-4ad0-8fc6-c1591c1da37e"
Jul 27 23:16:50.557921 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:50.557148    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:50.557921 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:50.557359    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhqmk" podUID="03e4d553-7ad4-4072-acf6-1d8424cbff5b"
Jul 27 23:16:50.557921 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:50.557491    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:16:51.287235 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:51.287192    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:51.287470 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:51.287259    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x87\" (UniqueName: \"kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87\") pod \"network-check-target-s8znn\" (UID: \"03ff5096-6d95-4ad0-8fc6-c1591c1da37e\") " pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:51.287470 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:51.287298    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:51.287470 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:51.287338    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:51.287470 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:51.287446    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs podName:ee785ab1-83ca-4254-b144-c99f9114660f nodeName:}" failed. No retries permitted until 2026-07-27 23:17:23.287417728 +0000 UTC m=+66.231648469 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs") pod "network-metrics-daemon-hrrmv" (UID: "ee785ab1-83ca-4254-b144-c99f9114660f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:51.287621 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:51.287471    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 23:16:51.287621 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:51.287473    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:51.287621 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:51.287488    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 23:16:51.287621 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:51.287501    2559 projected.go:196] Error preparing data for projected volume kube-api-access-b8x87 for pod openshift-network-diagnostics/network-check-target-s8znn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:51.287621 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:51.287527    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret podName:03e4d553-7ad4-4072-acf6-1d8424cbff5b nodeName:}" failed. No retries permitted until 2026-07-27 23:17:23.287513959 +0000 UTC m=+66.231744685 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret") pod "global-pull-secret-syncer-fhqmk" (UID: "03e4d553-7ad4-4072-acf6-1d8424cbff5b") : object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:51.287621 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:51.287544    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87 podName:03ff5096-6d95-4ad0-8fc6-c1591c1da37e nodeName:}" failed. No retries permitted until 2026-07-27 23:17:23.287533282 +0000 UTC m=+66.231764018 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b8x87" (UniqueName: "kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87") pod "network-check-target-s8znn" (UID: "03ff5096-6d95-4ad0-8fc6-c1591c1da37e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:52.418313 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.418256    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeReady"
Jul 27 23:16:52.418728 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.418388    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 23:16:52.459161 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.459129    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cgwgh"]
Jul 27 23:16:52.471890 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.471865    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rcxmb"]
Jul 27 23:16:52.472577 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.472206    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:52.476053 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.475755    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 23:16:52.476053 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.475820    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t2qdw\""
Jul 27 23:16:52.476680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.476218    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 23:16:52.489584 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.489562    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rcxmb"]
Jul 27 23:16:52.489681 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.489601    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cgwgh"]
Jul 27 23:16:52.489681 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.489634    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rcxmb"
Jul 27 23:16:52.491466 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.491445    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 23:16:52.491616 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.491604    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-txnj2\""
Jul 27 23:16:52.491676 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.491648    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 23:16:52.491730 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.491720    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 23:16:52.557291 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.557265    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:16:52.557418 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.557301    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:16:52.557477 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.557463    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:16:52.559486 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.559465    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 23:16:52.559589 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.559490    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 23:16:52.559589 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.559544    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ft576\""
Jul 27 23:16:52.559701 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.559603    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 23:16:52.559823 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.559809    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 23:16:52.559893 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.559847    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n98qx\""
Jul 27 23:16:52.598868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.598852    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg522\" (UniqueName: \"kubernetes.io/projected/6cb94d17-291c-4e88-8164-7e4664815fdf-kube-api-access-dg522\") pod \"ingress-canary-rcxmb\" (UID: \"6cb94d17-291c-4e88-8164-7e4664815fdf\") " pod="openshift-ingress-canary/ingress-canary-rcxmb"
Jul 27 23:16:52.598967 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.598875    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zdxc\" (UniqueName: \"kubernetes.io/projected/4e467024-a191-49ad-a8ec-29e7923d652e-kube-api-access-4zdxc\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:52.598967 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.598916    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e467024-a191-49ad-a8ec-29e7923d652e-config-volume\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:52.599047 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.599002    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e467024-a191-49ad-a8ec-29e7923d652e-tmp-dir\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:52.599047 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.599028    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cb94d17-291c-4e88-8164-7e4664815fdf-cert\") pod \"ingress-canary-rcxmb\" (UID: \"6cb94d17-291c-4e88-8164-7e4664815fdf\") " pod="openshift-ingress-canary/ingress-canary-rcxmb"
Jul 27 23:16:52.599047 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.599043    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:52.700306 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.700233    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cb94d17-291c-4e88-8164-7e4664815fdf-cert\") pod \"ingress-canary-rcxmb\" (UID: \"6cb94d17-291c-4e88-8164-7e4664815fdf\") " pod="openshift-ingress-canary/ingress-canary-rcxmb"
Jul 27 23:16:52.700306 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.700274    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:52.700518 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.700319    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg522\" (UniqueName: \"kubernetes.io/projected/6cb94d17-291c-4e88-8164-7e4664815fdf-kube-api-access-dg522\") pod \"ingress-canary-rcxmb\" (UID: \"6cb94d17-291c-4e88-8164-7e4664815fdf\") " pod="openshift-ingress-canary/ingress-canary-rcxmb"
Jul 27 23:16:52.700518 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.700344    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zdxc\" (UniqueName: \"kubernetes.io/projected/4e467024-a191-49ad-a8ec-29e7923d652e-kube-api-access-4zdxc\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:52.700518 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:52.700455    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:16:52.700518 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:52.700518    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls podName:4e467024-a191-49ad-a8ec-29e7923d652e nodeName:}" failed. No retries permitted until 2026-07-27 23:16:53.200503373 +0000 UTC m=+36.144734097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls") pod "dns-default-cgwgh" (UID: "4e467024-a191-49ad-a8ec-29e7923d652e") : secret "dns-default-metrics-tls" not found
Jul 27 23:16:52.700692 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.700574    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e467024-a191-49ad-a8ec-29e7923d652e-config-volume\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:52.700744 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.700721    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e467024-a191-49ad-a8ec-29e7923d652e-tmp-dir\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:52.700999 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.700980    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e467024-a191-49ad-a8ec-29e7923d652e-tmp-dir\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:52.701068 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.701054    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e467024-a191-49ad-a8ec-29e7923d652e-config-volume\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:52.709798 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.709775    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zdxc\" (UniqueName: \"kubernetes.io/projected/4e467024-a191-49ad-a8ec-29e7923d652e-kube-api-access-4zdxc\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:52.713802 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.713784    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cb94d17-291c-4e88-8164-7e4664815fdf-cert\") pod \"ingress-canary-rcxmb\" (UID: \"6cb94d17-291c-4e88-8164-7e4664815fdf\") " pod="openshift-ingress-canary/ingress-canary-rcxmb"
Jul 27 23:16:52.715629 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.715609    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg522\" (UniqueName: \"kubernetes.io/projected/6cb94d17-291c-4e88-8164-7e4664815fdf-kube-api-access-dg522\") pod \"ingress-canary-rcxmb\" (UID: \"6cb94d17-291c-4e88-8164-7e4664815fdf\") " pod="openshift-ingress-canary/ingress-canary-rcxmb"
Jul 27 23:16:52.797306 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.797272    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rcxmb"
Jul 27 23:16:52.809107 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.809080    2559 generic.go:356] "Generic (PLEG): container finished" podID="549494f8-ca88-4fde-84cd-06f3af312efc" containerID="799c1eaf7b9e64f65a73384ef869580568c798122cdf177656cba5dd33c311c4" exitCode=0
Jul 27 23:16:52.809226 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.809140    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxt99" event={"ID":"549494f8-ca88-4fde-84cd-06f3af312efc","Type":"ContainerDied","Data":"799c1eaf7b9e64f65a73384ef869580568c798122cdf177656cba5dd33c311c4"}
Jul 27 23:16:52.960251 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:52.959940    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rcxmb"]
Jul 27 23:16:52.963290 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:16:52.963270    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb94d17_291c_4e88_8164_7e4664815fdf.slice/crio-ed9c6329d9ad5006a0d5be0578e67cf9800c1d38415570c3ad5371041e791111 WatchSource:0}: Error finding container ed9c6329d9ad5006a0d5be0578e67cf9800c1d38415570c3ad5371041e791111: Status 404 returned error can't find the container with id ed9c6329d9ad5006a0d5be0578e67cf9800c1d38415570c3ad5371041e791111
Jul 27 23:16:53.205217 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:53.205157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:53.205450 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:53.205313    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:16:53.205450 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:53.205398    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls podName:4e467024-a191-49ad-a8ec-29e7923d652e nodeName:}" failed. No retries permitted until 2026-07-27 23:16:54.205363419 +0000 UTC m=+37.149594147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls") pod "dns-default-cgwgh" (UID: "4e467024-a191-49ad-a8ec-29e7923d652e") : secret "dns-default-metrics-tls" not found
Jul 27 23:16:53.812934 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:53.812895    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rcxmb" event={"ID":"6cb94d17-291c-4e88-8164-7e4664815fdf","Type":"ContainerStarted","Data":"ed9c6329d9ad5006a0d5be0578e67cf9800c1d38415570c3ad5371041e791111"}
Jul 27 23:16:53.817096 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:53.816557    2559 generic.go:356] "Generic (PLEG): container finished" podID="549494f8-ca88-4fde-84cd-06f3af312efc" containerID="e3c6bb7a0447a21902881070ed1b1b4d503f943a8adec871338547afcbcd872f" exitCode=0
Jul 27 23:16:53.817096 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:53.816621    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxt99" event={"ID":"549494f8-ca88-4fde-84cd-06f3af312efc","Type":"ContainerDied","Data":"e3c6bb7a0447a21902881070ed1b1b4d503f943a8adec871338547afcbcd872f"}
Jul 27 23:16:54.212777 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:54.212689    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:54.212945 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:54.212861    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:16:54.212945 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:54.212944    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls podName:4e467024-a191-49ad-a8ec-29e7923d652e nodeName:}" failed. No retries permitted until 2026-07-27 23:16:56.212920633 +0000 UTC m=+39.157151358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls") pod "dns-default-cgwgh" (UID: "4e467024-a191-49ad-a8ec-29e7923d652e") : secret "dns-default-metrics-tls" not found
Jul 27 23:16:54.822577 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:54.822540    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxt99" event={"ID":"549494f8-ca88-4fde-84cd-06f3af312efc","Type":"ContainerStarted","Data":"3ce219cb87da75b6999f8d3fec165f2217e34c567a7214a3ab0b7ea7844b04bc"}
Jul 27 23:16:54.846876 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:54.846826    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nxt99" podStartSLOduration=8.433706169 podStartE2EDuration="36.84680891s" totalImagesPullingTime="28.413102741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:54.845740244 +0000 UTC m=+37.789971189" watchObservedRunningTime="2026-07-27 23:16:54.84680891 +0000 UTC m=+37.791039655"
Jul 27 23:16:55.826889 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:55.826685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rcxmb" event={"ID":"6cb94d17-291c-4e88-8164-7e4664815fdf","Type":"ContainerStarted","Data":"e36da1591c64aa775fde01dc233879bdb4d7fdf40cb0072e0f491729b68e2ddd"}
Jul 27 23:16:55.840884 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:55.840828    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rcxmb" podStartSLOduration=1.19658592 podStartE2EDuration="3.840812828s" totalImagesPullingTime="2.644226908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:55.840019154 +0000 UTC m=+38.784249901" watchObservedRunningTime="2026-07-27 23:16:55.840812828 +0000 UTC m=+38.785043573"
Jul 27 23:16:56.229598 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:16:56.229563    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:16:56.229792 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:56.229739    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:16:56.229862 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:16:56.229817    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls podName:4e467024-a191-49ad-a8ec-29e7923d652e nodeName:}" failed. No retries permitted until 2026-07-27 23:17:00.229797915 +0000 UTC m=+43.174028657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls") pod "dns-default-cgwgh" (UID: "4e467024-a191-49ad-a8ec-29e7923d652e") : secret "dns-default-metrics-tls" not found
Jul 27 23:17:00.259047 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:00.259001    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:17:00.259548 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:17:00.259157    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:17:00.259548 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:17:00.259222    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls podName:4e467024-a191-49ad-a8ec-29e7923d652e nodeName:}" failed. No retries permitted until 2026-07-27 23:17:08.259207309 +0000 UTC m=+51.203438034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls") pod "dns-default-cgwgh" (UID: "4e467024-a191-49ad-a8ec-29e7923d652e") : secret "dns-default-metrics-tls" not found
Jul 27 23:17:08.312069 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:08.312028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:17:08.312546 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:17:08.312189    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:17:08.312546 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:17:08.312271    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls podName:4e467024-a191-49ad-a8ec-29e7923d652e nodeName:}" failed. No retries permitted until 2026-07-27 23:17:24.312252907 +0000 UTC m=+67.256483631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls") pod "dns-default-cgwgh" (UID: "4e467024-a191-49ad-a8ec-29e7923d652e") : secret "dns-default-metrics-tls" not found
Jul 27 23:17:17.805144 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:17.805112    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c27p5"
Jul 27 23:17:23.311567 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.311525    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:17:23.311975 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.311590    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:17:23.311975 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.311631    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x87\" (UniqueName: \"kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87\") pod \"network-check-target-s8znn\" (UID: \"03ff5096-6d95-4ad0-8fc6-c1591c1da37e\") " pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:17:23.313538 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.313519    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 23:17:23.313639 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.313554    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 23:17:23.313639 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.313585    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 23:17:23.322098 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:17:23.322081    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 23:17:23.322153 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:17:23.322136    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs podName:ee785ab1-83ca-4254-b144-c99f9114660f nodeName:}" failed. No retries permitted until 2026-07-27 23:18:27.322120137 +0000 UTC m=+130.266350861 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs") pod "network-metrics-daemon-hrrmv" (UID: "ee785ab1-83ca-4254-b144-c99f9114660f") : secret "metrics-daemon-secret" not found
Jul 27 23:17:23.323639 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.323622    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 23:17:23.325948 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.325929    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03e4d553-7ad4-4072-acf6-1d8424cbff5b-original-pull-secret\") pod \"global-pull-secret-syncer-fhqmk\" (UID: \"03e4d553-7ad4-4072-acf6-1d8424cbff5b\") " pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:17:23.334925 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.334902    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8x87\" (UniqueName: \"kubernetes.io/projected/03ff5096-6d95-4ad0-8fc6-c1591c1da37e-kube-api-access-b8x87\") pod \"network-check-target-s8znn\" (UID: \"03ff5096-6d95-4ad0-8fc6-c1591c1da37e\") " pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:17:23.467889 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.467862    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ft576\""
Jul 27 23:17:23.475609 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.475588    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhqmk"
Jul 27 23:17:23.476679 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.476660    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:17:23.642893 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.642865    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fhqmk"]
Jul 27 23:17:23.646437 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:17:23.646404    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e4d553_7ad4_4072_acf6_1d8424cbff5b.slice/crio-bac8d5fb4adc303391212e9b4b8a82c2704a52b32095e36170780bf52256e098 WatchSource:0}: Error finding container bac8d5fb4adc303391212e9b4b8a82c2704a52b32095e36170780bf52256e098: Status 404 returned error can't find the container with id bac8d5fb4adc303391212e9b4b8a82c2704a52b32095e36170780bf52256e098
Jul 27 23:17:23.660163 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.660143    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s8znn"]
Jul 27 23:17:23.663141 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:17:23.663122    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ff5096_6d95_4ad0_8fc6_c1591c1da37e.slice/crio-2ae0efe76c38471f4c396320facab1244e473afe977b5a00cab5d946d9a6095d WatchSource:0}: Error finding container 2ae0efe76c38471f4c396320facab1244e473afe977b5a00cab5d946d9a6095d: Status 404 returned error can't find the container with id 2ae0efe76c38471f4c396320facab1244e473afe977b5a00cab5d946d9a6095d
Jul 27 23:17:23.895566 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.895486    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fhqmk" event={"ID":"03e4d553-7ad4-4072-acf6-1d8424cbff5b","Type":"ContainerStarted","Data":"bac8d5fb4adc303391212e9b4b8a82c2704a52b32095e36170780bf52256e098"}
Jul 27 23:17:23.896487 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:23.896462    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s8znn" event={"ID":"03ff5096-6d95-4ad0-8fc6-c1591c1da37e","Type":"ContainerStarted","Data":"2ae0efe76c38471f4c396320facab1244e473afe977b5a00cab5d946d9a6095d"}
Jul 27 23:17:24.318289 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:24.318226    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:17:24.318863 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:17:24.318417    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:17:24.318863 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:17:24.318501    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls podName:4e467024-a191-49ad-a8ec-29e7923d652e nodeName:}" failed. No retries permitted until 2026-07-27 23:17:56.318477565 +0000 UTC m=+99.262708290 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls") pod "dns-default-cgwgh" (UID: "4e467024-a191-49ad-a8ec-29e7923d652e") : secret "dns-default-metrics-tls" not found
Jul 27 23:17:28.911102 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:28.911048    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fhqmk" event={"ID":"03e4d553-7ad4-4072-acf6-1d8424cbff5b","Type":"ContainerStarted","Data":"964a044e273fc2fe6549f040fbfb329600dae95a60119be5e8d0a1544969dbbb"}
Jul 27 23:17:28.912462 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:28.912442    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s8znn" event={"ID":"03ff5096-6d95-4ad0-8fc6-c1591c1da37e","Type":"ContainerStarted","Data":"b0020a35f858b3be40a9f5d9d0aa923d0274a1f9fd9fa9117f20ef953e9f12f9"}
Jul 27 23:17:28.912587 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:28.912571    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:17:28.923790 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:28.923743    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fhqmk" podStartSLOduration=66.639487697 podStartE2EDuration="1m10.923728136s" totalImagesPullingTime="4.284240439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:17:28.923347133 +0000 UTC m=+71.867577881" watchObservedRunningTime="2026-07-27 23:17:28.923728136 +0000 UTC m=+71.867958882"
Jul 27 23:17:28.935786 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:28.935751    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s8znn" podStartSLOduration=66.671457364 podStartE2EDuration="1m10.935739058s" totalImagesPullingTime="4.264281694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:17:28.935207555 +0000 UTC m=+71.879438313" watchObservedRunningTime="2026-07-27 23:17:28.935739058 +0000 UTC m=+71.879969803"
Jul 27 23:17:43.406029 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.405892    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"]
Jul 27 23:17:43.409750 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.409732    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"]
Jul 27 23:17:43.409888 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.409870    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:17:43.411808 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.411790    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 23:17:43.411903 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.411880    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 23:17:43.411955 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.411896    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 23:17:43.411992 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.411950    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 27 23:17:43.413434 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.413416    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.416939 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.416919    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 27 23:17:43.418341 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.418319    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 27 23:17:43.418595 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.418577    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 27 23:17:43.419864 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.419849    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 27 23:17:43.421589 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.421550    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"]
Jul 27 23:17:43.422618 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.422598    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"]
Jul 27 23:17:43.447123 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.447100    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1fd33a4-daa0-4020-8356-f847d94d35dc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.447227 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.447129    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q62l2\" (UniqueName: \"kubernetes.io/projected/d1fd33a4-daa0-4020-8356-f847d94d35dc-kube-api-access-q62l2\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.447227 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.447150    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1fd33a4-daa0-4020-8356-f847d94d35dc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.447227 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.447164    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab60ae7c-7cdf-4c47-8c48-5465993cf45b-tmp\") pod \"klusterlet-addon-workmgr-5b794855c6-fhtpq\" (UID: \"ab60ae7c-7cdf-4c47-8c48-5465993cf45b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:17:43.447227 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.447213    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d1fd33a4-daa0-4020-8356-f847d94d35dc-hub\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.447424 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.447301    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ab60ae7c-7cdf-4c47-8c48-5465993cf45b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b794855c6-fhtpq\" (UID: \"ab60ae7c-7cdf-4c47-8c48-5465993cf45b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:17:43.447424 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.447366    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp96j\" (UniqueName: \"kubernetes.io/projected/ab60ae7c-7cdf-4c47-8c48-5465993cf45b-kube-api-access-wp96j\") pod \"klusterlet-addon-workmgr-5b794855c6-fhtpq\" (UID: \"ab60ae7c-7cdf-4c47-8c48-5465993cf45b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:17:43.447509 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.447432    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d1fd33a4-daa0-4020-8356-f847d94d35dc-ca\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.447509 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.447471    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d1fd33a4-daa0-4020-8356-f847d94d35dc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.548350 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.548316    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1fd33a4-daa0-4020-8356-f847d94d35dc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.548350 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.548349    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab60ae7c-7cdf-4c47-8c48-5465993cf45b-tmp\") pod \"klusterlet-addon-workmgr-5b794855c6-fhtpq\" (UID: \"ab60ae7c-7cdf-4c47-8c48-5465993cf45b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:17:43.548608 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.548366    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d1fd33a4-daa0-4020-8356-f847d94d35dc-hub\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.548608 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.548409    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ab60ae7c-7cdf-4c47-8c48-5465993cf45b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b794855c6-fhtpq\" (UID: \"ab60ae7c-7cdf-4c47-8c48-5465993cf45b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:17:43.548608 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.548439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp96j\" (UniqueName: \"kubernetes.io/projected/ab60ae7c-7cdf-4c47-8c48-5465993cf45b-kube-api-access-wp96j\") pod \"klusterlet-addon-workmgr-5b794855c6-fhtpq\" (UID: \"ab60ae7c-7cdf-4c47-8c48-5465993cf45b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:17:43.548608 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.548475    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d1fd33a4-daa0-4020-8356-f847d94d35dc-ca\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.548608 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.548507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d1fd33a4-daa0-4020-8356-f847d94d35dc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.548608 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.548568    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1fd33a4-daa0-4020-8356-f847d94d35dc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.548608 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.548594    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q62l2\" (UniqueName: \"kubernetes.io/projected/d1fd33a4-daa0-4020-8356-f847d94d35dc-kube-api-access-q62l2\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.548919 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.548827    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab60ae7c-7cdf-4c47-8c48-5465993cf45b-tmp\") pod \"klusterlet-addon-workmgr-5b794855c6-fhtpq\" (UID: \"ab60ae7c-7cdf-4c47-8c48-5465993cf45b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:17:43.549137 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.549104    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1fd33a4-daa0-4020-8356-f847d94d35dc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.551335 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.551312    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ab60ae7c-7cdf-4c47-8c48-5465993cf45b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b794855c6-fhtpq\" (UID: \"ab60ae7c-7cdf-4c47-8c48-5465993cf45b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:17:43.551735 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.551717    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d1fd33a4-daa0-4020-8356-f847d94d35dc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.551827 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.551745    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d1fd33a4-daa0-4020-8356-f847d94d35dc-ca\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.551827 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.551771    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d1fd33a4-daa0-4020-8356-f847d94d35dc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.551827 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.551793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d1fd33a4-daa0-4020-8356-f847d94d35dc-hub\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.555945 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.555927    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q62l2\" (UniqueName: \"kubernetes.io/projected/d1fd33a4-daa0-4020-8356-f847d94d35dc-kube-api-access-q62l2\") pod \"cluster-proxy-proxy-agent-6b76c47ddc-47bfq\" (UID: \"d1fd33a4-daa0-4020-8356-f847d94d35dc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.556213 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.556194    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp96j\" (UniqueName: \"kubernetes.io/projected/ab60ae7c-7cdf-4c47-8c48-5465993cf45b-kube-api-access-wp96j\") pod \"klusterlet-addon-workmgr-5b794855c6-fhtpq\" (UID: \"ab60ae7c-7cdf-4c47-8c48-5465993cf45b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:17:43.720813 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.720702    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:17:43.744033 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.744009    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:17:43.857042 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.857004    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"]
Jul 27 23:17:43.860715 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:17:43.860687    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab60ae7c_7cdf_4c47_8c48_5465993cf45b.slice/crio-e3bce158645ef4925ddbce3e0f64153e04a124c8390dab3111ac32cb0c227b33 WatchSource:0}: Error finding container e3bce158645ef4925ddbce3e0f64153e04a124c8390dab3111ac32cb0c227b33: Status 404 returned error can't find the container with id e3bce158645ef4925ddbce3e0f64153e04a124c8390dab3111ac32cb0c227b33
Jul 27 23:17:43.879936 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.879915    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"]
Jul 27 23:17:43.882979 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:17:43.882958    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1fd33a4_daa0_4020_8356_f847d94d35dc.slice/crio-4dc2d4777adc59f868593dc6cdb7cfd01b9c28dc4a978bfb575c149cc7f08762 WatchSource:0}: Error finding container 4dc2d4777adc59f868593dc6cdb7cfd01b9c28dc4a978bfb575c149cc7f08762: Status 404 returned error can't find the container with id 4dc2d4777adc59f868593dc6cdb7cfd01b9c28dc4a978bfb575c149cc7f08762
Jul 27 23:17:43.951656 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.951620    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq" event={"ID":"d1fd33a4-daa0-4020-8356-f847d94d35dc","Type":"ContainerStarted","Data":"4dc2d4777adc59f868593dc6cdb7cfd01b9c28dc4a978bfb575c149cc7f08762"}
Jul 27 23:17:43.952679 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:43.952653    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq" event={"ID":"ab60ae7c-7cdf-4c47-8c48-5465993cf45b","Type":"ContainerStarted","Data":"e3bce158645ef4925ddbce3e0f64153e04a124c8390dab3111ac32cb0c227b33"}
Jul 27 23:17:49.970901 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:49.970864    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq" event={"ID":"d1fd33a4-daa0-4020-8356-f847d94d35dc","Type":"ContainerStarted","Data":"1049df7440e39800f8c6999d1c5165f1072dbc3c710b30a215ec581e6a076085"}
Jul 27 23:17:49.971342 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:49.970910    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq" event={"ID":"d1fd33a4-daa0-4020-8356-f847d94d35dc","Type":"ContainerStarted","Data":"47b422a54b15534f30a14a9b3cb31649b0925a5e764cf0193c3748cc0d54eaf9"}
Jul 27 23:17:49.971342 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:49.970927    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq" event={"ID":"d1fd33a4-daa0-4020-8356-f847d94d35dc","Type":"ContainerStarted","Data":"3af1448f52d110bab401d87b761cd83b32fa5b0967330ef2a39618bab34311a0"}
Jul 27 23:17:49.972266 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:49.972243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq" event={"ID":"ab60ae7c-7cdf-4c47-8c48-5465993cf45b","Type":"ContainerStarted","Data":"e5746682a76f1301c681e5a3e3b69ee23e2cdc78e285df60ab59754582decc8f"}
Jul 27 23:17:49.972401 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:49.972390    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:17:49.973969 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:49.973950    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:17:49.988821 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:49.988763    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq" podStartSLOduration=1.8327334849999999 podStartE2EDuration="6.988747882s" totalImagesPullingTime="5.156014397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:17:49.98743561 +0000 UTC m=+92.931666355" watchObservedRunningTime="2026-07-27 23:17:49.988747882 +0000 UTC m=+92.932978627"
Jul 27 23:17:50.000754 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:50.000714    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq" podStartSLOduration=1.826325294 podStartE2EDuration="7.000700832s" totalImagesPullingTime="5.174375538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:17:49.999692181 +0000 UTC m=+92.943922926" watchObservedRunningTime="2026-07-27 23:17:50.000700832 +0000 UTC m=+92.944931559"
Jul 27 23:17:56.342452 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:56.342399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:17:56.342948 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:17:56.342608    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:17:56.342948 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:17:56.342699    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls podName:4e467024-a191-49ad-a8ec-29e7923d652e nodeName:}" failed. No retries permitted until 2026-07-27 23:19:00.342676704 +0000 UTC m=+163.286907438 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls") pod "dns-default-cgwgh" (UID: "4e467024-a191-49ad-a8ec-29e7923d652e") : secret "dns-default-metrics-tls" not found
Jul 27 23:17:59.917521 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:17:59.917491    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s8znn"
Jul 27 23:18:27.347808 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:18:27.347753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:18:27.348295 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:18:27.347911    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 23:18:27.348295 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:18:27.347985    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs podName:ee785ab1-83ca-4254-b144-c99f9114660f nodeName:}" failed. No retries permitted until 2026-07-27 23:20:29.347968054 +0000 UTC m=+252.292198797 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs") pod "network-metrics-daemon-hrrmv" (UID: "ee785ab1-83ca-4254-b144-c99f9114660f") : secret "metrics-daemon-secret" not found
Jul 27 23:18:55.483939 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:18:55.483896    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-cgwgh" podUID="4e467024-a191-49ad-a8ec-29e7923d652e"
Jul 27 23:18:55.571287 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:18:55.571255    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hrrmv" podUID="ee785ab1-83ca-4254-b144-c99f9114660f"
Jul 27 23:18:56.167298 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:18:56.167269    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:19:00.373135 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:00.373101    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:19:00.373554 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:19:00.373229    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:19:00.373554 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:19:00.373296    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls podName:4e467024-a191-49ad-a8ec-29e7923d652e nodeName:}" failed. No retries permitted until 2026-07-27 23:21:02.373278218 +0000 UTC m=+285.317508943 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls") pod "dns-default-cgwgh" (UID: "4e467024-a191-49ad-a8ec-29e7923d652e") : secret "dns-default-metrics-tls" not found
Jul 27 23:19:07.558387 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:07.558348    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:19:49.317269 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:49.317234    2559 generic.go:356] "Generic (PLEG): container finished" podID="ab60ae7c-7cdf-4c47-8c48-5465993cf45b" containerID="e5746682a76f1301c681e5a3e3b69ee23e2cdc78e285df60ab59754582decc8f" exitCode=1
Jul 27 23:19:49.317675 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:49.317289    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq" event={"ID":"ab60ae7c-7cdf-4c47-8c48-5465993cf45b","Type":"ContainerDied","Data":"e5746682a76f1301c681e5a3e3b69ee23e2cdc78e285df60ab59754582decc8f"}
Jul 27 23:19:49.317675 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:49.317563    2559 scope.go:117] "RemoveContainer" containerID="e5746682a76f1301c681e5a3e3b69ee23e2cdc78e285df60ab59754582decc8f"
Jul 27 23:19:49.973106 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:49.973077    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:19:50.321450 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:50.321418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq" event={"ID":"ab60ae7c-7cdf-4c47-8c48-5465993cf45b","Type":"ContainerStarted","Data":"7ac03b6fec6ee34d0262981a0bb54ed22814e2317b0ac65bd9d4bd6cb56a69ae"}
Jul 27 23:19:50.321876 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:50.321655    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:19:50.322259 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:50.322241    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b794855c6-fhtpq"
Jul 27 23:19:59.372605 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.370509    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-brxvq"]
Jul 27 23:19:59.374136 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.373816    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:19:59.378164 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.378143    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 23:19:59.378615 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.378596    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26z57\""
Jul 27 23:19:59.378942 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.378926    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 23:19:59.379032 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.378967    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 23:19:59.379103 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.379076    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 23:19:59.383548 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.383529    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-brxvq"]
Jul 27 23:19:59.465947 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.465917    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45968a96-5d68-474f-8cc5-2a1520d7fbcc-crio-socket\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:19:59.465947 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.465946    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45968a96-5d68-474f-8cc5-2a1520d7fbcc-data-volume\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:19:59.466160 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.465982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:19:59.466160 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.466020    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vckvv\" (UniqueName: \"kubernetes.io/projected/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-api-access-vckvv\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:19:59.466160 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.466148    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:19:59.566511 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.566483    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:19:59.566511 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.566512    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vckvv\" (UniqueName: \"kubernetes.io/projected/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-api-access-vckvv\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:19:59.566716 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.566540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:19:59.566716 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.566581    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45968a96-5d68-474f-8cc5-2a1520d7fbcc-crio-socket\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:19:59.566716 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.566599    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45968a96-5d68-474f-8cc5-2a1520d7fbcc-data-volume\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:19:59.566716 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:19:59.566628    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 23:19:59.566716 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:19:59.566692    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 23:19:59.566716 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:19:59.566709    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls podName:45968a96-5d68-474f-8cc5-2a1520d7fbcc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:00.066688412 +0000 UTC m=+223.010919141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-brxvq" (UID: "45968a96-5d68-474f-8cc5-2a1520d7fbcc") : secret "insights-runtime-extractor-tls" not found
Jul 27 23:19:59.566967 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.566723    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/45968a96-5d68-474f-8cc5-2a1520d7fbcc-crio-socket\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:19:59.566967 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:19:59.566757    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm podName:45968a96-5d68-474f-8cc5-2a1520d7fbcc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:00.066742245 +0000 UTC m=+223.010972969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-brxvq" (UID: "45968a96-5d68-474f-8cc5-2a1520d7fbcc") : configmap "kube-rbac-proxy" not found
Jul 27 23:19:59.566967 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.566913    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/45968a96-5d68-474f-8cc5-2a1520d7fbcc-data-volume\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:19:59.574464 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:19:59.574439    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vckvv\" (UniqueName: \"kubernetes.io/projected/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-api-access-vckvv\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:20:00.069973 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:00.069938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:20:00.070152 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:00.070005    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:20:00.070152 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:00.070065    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 23:20:00.070152 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:00.070092    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:00.070152 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:00.070143    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm podName:45968a96-5d68-474f-8cc5-2a1520d7fbcc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:01.070126379 +0000 UTC m=+224.014357105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-brxvq" (UID: "45968a96-5d68-474f-8cc5-2a1520d7fbcc") : configmap "kube-rbac-proxy" not found
Jul 27 23:20:00.070293 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:00.070162    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls podName:45968a96-5d68-474f-8cc5-2a1520d7fbcc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:01.070155381 +0000 UTC m=+224.014386109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-brxvq" (UID: "45968a96-5d68-474f-8cc5-2a1520d7fbcc") : secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:01.079790 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:01.079742    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:20:01.079790 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:01.079805    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:20:01.080205 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:01.079888    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 23:20:01.080205 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:01.079902    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:01.080205 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:01.079938    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm podName:45968a96-5d68-474f-8cc5-2a1520d7fbcc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:03.079926111 +0000 UTC m=+226.024156835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-brxvq" (UID: "45968a96-5d68-474f-8cc5-2a1520d7fbcc") : configmap "kube-rbac-proxy" not found
Jul 27 23:20:01.080205 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:01.079957    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls podName:45968a96-5d68-474f-8cc5-2a1520d7fbcc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:03.079944305 +0000 UTC m=+226.024175029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-brxvq" (UID: "45968a96-5d68-474f-8cc5-2a1520d7fbcc") : secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:01.956606 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:01.956578    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h8v75_7ec18511-c498-4407-93eb-d0edabcd4311/dns-node-resolver/0.log"
Jul 27 23:20:02.956890 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:02.956865    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mzw9d_906d2765-8007-4aed-8c0d-2aa12031fa5a/node-ca/0.log"
Jul 27 23:20:03.094491 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:03.094450    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:20:03.094491 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:03.094500    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:20:03.094700 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:03.094578    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 23:20:03.094700 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:03.094611    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:03.094700 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:03.094642    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm podName:45968a96-5d68-474f-8cc5-2a1520d7fbcc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:07.094625674 +0000 UTC m=+230.038856404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-brxvq" (UID: "45968a96-5d68-474f-8cc5-2a1520d7fbcc") : configmap "kube-rbac-proxy" not found
Jul 27 23:20:03.094700 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:03.094664    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls podName:45968a96-5d68-474f-8cc5-2a1520d7fbcc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:07.094650078 +0000 UTC m=+230.038880802 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-brxvq" (UID: "45968a96-5d68-474f-8cc5-2a1520d7fbcc") : secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:04.157057 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:04.157027    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rcxmb_6cb94d17-291c-4e88-8164-7e4664815fdf/serve-healthcheck-canary/0.log"
Jul 27 23:20:07.122384 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:07.122347    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:20:07.122795 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:07.122411    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:20:07.122795 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:07.122488    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 23:20:07.122795 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:07.122489    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:07.122795 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:07.122541    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm podName:45968a96-5d68-474f-8cc5-2a1520d7fbcc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:15.122527189 +0000 UTC m=+238.066757913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-brxvq" (UID: "45968a96-5d68-474f-8cc5-2a1520d7fbcc") : configmap "kube-rbac-proxy" not found
Jul 27 23:20:07.122795 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:07.122554    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls podName:45968a96-5d68-474f-8cc5-2a1520d7fbcc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:15.122548103 +0000 UTC m=+238.066778827 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-brxvq" (UID: "45968a96-5d68-474f-8cc5-2a1520d7fbcc") : secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:15.182742 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:15.182696    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:20:15.183219 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:15.182768    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:20:15.183394 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:15.183353    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/45968a96-5d68-474f-8cc5-2a1520d7fbcc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:20:15.185104 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:15.185081    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/45968a96-5d68-474f-8cc5-2a1520d7fbcc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-brxvq\" (UID: \"45968a96-5d68-474f-8cc5-2a1520d7fbcc\") " pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:20:15.282222 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:15.282199    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-brxvq"
Jul 27 23:20:15.403240 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:15.403208    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-brxvq"]
Jul 27 23:20:15.406068 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:20:15.406043    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45968a96_5d68_474f_8cc5_2a1520d7fbcc.slice/crio-6391e0704d9c31df1be59036446a04e8d3fb96e95bbb2cd7f3b120135e30e579 WatchSource:0}: Error finding container 6391e0704d9c31df1be59036446a04e8d3fb96e95bbb2cd7f3b120135e30e579: Status 404 returned error can't find the container with id 6391e0704d9c31df1be59036446a04e8d3fb96e95bbb2cd7f3b120135e30e579
Jul 27 23:20:16.396064 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:16.396015    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-brxvq" event={"ID":"45968a96-5d68-474f-8cc5-2a1520d7fbcc","Type":"ContainerStarted","Data":"6391e0704d9c31df1be59036446a04e8d3fb96e95bbb2cd7f3b120135e30e579"}
Jul 27 23:20:20.411434 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:20.411400    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-brxvq" event={"ID":"45968a96-5d68-474f-8cc5-2a1520d7fbcc","Type":"ContainerStarted","Data":"7bf6883a3dd2f9408b7897a89cea1bd692fe62a53a4b22ec9599a3b0e5c2ff4f"}
Jul 27 23:20:24.037533 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.037463    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gpjrq"]
Jul 27 23:20:24.040317 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.040299    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gpjrq"
Jul 27 23:20:24.042572 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.042553    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hfcdl\""
Jul 27 23:20:24.043062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.043047    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 27 23:20:24.052770 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.052713    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gpjrq"]
Jul 27 23:20:24.066298 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.066282    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-nrjl2"]
Jul 27 23:20:24.068971 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.068958    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-nrjl2"
Jul 27 23:20:24.071403 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.071367    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 23:20:24.071509 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.071495    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fvvbm\""
Jul 27 23:20:24.071575 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.071558    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 23:20:24.079450 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.079434    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-nrjl2"]
Jul 27 23:20:24.149076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.149050    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj8mr\" (UniqueName: \"kubernetes.io/projected/2423c014-fc07-4851-addd-b65230b2631e-kube-api-access-kj8mr\") pod \"downloads-7c97dc4555-nrjl2\" (UID: \"2423c014-fc07-4851-addd-b65230b2631e\") " pod="openshift-console/downloads-7c97dc4555-nrjl2"
Jul 27 23:20:24.149187 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.149083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3545a128-2826-409f-a4bc-a91cc78d34a2-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-gpjrq\" (UID: \"3545a128-2826-409f-a4bc-a91cc78d34a2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gpjrq"
Jul 27 23:20:24.250312 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.250289    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3545a128-2826-409f-a4bc-a91cc78d34a2-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-gpjrq\" (UID: \"3545a128-2826-409f-a4bc-a91cc78d34a2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gpjrq"
Jul 27 23:20:24.250429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.250389    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj8mr\" (UniqueName: \"kubernetes.io/projected/2423c014-fc07-4851-addd-b65230b2631e-kube-api-access-kj8mr\") pod \"downloads-7c97dc4555-nrjl2\" (UID: \"2423c014-fc07-4851-addd-b65230b2631e\") " pod="openshift-console/downloads-7c97dc4555-nrjl2"
Jul 27 23:20:24.252759 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.252735    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3545a128-2826-409f-a4bc-a91cc78d34a2-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-gpjrq\" (UID: \"3545a128-2826-409f-a4bc-a91cc78d34a2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gpjrq"
Jul 27 23:20:24.261100 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.261074    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj8mr\" (UniqueName: \"kubernetes.io/projected/2423c014-fc07-4851-addd-b65230b2631e-kube-api-access-kj8mr\") pod \"downloads-7c97dc4555-nrjl2\" (UID: \"2423c014-fc07-4851-addd-b65230b2631e\") " pod="openshift-console/downloads-7c97dc4555-nrjl2"
Jul 27 23:20:24.349214 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.349138    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gpjrq"
Jul 27 23:20:24.377626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.377584    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-nrjl2"
Jul 27 23:20:24.427143 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.427102    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-brxvq" event={"ID":"45968a96-5d68-474f-8cc5-2a1520d7fbcc","Type":"ContainerStarted","Data":"d07afaf47d82487c31c00ae905c359c108c278cf953ed4bd1ba51a4efdb48e0f"}
Jul 27 23:20:24.483812 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.483785    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gpjrq"]
Jul 27 23:20:24.486888 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:20:24.486859    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3545a128_2826_409f_a4bc_a91cc78d34a2.slice/crio-bdac8cceb147a07b8e065b7a974b7f3de7c2d4eaa865b3cc5a810bf2d0042526 WatchSource:0}: Error finding container bdac8cceb147a07b8e065b7a974b7f3de7c2d4eaa865b3cc5a810bf2d0042526: Status 404 returned error can't find the container with id bdac8cceb147a07b8e065b7a974b7f3de7c2d4eaa865b3cc5a810bf2d0042526
Jul 27 23:20:24.508400 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:24.508355    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-nrjl2"]
Jul 27 23:20:24.510449 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:20:24.510418    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2423c014_fc07_4851_addd_b65230b2631e.slice/crio-db09546766cebcb22e820af4a3cdfee78941dd2b0555e6643a03a753f12065bd WatchSource:0}: Error finding container db09546766cebcb22e820af4a3cdfee78941dd2b0555e6643a03a753f12065bd: Status 404 returned error can't find the container with id db09546766cebcb22e820af4a3cdfee78941dd2b0555e6643a03a753f12065bd
Jul 27 23:20:25.434068 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:25.434027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-nrjl2" event={"ID":"2423c014-fc07-4851-addd-b65230b2631e","Type":"ContainerStarted","Data":"db09546766cebcb22e820af4a3cdfee78941dd2b0555e6643a03a753f12065bd"}
Jul 27 23:20:25.435801 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:25.435763    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gpjrq" event={"ID":"3545a128-2826-409f-a4bc-a91cc78d34a2","Type":"ContainerStarted","Data":"bdac8cceb147a07b8e065b7a974b7f3de7c2d4eaa865b3cc5a810bf2d0042526"}
Jul 27 23:20:26.440749 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:26.440714    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gpjrq" event={"ID":"3545a128-2826-409f-a4bc-a91cc78d34a2","Type":"ContainerStarted","Data":"077840d3b66b5f559f981ab854e23d9c25bcd4de5cedffa7d9d611ab3d0ef1c3"}
Jul 27 23:20:26.441185 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:26.440942    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gpjrq"
Jul 27 23:20:26.446434 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:26.446412    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gpjrq"
Jul 27 23:20:26.455092 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:26.455023    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gpjrq" podStartSLOduration=0.977776292 podStartE2EDuration="2.455006699s" totalImagesPullingTime="1.477230407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:26.454791979 +0000 UTC m=+249.399022725" watchObservedRunningTime="2026-07-27 23:20:26.455006699 +0000 UTC m=+249.399237447"
Jul 27 23:20:28.451623 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:28.451583    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-brxvq" event={"ID":"45968a96-5d68-474f-8cc5-2a1520d7fbcc","Type":"ContainerStarted","Data":"30416b17c30f5927db16f769aeed58311bdaf650f298d384de3110f71cbaf50c"}
Jul 27 23:20:28.468986 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:28.468879    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-brxvq" podStartSLOduration=16.772716791 podStartE2EDuration="29.468860738s" totalImagesPullingTime="12.696143947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:19:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:28.467360399 +0000 UTC m=+251.411591146" watchObservedRunningTime="2026-07-27 23:20:28.468860738 +0000 UTC m=+251.413091485"
Jul 27 23:20:29.385933 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:29.385894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:20:29.388617 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:29.388593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee785ab1-83ca-4254-b144-c99f9114660f-metrics-certs\") pod \"network-metrics-daemon-hrrmv\" (UID: \"ee785ab1-83ca-4254-b144-c99f9114660f\") " pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:20:29.461014 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:29.460986    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n98qx\""
Jul 27 23:20:29.469734 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:29.469707    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hrrmv"
Jul 27 23:20:29.608601 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:29.608571    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hrrmv"]
Jul 27 23:20:29.611938 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:20:29.611912    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee785ab1_83ca_4254_b144_c99f9114660f.slice/crio-c3d3a7d9d232d8861e4fa1db9d2cfd1639c2f70f1690d994abc56f66f9b95fec WatchSource:0}: Error finding container c3d3a7d9d232d8861e4fa1db9d2cfd1639c2f70f1690d994abc56f66f9b95fec: Status 404 returned error can't find the container with id c3d3a7d9d232d8861e4fa1db9d2cfd1639c2f70f1690d994abc56f66f9b95fec
Jul 27 23:20:30.461072 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:30.461042    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hrrmv" event={"ID":"ee785ab1-83ca-4254-b144-c99f9114660f","Type":"ContainerStarted","Data":"c3d3a7d9d232d8861e4fa1db9d2cfd1639c2f70f1690d994abc56f66f9b95fec"}
Jul 27 23:20:31.466205 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:31.466164    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hrrmv" event={"ID":"ee785ab1-83ca-4254-b144-c99f9114660f","Type":"ContainerStarted","Data":"0f548ff9884a2ff14a473176e63477ec30473dc755e9eb2864d8c794162ecb73"}
Jul 27 23:20:31.466701 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:31.466232    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hrrmv" event={"ID":"ee785ab1-83ca-4254-b144-c99f9114660f","Type":"ContainerStarted","Data":"2b87ef84ac981e8eca238ee9a453ded5435b4d34daab56f5863519090aa9ba2d"}
Jul 27 23:20:31.479694 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:31.479494    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hrrmv" podStartSLOduration=252.579083463 podStartE2EDuration="4m13.479476954s" totalImagesPullingTime="900.393491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:31.479339165 +0000 UTC m=+254.423569904" watchObservedRunningTime="2026-07-27 23:20:31.479476954 +0000 UTC m=+254.423707700"
Jul 27 23:20:32.744549 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.744520    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"]
Jul 27 23:20:32.747855 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.747830    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"
Jul 27 23:20:32.750536 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.750511    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 27 23:20:32.751467 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.751105    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 23:20:32.751467 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.751234    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2f4ws\""
Jul 27 23:20:32.751467 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.751233    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 23:20:32.751467 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.751328    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 23:20:32.751720 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.751516    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 27 23:20:32.757761 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.757739    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"]
Jul 27 23:20:32.769277 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.769252    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kxntv"]
Jul 27 23:20:32.772451 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.772435    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:32.774564 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.774532    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 23:20:32.774677 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.774657    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2gwgm\""
Jul 27 23:20:32.774778 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.774747    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 23:20:32.774910 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.774873    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 23:20:32.915301 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.915267    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/21dfaac0-d939-42c1-a6d7-8259fd89dc40-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-dskc8\" (UID: \"21dfaac0-d939-42c1-a6d7-8259fd89dc40\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"
Jul 27 23:20:32.915497 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.915315    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21dfaac0-d939-42c1-a6d7-8259fd89dc40-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-dskc8\" (UID: \"21dfaac0-d939-42c1-a6d7-8259fd89dc40\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"
Jul 27 23:20:32.915497 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.915348    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-textfile\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:32.915497 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.915446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/181cca67-ce91-4b5b-b2dc-d905813b1b16-sys\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:32.915497 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.915484    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:32.915690 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.915514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21dfaac0-d939-42c1-a6d7-8259fd89dc40-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-dskc8\" (UID: \"21dfaac0-d939-42c1-a6d7-8259fd89dc40\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"
Jul 27 23:20:32.915690 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.915567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/181cca67-ce91-4b5b-b2dc-d905813b1b16-root\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:32.915690 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.915613    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgw95\" (UniqueName: \"kubernetes.io/projected/21dfaac0-d939-42c1-a6d7-8259fd89dc40-kube-api-access-mgw95\") pod \"openshift-state-metrics-b8f49f5c6-dskc8\" (UID: \"21dfaac0-d939-42c1-a6d7-8259fd89dc40\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"
Jul 27 23:20:32.915690 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.915653    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-accelerators-collector-config\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:32.915690 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.915671    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-wtmp\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:32.916075 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.915692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-tls\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:32.916075 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.915714    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/181cca67-ce91-4b5b-b2dc-d905813b1b16-metrics-client-ca\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:32.916075 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:32.915760    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755qz\" (UniqueName: \"kubernetes.io/projected/181cca67-ce91-4b5b-b2dc-d905813b1b16-kube-api-access-755qz\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.016610 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.016534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755qz\" (UniqueName: \"kubernetes.io/projected/181cca67-ce91-4b5b-b2dc-d905813b1b16-kube-api-access-755qz\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.016610 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.016586    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/21dfaac0-d939-42c1-a6d7-8259fd89dc40-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-dskc8\" (UID: \"21dfaac0-d939-42c1-a6d7-8259fd89dc40\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"
Jul 27 23:20:33.016922 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.016612    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21dfaac0-d939-42c1-a6d7-8259fd89dc40-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-dskc8\" (UID: \"21dfaac0-d939-42c1-a6d7-8259fd89dc40\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"
Jul 27 23:20:33.016922 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.016644    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-textfile\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.016922 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.016674    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/181cca67-ce91-4b5b-b2dc-d905813b1b16-sys\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.016922 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.016696    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.016922 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.016723    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21dfaac0-d939-42c1-a6d7-8259fd89dc40-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-dskc8\" (UID: \"21dfaac0-d939-42c1-a6d7-8259fd89dc40\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"
Jul 27 23:20:33.016922 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.016752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/181cca67-ce91-4b5b-b2dc-d905813b1b16-root\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.016922 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.016790    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgw95\" (UniqueName: \"kubernetes.io/projected/21dfaac0-d939-42c1-a6d7-8259fd89dc40-kube-api-access-mgw95\") pod \"openshift-state-metrics-b8f49f5c6-dskc8\" (UID: \"21dfaac0-d939-42c1-a6d7-8259fd89dc40\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"
Jul 27 23:20:33.016922 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.016816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-accelerators-collector-config\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.016922 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.016840    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-wtmp\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.016922 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.016864    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-tls\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.017448 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:33.016939    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 23:20:33.017448 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.016980    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-textfile\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.017448 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:33.016992    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-tls podName:181cca67-ce91-4b5b-b2dc-d905813b1b16 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:33.516973156 +0000 UTC m=+256.461203884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-tls") pod "node-exporter-kxntv" (UID: "181cca67-ce91-4b5b-b2dc-d905813b1b16") : secret "node-exporter-tls" not found
Jul 27 23:20:33.017448 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.017048    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/181cca67-ce91-4b5b-b2dc-d905813b1b16-root\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.017448 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.017108    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/181cca67-ce91-4b5b-b2dc-d905813b1b16-sys\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.017448 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.017133    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-wtmp\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.017448 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.017162    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/181cca67-ce91-4b5b-b2dc-d905813b1b16-metrics-client-ca\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.017816 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.017628    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21dfaac0-d939-42c1-a6d7-8259fd89dc40-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-dskc8\" (UID: \"21dfaac0-d939-42c1-a6d7-8259fd89dc40\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"
Jul 27 23:20:33.017816 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.017754    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/181cca67-ce91-4b5b-b2dc-d905813b1b16-metrics-client-ca\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.017917 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.017842    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-accelerators-collector-config\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.019490 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.019464    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/21dfaac0-d939-42c1-a6d7-8259fd89dc40-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-dskc8\" (UID: \"21dfaac0-d939-42c1-a6d7-8259fd89dc40\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"
Jul 27 23:20:33.019490 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.019483    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21dfaac0-d939-42c1-a6d7-8259fd89dc40-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-dskc8\" (UID: \"21dfaac0-d939-42c1-a6d7-8259fd89dc40\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"
Jul 27 23:20:33.019668 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.019585    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.024086 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.024062    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgw95\" (UniqueName: \"kubernetes.io/projected/21dfaac0-d939-42c1-a6d7-8259fd89dc40-kube-api-access-mgw95\") pod \"openshift-state-metrics-b8f49f5c6-dskc8\" (UID: \"21dfaac0-d939-42c1-a6d7-8259fd89dc40\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"
Jul 27 23:20:33.024193 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.024151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-755qz\" (UniqueName: \"kubernetes.io/projected/181cca67-ce91-4b5b-b2dc-d905813b1b16-kube-api-access-755qz\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.059367 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.059318    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"
Jul 27 23:20:33.207643 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.204363    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8"]
Jul 27 23:20:33.207775 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:20:33.207747    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21dfaac0_d939_42c1_a6d7_8259fd89dc40.slice/crio-e41855589face79f8ec39ba55fa7b04739d12074eaa4efb42e93dcd3d96f98c3 WatchSource:0}: Error finding container e41855589face79f8ec39ba55fa7b04739d12074eaa4efb42e93dcd3d96f98c3: Status 404 returned error can't find the container with id e41855589face79f8ec39ba55fa7b04739d12074eaa4efb42e93dcd3d96f98c3
Jul 27 23:20:33.476907 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.476874    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8" event={"ID":"21dfaac0-d939-42c1-a6d7-8259fd89dc40","Type":"ContainerStarted","Data":"4e0c857f8972e073ced79a137b28f4d2a893ea0bab6481f0a983098b51a20f64"}
Jul 27 23:20:33.477082 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.476940    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8" event={"ID":"21dfaac0-d939-42c1-a6d7-8259fd89dc40","Type":"ContainerStarted","Data":"32d13237c800db9859063a053cfae25cc88ca54d86cf1897eb6c492de03f6f5d"}
Jul 27 23:20:33.477082 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.476969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8" event={"ID":"21dfaac0-d939-42c1-a6d7-8259fd89dc40","Type":"ContainerStarted","Data":"e41855589face79f8ec39ba55fa7b04739d12074eaa4efb42e93dcd3d96f98c3"}
Jul 27 23:20:33.522361 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.522328    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-tls\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.524846 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.524793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/181cca67-ce91-4b5b-b2dc-d905813b1b16-node-exporter-tls\") pod \"node-exporter-kxntv\" (UID: \"181cca67-ce91-4b5b-b2dc-d905813b1b16\") " pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.684807 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:33.684782    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kxntv"
Jul 27 23:20:33.695645 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:20:33.695621    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181cca67_ce91_4b5b_b2dc_d905813b1b16.slice/crio-a6c9c1e149788172cb1fbc575e9cf29ac4c4929009656e7b368b526e8f53a805 WatchSource:0}: Error finding container a6c9c1e149788172cb1fbc575e9cf29ac4c4929009656e7b368b526e8f53a805: Status 404 returned error can't find the container with id a6c9c1e149788172cb1fbc575e9cf29ac4c4929009656e7b368b526e8f53a805
Jul 27 23:20:34.482451 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:34.482420    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxntv" event={"ID":"181cca67-ce91-4b5b-b2dc-d905813b1b16","Type":"ContainerStarted","Data":"a6c9c1e149788172cb1fbc575e9cf29ac4c4929009656e7b368b526e8f53a805"}
Jul 27 23:20:38.941967 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.941891    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 23:20:38.948445 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.948405    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.950860 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.950836    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 23:20:38.950991 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.950864    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 23:20:38.951247 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.951164    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 23:20:38.951247 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.951168    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 23:20:38.951466 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.951307    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 23:20:38.951466 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.951358    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gd4nk\""
Jul 27 23:20:38.951466 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.951459    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 23:20:38.951641 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.951471    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 23:20:38.951641 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.951549    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 23:20:38.951919 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.951759    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fvksq3en77u6e\""
Jul 27 23:20:38.951919 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.951765    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 23:20:38.951919 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.951797    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 23:20:38.952131 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.951935    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 23:20:38.952131 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.952090    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 23:20:38.952232 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.952202    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 23:20:38.952232 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.952222    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 23:20:38.953485 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.953462    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 23:20:38.961465 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.961441    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 23:20:38.961938 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.961919    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8546e214-fa5f-4016-b32b-fd76bf309988-config-out\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962038 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.961949    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-config\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962038 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.961972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962038 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.961999    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8546e214-fa5f-4016-b32b-fd76bf309988-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962187 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962187 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962110    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-web-config\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962187 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962158    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962328 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962328 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962221    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962328 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962238    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962328 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962256    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962328 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962304    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962572 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962339    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962572 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962363    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962572 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962414    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962572 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962462    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962572 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962572 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962533    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962572 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962566    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4zn\" (UniqueName: \"kubernetes.io/projected/8546e214-fa5f-4016-b32b-fd76bf309988-kube-api-access-hm4zn\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:38.962827 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:38.962597    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063237 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063237 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063247    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4zn\" (UniqueName: \"kubernetes.io/projected/8546e214-fa5f-4016-b32b-fd76bf309988-kube-api-access-hm4zn\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063514 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063273    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063514 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:39.063403    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-trusted-ca-bundle podName:8546e214-fa5f-4016-b32b-fd76bf309988 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:39.563357578 +0000 UTC m=+262.507588306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-trusted-ca-bundle") pod "prometheus-k8s-0" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988") : configmap references non-existent config key: ca-bundle.crt
Jul 27 23:20:39.063514 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063499    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8546e214-fa5f-4016-b32b-fd76bf309988-config-out\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063677 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063522    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-config\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063677 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063538    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063677 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063559    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8546e214-fa5f-4016-b32b-fd76bf309988-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063677 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063580    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063677 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-web-config\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063677 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063946 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063691    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063946 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063946 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063742    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063946 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063946 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063946 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063818    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063946 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063846    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063946 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063872    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063946 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063904    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.063946 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.063939    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.064688 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.064663    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.064781 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.064763    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.064924 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.064905    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.066829 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.066799    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8546e214-fa5f-4016-b32b-fd76bf309988-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.067032 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.066992    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.067639 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.067211    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.067812 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.067744    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-config\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.068884 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.068430    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.068884 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.068503    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.068884 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.068845    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.069070 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.068893    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-web-config\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.069592 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.069450    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.069845 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.069819    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.070341 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.070210    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8546e214-fa5f-4016-b32b-fd76bf309988-config-out\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.070341 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.070253    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.070904 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.070880    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.071258 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.071240    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.072290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.072271    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.072914 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.072896    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4zn\" (UniqueName: \"kubernetes.io/projected/8546e214-fa5f-4016-b32b-fd76bf309988-kube-api-access-hm4zn\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.567330 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.567292    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.568197 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.568161    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:39.859789 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:39.859705    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:42.136360 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:42.136338    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 23:20:42.138456 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:20:42.138432    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8546e214_fa5f_4016_b32b_fd76bf309988.slice/crio-f7ec5758970958aed9ac2e21ed34ea72947c4d47a7a6b0257d05c55018055c36 WatchSource:0}: Error finding container f7ec5758970958aed9ac2e21ed34ea72947c4d47a7a6b0257d05c55018055c36: Status 404 returned error can't find the container with id f7ec5758970958aed9ac2e21ed34ea72947c4d47a7a6b0257d05c55018055c36
Jul 27 23:20:42.518746 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:42.518655    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerStarted","Data":"f7ec5758970958aed9ac2e21ed34ea72947c4d47a7a6b0257d05c55018055c36"}
Jul 27 23:20:42.520507 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:42.520480    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-nrjl2" event={"ID":"2423c014-fc07-4851-addd-b65230b2631e","Type":"ContainerStarted","Data":"6a2b267fc0dba7a5f2912499714375ace893aaafda26b1a178462cf68c4858d4"}
Jul 27 23:20:42.520910 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:42.520888    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-nrjl2"
Jul 27 23:20:42.522832 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:42.522811    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8" event={"ID":"21dfaac0-d939-42c1-a6d7-8259fd89dc40","Type":"ContainerStarted","Data":"9873cd9cd568766cc2bfa6a629570005f347e61af2cb27a2e1a9095c249a63cc"}
Jul 27 23:20:42.524549 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:42.524525    2559 generic.go:356] "Generic (PLEG): container finished" podID="181cca67-ce91-4b5b-b2dc-d905813b1b16" containerID="1d26e4fe40b737588d3e22d8d7328ea1a5f0d00aeb9baa663a51454ac5233da0" exitCode=0
Jul 27 23:20:42.524678 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:42.524612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxntv" event={"ID":"181cca67-ce91-4b5b-b2dc-d905813b1b16","Type":"ContainerDied","Data":"1d26e4fe40b737588d3e22d8d7328ea1a5f0d00aeb9baa663a51454ac5233da0"}
Jul 27 23:20:42.530468 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:42.530453    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-nrjl2"
Jul 27 23:20:42.535895 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:42.535858    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-nrjl2" podStartSLOduration=1.022726521 podStartE2EDuration="18.535845744s" totalImagesPullingTime="17.513119223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:42.533425158 +0000 UTC m=+265.477655901" watchObservedRunningTime="2026-07-27 23:20:42.535845744 +0000 UTC m=+265.480076484"
Jul 27 23:20:42.578104 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:42.578050    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dskc8" podStartSLOduration=1.945039507 podStartE2EDuration="10.578031583s" totalImagesPullingTime="8.632992076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:42.57669954 +0000 UTC m=+265.520930284" watchObservedRunningTime="2026-07-27 23:20:42.578031583 +0000 UTC m=+265.522262327"
Jul 27 23:20:43.532088 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:43.532048    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxntv" event={"ID":"181cca67-ce91-4b5b-b2dc-d905813b1b16","Type":"ContainerStarted","Data":"eaa3f36fcf69a9d93d1a704910cc52b2f4ba43ce20922b270a1cb19871c66889"}
Jul 27 23:20:43.532581 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:43.532113    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kxntv" event={"ID":"181cca67-ce91-4b5b-b2dc-d905813b1b16","Type":"ContainerStarted","Data":"09ac67f509d72decc659a8a5540ff5d9f638a3d733fc26db227dce084bae856e"}
Jul 27 23:20:43.548003 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:43.547953    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kxntv" podStartSLOduration=3.149239243 podStartE2EDuration="11.547937648s" totalImagesPullingTime="8.294432278s" totalInitContainerRuntime="104.266127ms" isStatefulPod=true podCreationTimestamp="2026-07-27 23:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:43.54648205 +0000 UTC m=+266.490712807" watchObservedRunningTime="2026-07-27 23:20:43.547937648 +0000 UTC m=+266.492168393"
Jul 27 23:20:44.538272 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:44.538236    2559 generic.go:356] "Generic (PLEG): container finished" podID="8546e214-fa5f-4016-b32b-fd76bf309988" containerID="7298c6a7c742a477ba592096c22f49e25ac9fcfa20faeeeee4803652a918ca27" exitCode=0
Jul 27 23:20:44.538750 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:44.538324    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerDied","Data":"7298c6a7c742a477ba592096c22f49e25ac9fcfa20faeeeee4803652a918ca27"}
Jul 27 23:20:49.569708 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:49.569575    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerStarted","Data":"a54514182cbf1691d7ec2010f7ffb31f07ea86b738e7abdfd07b32dd543ac820"}
Jul 27 23:20:49.569708 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:49.569653    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerStarted","Data":"1e94745b0a05defcd85b809d40dfc98319a1e89f07ca8547b9b6e261486c29b2"}
Jul 27 23:20:51.585525 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:51.585489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerStarted","Data":"989c05df3a525506dc194c8154f2d45e96ab1ec860a8d03a171592116bea116f"}
Jul 27 23:20:51.585912 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:51.585564    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerStarted","Data":"72e10da57d805447c9b4297c168a8eba8a65a517606c780863099566e2ecbd8a"}
Jul 27 23:20:51.585912 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:51.585605    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerStarted","Data":"4d929f319e458d182e74e2eb306f6f388c72ec35e03e3dd8afe002ee60ed7509"}
Jul 27 23:20:52.591745 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:52.591707    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerStarted","Data":"eed83602c058b274eb956c832243201bae9c38f993b7dd22ab78f6f0e0286f5d"}
Jul 27 23:20:52.616735 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:52.616684    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.687487904 podStartE2EDuration="14.616662935s" totalImagesPullingTime="7.907899674s" totalInitContainerRuntime="21.275357ms" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:52.614687167 +0000 UTC m=+275.558917913" watchObservedRunningTime="2026-07-27 23:20:52.616662935 +0000 UTC m=+275.560893681"
Jul 27 23:20:53.745942 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:53.745886    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq" podUID="d1fd33a4-daa0-4020-8356-f847d94d35dc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 27 23:20:54.860084 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:54.860048    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:20:59.167954 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:20:59.167909    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-cgwgh" podUID="4e467024-a191-49ad-a8ec-29e7923d652e"
Jul 27 23:20:59.618040 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:20:59.618005    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:21:02.460805 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:02.460770    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:21:02.463216 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:02.463196    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e467024-a191-49ad-a8ec-29e7923d652e-metrics-tls\") pod \"dns-default-cgwgh\" (UID: \"4e467024-a191-49ad-a8ec-29e7923d652e\") " pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:21:02.621407 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:02.621359    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t2qdw\""
Jul 27 23:21:02.629621 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:02.629599    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:21:02.789326 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:02.789289    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cgwgh"]
Jul 27 23:21:02.792364 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:21:02.792338    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e467024_a191_49ad_a8ec_29e7923d652e.slice/crio-5f9ed232a46c79c64dbd1529d6ee581e81d279c7758811d1204fa17b84bd6126 WatchSource:0}: Error finding container 5f9ed232a46c79c64dbd1529d6ee581e81d279c7758811d1204fa17b84bd6126: Status 404 returned error can't find the container with id 5f9ed232a46c79c64dbd1529d6ee581e81d279c7758811d1204fa17b84bd6126
Jul 27 23:21:03.635213 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:03.635173    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cgwgh" event={"ID":"4e467024-a191-49ad-a8ec-29e7923d652e","Type":"ContainerStarted","Data":"5f9ed232a46c79c64dbd1529d6ee581e81d279c7758811d1204fa17b84bd6126"}
Jul 27 23:21:03.745582 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:03.745531    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq" podUID="d1fd33a4-daa0-4020-8356-f847d94d35dc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 27 23:21:05.644193 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:05.644161    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cgwgh" event={"ID":"4e467024-a191-49ad-a8ec-29e7923d652e","Type":"ContainerStarted","Data":"67c0fe3a5dd7aa9ce115a802b4dd4764aad5d3e166625649301d394b922a988a"}
Jul 27 23:21:05.644597 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:05.644209    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cgwgh" event={"ID":"4e467024-a191-49ad-a8ec-29e7923d652e","Type":"ContainerStarted","Data":"174bfefcc56826d19bf0966ade531599d97f2149c950d4b7a4d54a5e5bedde31"}
Jul 27 23:21:05.644597 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:05.644301    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:21:05.660647 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:05.660602    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cgwgh" podStartSLOduration=251.586301148 podStartE2EDuration="4m13.660588587s" totalImagesPullingTime="2.074287439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:21:05.659038678 +0000 UTC m=+288.603269424" watchObservedRunningTime="2026-07-27 23:21:05.660588587 +0000 UTC m=+288.604819333"
Jul 27 23:21:13.745447 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:13.745411    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq" podUID="d1fd33a4-daa0-4020-8356-f847d94d35dc" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 27 23:21:13.745896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:13.745484    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq"
Jul 27 23:21:13.745971 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:13.745911    2559 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"1049df7440e39800f8c6999d1c5165f1072dbc3c710b30a215ec581e6a076085"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 27 23:21:13.745971 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:13.745957    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq" podUID="d1fd33a4-daa0-4020-8356-f847d94d35dc" containerName="service-proxy" containerID="cri-o://1049df7440e39800f8c6999d1c5165f1072dbc3c710b30a215ec581e6a076085" gracePeriod=30
Jul 27 23:21:14.677128 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:14.677095    2559 generic.go:356] "Generic (PLEG): container finished" podID="d1fd33a4-daa0-4020-8356-f847d94d35dc" containerID="1049df7440e39800f8c6999d1c5165f1072dbc3c710b30a215ec581e6a076085" exitCode=2
Jul 27 23:21:14.677389 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:14.677176    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq" event={"ID":"d1fd33a4-daa0-4020-8356-f847d94d35dc","Type":"ContainerDied","Data":"1049df7440e39800f8c6999d1c5165f1072dbc3c710b30a215ec581e6a076085"}
Jul 27 23:21:14.677389 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:14.677219    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b76c47ddc-47bfq" event={"ID":"d1fd33a4-daa0-4020-8356-f847d94d35dc","Type":"ContainerStarted","Data":"aaa6799c38947039a0d6cccc2df94a5c42a203c82ddf9dba756a1ee9ddbdec01"}
Jul 27 23:21:15.650625 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:15.650598    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cgwgh"
Jul 27 23:21:17.423538 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:17.423500    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:21:17.435511 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:17.435480    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:21:17.435674 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:17.435603    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 23:21:29.873489 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:29.873464    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rcxmb_6cb94d17-291c-4e88-8164-7e4664815fdf/serve-healthcheck-canary/0.log"
Jul 27 23:21:39.860196 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:39.860150    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:39.879443 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:39.879420    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:40.781802 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:40.781774    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:57.282565 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.282527    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 23:21:57.283057 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.282832    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="prometheus" containerID="cri-o://1e94745b0a05defcd85b809d40dfc98319a1e89f07ca8547b9b6e261486c29b2" gracePeriod=600
Jul 27 23:21:57.283057 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.282853    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="kube-rbac-proxy" containerID="cri-o://989c05df3a525506dc194c8154f2d45e96ab1ec860a8d03a171592116bea116f" gracePeriod=600
Jul 27 23:21:57.283057 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.282900    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="kube-rbac-proxy-thanos" containerID="cri-o://eed83602c058b274eb956c832243201bae9c38f993b7dd22ab78f6f0e0286f5d" gracePeriod=600
Jul 27 23:21:57.283057 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.282894    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="config-reloader" containerID="cri-o://a54514182cbf1691d7ec2010f7ffb31f07ea86b738e7abdfd07b32dd543ac820" gracePeriod=600
Jul 27 23:21:57.283057 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.282872    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="thanos-sidecar" containerID="cri-o://4d929f319e458d182e74e2eb306f6f388c72ec35e03e3dd8afe002ee60ed7509" gracePeriod=600
Jul 27 23:21:57.283057 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.282903    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="kube-rbac-proxy-web" containerID="cri-o://72e10da57d805447c9b4297c168a8eba8a65a517606c780863099566e2ecbd8a" gracePeriod=600
Jul 27 23:21:57.829005 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.828977    2559 generic.go:356] "Generic (PLEG): container finished" podID="8546e214-fa5f-4016-b32b-fd76bf309988" containerID="eed83602c058b274eb956c832243201bae9c38f993b7dd22ab78f6f0e0286f5d" exitCode=0
Jul 27 23:21:57.829005 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.829000    2559 generic.go:356] "Generic (PLEG): container finished" podID="8546e214-fa5f-4016-b32b-fd76bf309988" containerID="989c05df3a525506dc194c8154f2d45e96ab1ec860a8d03a171592116bea116f" exitCode=0
Jul 27 23:21:57.829005 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.829006    2559 generic.go:356] "Generic (PLEG): container finished" podID="8546e214-fa5f-4016-b32b-fd76bf309988" containerID="4d929f319e458d182e74e2eb306f6f388c72ec35e03e3dd8afe002ee60ed7509" exitCode=0
Jul 27 23:21:57.829005 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.829011    2559 generic.go:356] "Generic (PLEG): container finished" podID="8546e214-fa5f-4016-b32b-fd76bf309988" containerID="a54514182cbf1691d7ec2010f7ffb31f07ea86b738e7abdfd07b32dd543ac820" exitCode=0
Jul 27 23:21:57.829252 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.829016    2559 generic.go:356] "Generic (PLEG): container finished" podID="8546e214-fa5f-4016-b32b-fd76bf309988" containerID="1e94745b0a05defcd85b809d40dfc98319a1e89f07ca8547b9b6e261486c29b2" exitCode=0
Jul 27 23:21:57.829252 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.829035    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerDied","Data":"eed83602c058b274eb956c832243201bae9c38f993b7dd22ab78f6f0e0286f5d"}
Jul 27 23:21:57.829252 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.829085    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerDied","Data":"989c05df3a525506dc194c8154f2d45e96ab1ec860a8d03a171592116bea116f"}
Jul 27 23:21:57.829252 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.829108    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerDied","Data":"4d929f319e458d182e74e2eb306f6f388c72ec35e03e3dd8afe002ee60ed7509"}
Jul 27 23:21:57.829252 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.829133    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerDied","Data":"a54514182cbf1691d7ec2010f7ffb31f07ea86b738e7abdfd07b32dd543ac820"}
Jul 27 23:21:57.829252 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:57.829156    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerDied","Data":"1e94745b0a05defcd85b809d40dfc98319a1e89f07ca8547b9b6e261486c29b2"}
Jul 27 23:21:58.518568 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.518547    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.578348 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.578323    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8546e214-fa5f-4016-b32b-fd76bf309988-config-out\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.578497 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.578356    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-grpc-tls\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.578497 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.578398    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-serving-certs-ca-bundle\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.578497 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.578424    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-config\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.578497 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.578448    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.578497 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.578480    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-metrics-client-ca\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.578747 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.578511    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.578997 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.578865    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:21:58.578997 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.578992    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-1\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.579182 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579027    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm4zn\" (UniqueName: \"kubernetes.io/projected/8546e214-fa5f-4016-b32b-fd76bf309988-kube-api-access-hm4zn\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.579182 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579041    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:21:58.579287 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579184    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-tls\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.579287 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579211    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-metrics-client-certs\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.579418 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579291    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-web-config\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.579418 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579320    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-2\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.579418 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579344    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-0\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.579418 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579396    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-kubelet-serving-ca-bundle\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.579631 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579427    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-thanos-prometheus-http-client-file\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.579631 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579448    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-trusted-ca-bundle\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.579631 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579487    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-kube-rbac-proxy\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.579631 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579501    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:21:58.579631 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579531    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-db\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.579631 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579582    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8546e214-fa5f-4016-b32b-fd76bf309988-tls-assets\") pod \"8546e214-fa5f-4016-b32b-fd76bf309988\" (UID: \"8546e214-fa5f-4016-b32b-fd76bf309988\") "
Jul 27 23:21:58.579916 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579774    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:21:58.579916 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579791    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.579916 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579811    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-metrics-client-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.579916 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.579826    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.581604 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.581575    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:58.581934 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.581912    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:21:58.582030 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.581971    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:58.582147 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.582052    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:21:58.582420 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.582347    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:21:58.582517 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.582482    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:21:58.582698 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.582669    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-config" (OuterVolumeSpecName: "config") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:58.583319 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.583280    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8546e214-fa5f-4016-b32b-fd76bf309988-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:21:58.583585 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.583558    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:58.583879 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.583848    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8546e214-fa5f-4016-b32b-fd76bf309988-config-out" (OuterVolumeSpecName: "config-out") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:21:58.584847 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.584816    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:58.585012 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.584984    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:58.585012 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.585005    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:58.585012 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.584986    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8546e214-fa5f-4016-b32b-fd76bf309988-kube-api-access-hm4zn" (OuterVolumeSpecName: "kube-api-access-hm4zn") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "kube-api-access-hm4zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:21:58.585484 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.585469    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:58.594635 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.594614    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-web-config" (OuterVolumeSpecName: "web-config") pod "8546e214-fa5f-4016-b32b-fd76bf309988" (UID: "8546e214-fa5f-4016-b32b-fd76bf309988"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:58.680877 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680802    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.680877 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680830    2559 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.680877 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680839    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.680877 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680849    2559 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-kube-rbac-proxy\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.680877 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680858    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-db\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.680877 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680866    2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8546e214-fa5f-4016-b32b-fd76bf309988-tls-assets\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.680877 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680875    2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8546e214-fa5f-4016-b32b-fd76bf309988-config-out\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.680877 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680882    2559 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-grpc-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.681194 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680889    2559 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.681194 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680897    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.681194 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680906    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.681194 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680913    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm4zn\" (UniqueName: \"kubernetes.io/projected/8546e214-fa5f-4016-b32b-fd76bf309988-kube-api-access-hm4zn\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.681194 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680921    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.681194 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680929    2559 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-secret-metrics-client-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.681194 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680937    2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8546e214-fa5f-4016-b32b-fd76bf309988-web-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.681194 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680944    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.681194 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.680951    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8546e214-fa5f-4016-b32b-fd76bf309988-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:21:58.835772 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.835739    2559 generic.go:356] "Generic (PLEG): container finished" podID="8546e214-fa5f-4016-b32b-fd76bf309988" containerID="72e10da57d805447c9b4297c168a8eba8a65a517606c780863099566e2ecbd8a" exitCode=0
Jul 27 23:21:58.835924 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.835829    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerDied","Data":"72e10da57d805447c9b4297c168a8eba8a65a517606c780863099566e2ecbd8a"}
Jul 27 23:21:58.835924 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.835920    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8546e214-fa5f-4016-b32b-fd76bf309988","Type":"ContainerDied","Data":"f7ec5758970958aed9ac2e21ed34ea72947c4d47a7a6b0257d05c55018055c36"}
Jul 27 23:21:58.835993 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.835842    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.835993 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.835951    2559 scope.go:117] "RemoveContainer" containerID="eed83602c058b274eb956c832243201bae9c38f993b7dd22ab78f6f0e0286f5d"
Jul 27 23:21:58.844609 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.844587    2559 scope.go:117] "RemoveContainer" containerID="989c05df3a525506dc194c8154f2d45e96ab1ec860a8d03a171592116bea116f"
Jul 27 23:21:58.851157 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.851141    2559 scope.go:117] "RemoveContainer" containerID="72e10da57d805447c9b4297c168a8eba8a65a517606c780863099566e2ecbd8a"
Jul 27 23:21:58.857687 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.857673    2559 scope.go:117] "RemoveContainer" containerID="4d929f319e458d182e74e2eb306f6f388c72ec35e03e3dd8afe002ee60ed7509"
Jul 27 23:21:58.859495 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.859479    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 23:21:58.863305 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.863283    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 23:21:58.865677 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.865663    2559 scope.go:117] "RemoveContainer" containerID="a54514182cbf1691d7ec2010f7ffb31f07ea86b738e7abdfd07b32dd543ac820"
Jul 27 23:21:58.872190 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.872175    2559 scope.go:117] "RemoveContainer" containerID="1e94745b0a05defcd85b809d40dfc98319a1e89f07ca8547b9b6e261486c29b2"
Jul 27 23:21:58.879172 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.879156    2559 scope.go:117] "RemoveContainer" containerID="7298c6a7c742a477ba592096c22f49e25ac9fcfa20faeeeee4803652a918ca27"
Jul 27 23:21:58.885884 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.885850    2559 scope.go:117] "RemoveContainer" containerID="eed83602c058b274eb956c832243201bae9c38f993b7dd22ab78f6f0e0286f5d"
Jul 27 23:21:58.886351 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:21:58.886329    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed83602c058b274eb956c832243201bae9c38f993b7dd22ab78f6f0e0286f5d\": container with ID starting with eed83602c058b274eb956c832243201bae9c38f993b7dd22ab78f6f0e0286f5d not found: ID does not exist" containerID="eed83602c058b274eb956c832243201bae9c38f993b7dd22ab78f6f0e0286f5d"
Jul 27 23:21:58.886448 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.886359    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed83602c058b274eb956c832243201bae9c38f993b7dd22ab78f6f0e0286f5d"} err="failed to get container status \"eed83602c058b274eb956c832243201bae9c38f993b7dd22ab78f6f0e0286f5d\": rpc error: code = NotFound desc = could not find container \"eed83602c058b274eb956c832243201bae9c38f993b7dd22ab78f6f0e0286f5d\": container with ID starting with eed83602c058b274eb956c832243201bae9c38f993b7dd22ab78f6f0e0286f5d not found: ID does not exist"
Jul 27 23:21:58.886448 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.886396    2559 scope.go:117] "RemoveContainer" containerID="989c05df3a525506dc194c8154f2d45e96ab1ec860a8d03a171592116bea116f"
Jul 27 23:21:58.886691 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:21:58.886674    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989c05df3a525506dc194c8154f2d45e96ab1ec860a8d03a171592116bea116f\": container with ID starting with 989c05df3a525506dc194c8154f2d45e96ab1ec860a8d03a171592116bea116f not found: ID does not exist" containerID="989c05df3a525506dc194c8154f2d45e96ab1ec860a8d03a171592116bea116f"
Jul 27 23:21:58.886740 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.886697    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989c05df3a525506dc194c8154f2d45e96ab1ec860a8d03a171592116bea116f"} err="failed to get container status \"989c05df3a525506dc194c8154f2d45e96ab1ec860a8d03a171592116bea116f\": rpc error: code = NotFound desc = could not find container \"989c05df3a525506dc194c8154f2d45e96ab1ec860a8d03a171592116bea116f\": container with ID starting with 989c05df3a525506dc194c8154f2d45e96ab1ec860a8d03a171592116bea116f not found: ID does not exist"
Jul 27 23:21:58.886740 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.886721    2559 scope.go:117] "RemoveContainer" containerID="72e10da57d805447c9b4297c168a8eba8a65a517606c780863099566e2ecbd8a"
Jul 27 23:21:58.886953 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:21:58.886938    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72e10da57d805447c9b4297c168a8eba8a65a517606c780863099566e2ecbd8a\": container with ID starting with 72e10da57d805447c9b4297c168a8eba8a65a517606c780863099566e2ecbd8a not found: ID does not exist" containerID="72e10da57d805447c9b4297c168a8eba8a65a517606c780863099566e2ecbd8a"
Jul 27 23:21:58.886995 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.886956    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72e10da57d805447c9b4297c168a8eba8a65a517606c780863099566e2ecbd8a"} err="failed to get container status \"72e10da57d805447c9b4297c168a8eba8a65a517606c780863099566e2ecbd8a\": rpc error: code = NotFound desc = could not find container \"72e10da57d805447c9b4297c168a8eba8a65a517606c780863099566e2ecbd8a\": container with ID starting with 72e10da57d805447c9b4297c168a8eba8a65a517606c780863099566e2ecbd8a not found: ID does not exist"
Jul 27 23:21:58.886995 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.886968    2559 scope.go:117] "RemoveContainer" containerID="4d929f319e458d182e74e2eb306f6f388c72ec35e03e3dd8afe002ee60ed7509"
Jul 27 23:21:58.887105 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887088    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 23:21:58.887240 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:21:58.887224    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d929f319e458d182e74e2eb306f6f388c72ec35e03e3dd8afe002ee60ed7509\": container with ID starting with 4d929f319e458d182e74e2eb306f6f388c72ec35e03e3dd8afe002ee60ed7509 not found: ID does not exist" containerID="4d929f319e458d182e74e2eb306f6f388c72ec35e03e3dd8afe002ee60ed7509"
Jul 27 23:21:58.887278 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887246    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d929f319e458d182e74e2eb306f6f388c72ec35e03e3dd8afe002ee60ed7509"} err="failed to get container status \"4d929f319e458d182e74e2eb306f6f388c72ec35e03e3dd8afe002ee60ed7509\": rpc error: code = NotFound desc = could not find container \"4d929f319e458d182e74e2eb306f6f388c72ec35e03e3dd8afe002ee60ed7509\": container with ID starting with 4d929f319e458d182e74e2eb306f6f388c72ec35e03e3dd8afe002ee60ed7509 not found: ID does not exist"
Jul 27 23:21:58.887278 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887261    2559 scope.go:117] "RemoveContainer" containerID="a54514182cbf1691d7ec2010f7ffb31f07ea86b738e7abdfd07b32dd543ac820"
Jul 27 23:21:58.887383 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887355    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="init-config-reloader"
Jul 27 23:21:58.887472 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887435    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="init-config-reloader"
Jul 27 23:21:58.887472 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887454    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="kube-rbac-proxy-thanos"
Jul 27 23:21:58.887472 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887465    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="kube-rbac-proxy-thanos"
Jul 27 23:21:58.887575 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887477    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="prometheus"
Jul 27 23:21:58.887575 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887500    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="prometheus"
Jul 27 23:21:58.887575 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887512    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="config-reloader"
Jul 27 23:21:58.887575 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887518    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="config-reloader"
Jul 27 23:21:58.887575 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887525    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="kube-rbac-proxy-web"
Jul 27 23:21:58.887575 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887530    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="kube-rbac-proxy-web"
Jul 27 23:21:58.887575 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887537    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="thanos-sidecar"
Jul 27 23:21:58.887575 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887541    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="thanos-sidecar"
Jul 27 23:21:58.887575 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887546    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="kube-rbac-proxy"
Jul 27 23:21:58.887575 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887551    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="kube-rbac-proxy"
Jul 27 23:21:58.887845 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887608    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="prometheus"
Jul 27 23:21:58.887845 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:21:58.887540    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54514182cbf1691d7ec2010f7ffb31f07ea86b738e7abdfd07b32dd543ac820\": container with ID starting with a54514182cbf1691d7ec2010f7ffb31f07ea86b738e7abdfd07b32dd543ac820 not found: ID does not exist" containerID="a54514182cbf1691d7ec2010f7ffb31f07ea86b738e7abdfd07b32dd543ac820"
Jul 27 23:21:58.887845 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887634    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54514182cbf1691d7ec2010f7ffb31f07ea86b738e7abdfd07b32dd543ac820"} err="failed to get container status \"a54514182cbf1691d7ec2010f7ffb31f07ea86b738e7abdfd07b32dd543ac820\": rpc error: code = NotFound desc = could not find container \"a54514182cbf1691d7ec2010f7ffb31f07ea86b738e7abdfd07b32dd543ac820\": container with ID starting with a54514182cbf1691d7ec2010f7ffb31f07ea86b738e7abdfd07b32dd543ac820 not found: ID does not exist"
Jul 27 23:21:58.887845 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887654    2559 scope.go:117] "RemoveContainer" containerID="1e94745b0a05defcd85b809d40dfc98319a1e89f07ca8547b9b6e261486c29b2"
Jul 27 23:21:58.887845 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887616    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="thanos-sidecar"
Jul 27 23:21:58.887845 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887698    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="config-reloader"
Jul 27 23:21:58.887845 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887703    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="kube-rbac-proxy"
Jul 27 23:21:58.887845 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887710    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="kube-rbac-proxy-web"
Jul 27 23:21:58.887845 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887715    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" containerName="kube-rbac-proxy-thanos"
Jul 27 23:21:58.888118 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:21:58.887869    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e94745b0a05defcd85b809d40dfc98319a1e89f07ca8547b9b6e261486c29b2\": container with ID starting with 1e94745b0a05defcd85b809d40dfc98319a1e89f07ca8547b9b6e261486c29b2 not found: ID does not exist" containerID="1e94745b0a05defcd85b809d40dfc98319a1e89f07ca8547b9b6e261486c29b2"
Jul 27 23:21:58.888118 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887888    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e94745b0a05defcd85b809d40dfc98319a1e89f07ca8547b9b6e261486c29b2"} err="failed to get container status \"1e94745b0a05defcd85b809d40dfc98319a1e89f07ca8547b9b6e261486c29b2\": rpc error: code = NotFound desc = could not find container \"1e94745b0a05defcd85b809d40dfc98319a1e89f07ca8547b9b6e261486c29b2\": container with ID starting with 1e94745b0a05defcd85b809d40dfc98319a1e89f07ca8547b9b6e261486c29b2 not found: ID does not exist"
Jul 27 23:21:58.888118 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.887902    2559 scope.go:117] "RemoveContainer" containerID="7298c6a7c742a477ba592096c22f49e25ac9fcfa20faeeeee4803652a918ca27"
Jul 27 23:21:58.888118 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:21:58.888098    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7298c6a7c742a477ba592096c22f49e25ac9fcfa20faeeeee4803652a918ca27\": container with ID starting with 7298c6a7c742a477ba592096c22f49e25ac9fcfa20faeeeee4803652a918ca27 not found: ID does not exist" containerID="7298c6a7c742a477ba592096c22f49e25ac9fcfa20faeeeee4803652a918ca27"
Jul 27 23:21:58.888234 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.888127    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7298c6a7c742a477ba592096c22f49e25ac9fcfa20faeeeee4803652a918ca27"} err="failed to get container status \"7298c6a7c742a477ba592096c22f49e25ac9fcfa20faeeeee4803652a918ca27\": rpc error: code = NotFound desc = could not find container \"7298c6a7c742a477ba592096c22f49e25ac9fcfa20faeeeee4803652a918ca27\": container with ID starting with 7298c6a7c742a477ba592096c22f49e25ac9fcfa20faeeeee4803652a918ca27 not found: ID does not exist"
Jul 27 23:21:58.891001 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.890987    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.893063 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.893045    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 23:21:58.893141 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.893131    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fvksq3en77u6e\""
Jul 27 23:21:58.893206 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.893138    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 23:21:58.893206 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.893153    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 23:21:58.893678 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.893647    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 23:21:58.893774 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.893681    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 23:21:58.893774 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.893692    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 23:21:58.893774 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.893676    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 23:21:58.893774 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.893729    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 23:21:58.893961 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.893798    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gd4nk\""
Jul 27 23:21:58.893961 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.893808    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 23:21:58.893961 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.893845    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 23:21:58.893961 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.893882    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 23:21:58.894106 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.894069    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 23:21:58.894106 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.894102    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 23:21:58.896702 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.896685    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 23:21:58.898707 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.898681    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 23:21:58.904508 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.904491    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 23:21:58.982984 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.982929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.982984 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.982956    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.982984 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.982975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983126 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.982990    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983126 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983126 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983069    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a07de815-9624-4abe-bbf5-00fcf93109fd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983204 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983132    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983204 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983160    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-web-config\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983204 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983195    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983211    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983227    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983245    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-config\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983263    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqjgs\" (UniqueName: \"kubernetes.io/projected/a07de815-9624-4abe-bbf5-00fcf93109fd-kube-api-access-lqjgs\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983287    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a07de815-9624-4abe-bbf5-00fcf93109fd-config-out\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983461 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983300    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983461 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983329    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983461 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983358    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983461 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983461 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983428    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:58.983461 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:58.983443    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.084399 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084350    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-config\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.084509 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084409    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqjgs\" (UniqueName: \"kubernetes.io/projected/a07de815-9624-4abe-bbf5-00fcf93109fd-kube-api-access-lqjgs\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.084509 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a07de815-9624-4abe-bbf5-00fcf93109fd-config-out\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.084509 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084466    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.084509 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084490    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.084702 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084514    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.084702 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.084702 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084574    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.084702 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.084702 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084633    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.084702 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.084702 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.085028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.085028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.085028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a07de815-9624-4abe-bbf5-00fcf93109fd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.085028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.085028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-web-config\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.085028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084866    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.085028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084890    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.085028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.084915    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.085420 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.085157    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.085612 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.085551    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.086094 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.086071    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.086303 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.086281    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.086391 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.086327    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.086976 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.086954    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.087450 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.087281    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a07de815-9624-4abe-bbf5-00fcf93109fd-config-out\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.087558 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.087460    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-config\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.087691 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.087673    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.088400 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.088342    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.088912 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.088885    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.089007 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.088942    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a07de815-9624-4abe-bbf5-00fcf93109fd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.089187 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.089166    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a07de815-9624-4abe-bbf5-00fcf93109fd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.089309 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.089179    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.089942 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.089923    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.090250 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.090230    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.090314 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.090300    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.090430 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.090415    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.090789 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.090771    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a07de815-9624-4abe-bbf5-00fcf93109fd-web-config\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.091719 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.091700    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqjgs\" (UniqueName: \"kubernetes.io/projected/a07de815-9624-4abe-bbf5-00fcf93109fd-kube-api-access-lqjgs\") pod \"prometheus-k8s-0\" (UID: \"a07de815-9624-4abe-bbf5-00fcf93109fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.200411 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.200391    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:21:59.330116 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.330090    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 23:21:59.331548 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:21:59.331527    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07de815_9624_4abe_bbf5_00fcf93109fd.slice/crio-9d06e808ae7d6cd0c02d199beada7178ca9d049339eedce77a2580bba8cbdaee WatchSource:0}: Error finding container 9d06e808ae7d6cd0c02d199beada7178ca9d049339eedce77a2580bba8cbdaee: Status 404 returned error can't find the container with id 9d06e808ae7d6cd0c02d199beada7178ca9d049339eedce77a2580bba8cbdaee
Jul 27 23:21:59.561746 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.561717    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8546e214-fa5f-4016-b32b-fd76bf309988" path="/var/lib/kubelet/pods/8546e214-fa5f-4016-b32b-fd76bf309988/volumes"
Jul 27 23:21:59.842524 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.842439    2559 generic.go:356] "Generic (PLEG): container finished" podID="a07de815-9624-4abe-bbf5-00fcf93109fd" containerID="00eab5776f28a0bf2c952f9a1ebb6777460958879e8d0a5d38c49e0bfdfbe377" exitCode=0
Jul 27 23:21:59.842725 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.842527    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a07de815-9624-4abe-bbf5-00fcf93109fd","Type":"ContainerDied","Data":"00eab5776f28a0bf2c952f9a1ebb6777460958879e8d0a5d38c49e0bfdfbe377"}
Jul 27 23:21:59.842725 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:21:59.842613    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a07de815-9624-4abe-bbf5-00fcf93109fd","Type":"ContainerStarted","Data":"9d06e808ae7d6cd0c02d199beada7178ca9d049339eedce77a2580bba8cbdaee"}
Jul 27 23:22:00.849730 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:22:00.849694    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a07de815-9624-4abe-bbf5-00fcf93109fd","Type":"ContainerStarted","Data":"d0a61e4a2f4413e17c87df14753455b95366f03e616a701206df1a9558ce9360"}
Jul 27 23:22:00.850083 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:22:00.849785    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a07de815-9624-4abe-bbf5-00fcf93109fd","Type":"ContainerStarted","Data":"131eee03efcd647687fa551860f3c5e594af0518a3b9e132bfc073035677dfc6"}
Jul 27 23:22:00.850083 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:22:00.849810    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a07de815-9624-4abe-bbf5-00fcf93109fd","Type":"ContainerStarted","Data":"a47c42d78118b953806e0e98d001307c32a9ca20c7968537d23f4995e90dc788"}
Jul 27 23:22:00.850083 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:22:00.849834    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a07de815-9624-4abe-bbf5-00fcf93109fd","Type":"ContainerStarted","Data":"476a39d5be72611d7a425f5981038c8a8e70362828f374005357bff545d54363"}
Jul 27 23:22:00.850083 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:22:00.849856    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a07de815-9624-4abe-bbf5-00fcf93109fd","Type":"ContainerStarted","Data":"e93ffd4699c42afc79d210870bbba3cc339cdc51e83e8d4d5d2501288131a0b7"}
Jul 27 23:22:00.850083 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:22:00.849877    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a07de815-9624-4abe-bbf5-00fcf93109fd","Type":"ContainerStarted","Data":"b5b9ad56d5ee2584e3de5914203c49b3342c9cff92788a69f9ef7cf25daeb67b"}
Jul 27 23:22:00.873527 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:22:00.873482    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.854666805 podStartE2EDuration="2.873468801s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.801996ms" isStatefulPod=false podCreationTimestamp="2026-07-27 23:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:22:00.872540393 +0000 UTC m=+343.816771139" watchObservedRunningTime="2026-07-27 23:22:00.873468801 +0000 UTC m=+343.817699585"
Jul 27 23:22:04.201315 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:22:04.201279    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:22:59.201244 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:22:59.201211    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:22:59.216865 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:22:59.216836    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:23:00.071724 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:23:00.071697    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 23:25:09.047299 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.047221    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"]
Jul 27 23:25:09.049248 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.049233    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.055051 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.052077    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ts86f\""
Jul 27 23:25:09.055051 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.052169    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 23:25:09.055051 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.052348    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 23:25:09.055051 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.052361    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 23:25:09.055646 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.055624    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 23:25:09.067201 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.067177    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"]
Jul 27 23:25:09.071033 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.071007    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfnpb\" (UniqueName: \"kubernetes.io/projected/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-kube-api-access-gfnpb\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.071134 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.071074    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-manifests\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.071134 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.071102    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.071222 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.071145    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-tmp\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.071222 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.071174    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.172181 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.172148    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-manifests\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.172181 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.172185    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.172463 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.172387    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-tmp\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.172463 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.172423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.172576 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.172470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfnpb\" (UniqueName: \"kubernetes.io/projected/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-kube-api-access-gfnpb\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.172704 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.172674    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-manifests\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.174727 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.174690    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-tmp\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.174853 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.174833    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.174920 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.174883    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.181150 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.181125    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfnpb\" (UniqueName: \"kubernetes.io/projected/2f6fc328-0e1c-4e46-8a6d-81460e187c3d-kube-api-access-gfnpb\") pod \"opendatahub-operator-controller-manager-9cf867746-hb2hn\" (UID: \"2f6fc328-0e1c-4e46-8a6d-81460e187c3d\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.363191 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.363101    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:09.492049 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.492017    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"]
Jul 27 23:25:09.495320 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:25:09.495292    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6fc328_0e1c_4e46_8a6d_81460e187c3d.slice/crio-994c018112638b61dc1eada6ca67aa32bf0d47a18a789a81f3d9ee9385603052 WatchSource:0}: Error finding container 994c018112638b61dc1eada6ca67aa32bf0d47a18a789a81f3d9ee9385603052: Status 404 returned error can't find the container with id 994c018112638b61dc1eada6ca67aa32bf0d47a18a789a81f3d9ee9385603052
Jul 27 23:25:09.497128 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.497110    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 23:25:09.510356 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:09.510330    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn" event={"ID":"2f6fc328-0e1c-4e46-8a6d-81460e187c3d","Type":"ContainerStarted","Data":"994c018112638b61dc1eada6ca67aa32bf0d47a18a789a81f3d9ee9385603052"}
Jul 27 23:25:12.523645 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:12.523612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn" event={"ID":"2f6fc328-0e1c-4e46-8a6d-81460e187c3d","Type":"ContainerStarted","Data":"057d804afbffd515bbaa64075f91d6139c66249fe690250b0e60d4452772dd06"}
Jul 27 23:25:13.529021 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:13.528991    2559 generic.go:356] "Generic (PLEG): container finished" podID="2f6fc328-0e1c-4e46-8a6d-81460e187c3d" containerID="057d804afbffd515bbaa64075f91d6139c66249fe690250b0e60d4452772dd06" exitCode=0
Jul 27 23:25:13.529432 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:13.529027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn" event={"ID":"2f6fc328-0e1c-4e46-8a6d-81460e187c3d","Type":"ContainerDied","Data":"057d804afbffd515bbaa64075f91d6139c66249fe690250b0e60d4452772dd06"}
Jul 27 23:25:14.535657 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:14.535596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn" event={"ID":"2f6fc328-0e1c-4e46-8a6d-81460e187c3d","Type":"ContainerStarted","Data":"97fee90cde85ef95a280cf79f9fe15bb83362b6baad93bb8cea109dcbae72ec3"}
Jul 27 23:25:14.536056 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:14.535719    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:14.554829 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:14.554784    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn" podStartSLOduration=1.255388997 podStartE2EDuration="5.554770671s" totalImagesPullingTime="3.230848006s" totalInitContainerRuntime="1.068533668s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:25:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:25:14.554629225 +0000 UTC m=+537.498859970" watchObservedRunningTime="2026-07-27 23:25:14.554770671 +0000 UTC m=+537.499001416"
Jul 27 23:25:25.541184 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:25.541153    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-hb2hn"
Jul 27 23:25:28.480519 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.480482    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5946b997db-8t42c"]
Jul 27 23:25:28.491781 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.491755    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c"
Jul 27 23:25:28.493992 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.493972    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 23:25:28.494303 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.494288    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 23:25:28.494452 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.494334    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 23:25:28.494819 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.494795    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 23:25:28.494933 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.494836    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-npzxn\""
Jul 27 23:25:28.499306 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.499284    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5946b997db-8t42c"]
Jul 27 23:25:28.635235 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.635197    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69d6b5c0-df07-4c77-97c8-8a5cb5f70520-tmp\") pod \"kube-auth-proxy-5946b997db-8t42c\" (UID: \"69d6b5c0-df07-4c77-97c8-8a5cb5f70520\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c"
Jul 27 23:25:28.635235 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.635226    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm68v\" (UniqueName: \"kubernetes.io/projected/69d6b5c0-df07-4c77-97c8-8a5cb5f70520-kube-api-access-dm68v\") pod \"kube-auth-proxy-5946b997db-8t42c\" (UID: \"69d6b5c0-df07-4c77-97c8-8a5cb5f70520\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c"
Jul 27 23:25:28.635506 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.635252    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d6b5c0-df07-4c77-97c8-8a5cb5f70520-tls-certs\") pod \"kube-auth-proxy-5946b997db-8t42c\" (UID: \"69d6b5c0-df07-4c77-97c8-8a5cb5f70520\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c"
Jul 27 23:25:28.717256 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.717223    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kdl5f"]
Jul 27 23:25:28.719608 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.719587    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kdl5f"
Jul 27 23:25:28.721901 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.721878    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-b4t5p\""
Jul 27 23:25:28.722341 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.722320    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 27 23:25:28.731842 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.731777    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kdl5f"]
Jul 27 23:25:28.735891 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.735869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d6b5c0-df07-4c77-97c8-8a5cb5f70520-tls-certs\") pod \"kube-auth-proxy-5946b997db-8t42c\" (UID: \"69d6b5c0-df07-4c77-97c8-8a5cb5f70520\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c"
Jul 27 23:25:28.736005 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.735973    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69d6b5c0-df07-4c77-97c8-8a5cb5f70520-tmp\") pod \"kube-auth-proxy-5946b997db-8t42c\" (UID: \"69d6b5c0-df07-4c77-97c8-8a5cb5f70520\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c"
Jul 27 23:25:28.736005 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:25:28.735996    2559 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 27 23:25:28.736005 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.736001    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm68v\" (UniqueName: \"kubernetes.io/projected/69d6b5c0-df07-4c77-97c8-8a5cb5f70520-kube-api-access-dm68v\") pod \"kube-auth-proxy-5946b997db-8t42c\" (UID: \"69d6b5c0-df07-4c77-97c8-8a5cb5f70520\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c"
Jul 27 23:25:28.736541 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:25:28.736061    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69d6b5c0-df07-4c77-97c8-8a5cb5f70520-tls-certs podName:69d6b5c0-df07-4c77-97c8-8a5cb5f70520 nodeName:}" failed. No retries permitted until 2026-07-27 23:25:29.236039101 +0000 UTC m=+552.180269825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/69d6b5c0-df07-4c77-97c8-8a5cb5f70520-tls-certs") pod "kube-auth-proxy-5946b997db-8t42c" (UID: "69d6b5c0-df07-4c77-97c8-8a5cb5f70520") : secret "kube-auth-proxy-tls" not found
Jul 27 23:25:28.738421 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.738401    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69d6b5c0-df07-4c77-97c8-8a5cb5f70520-tmp\") pod \"kube-auth-proxy-5946b997db-8t42c\" (UID: \"69d6b5c0-df07-4c77-97c8-8a5cb5f70520\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c"
Jul 27 23:25:28.750840 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.750810    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm68v\" (UniqueName: \"kubernetes.io/projected/69d6b5c0-df07-4c77-97c8-8a5cb5f70520-kube-api-access-dm68v\") pod \"kube-auth-proxy-5946b997db-8t42c\" (UID: \"69d6b5c0-df07-4c77-97c8-8a5cb5f70520\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c"
Jul 27 23:25:28.837155 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.837104    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95464\" (UniqueName: \"kubernetes.io/projected/701ecf62-8af6-466f-b84b-f29df06b55fb-kube-api-access-95464\") pod \"odh-model-controller-7998c67dc4-kdl5f\" (UID: \"701ecf62-8af6-466f-b84b-f29df06b55fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-kdl5f"
Jul 27 23:25:28.837319 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.837190    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/701ecf62-8af6-466f-b84b-f29df06b55fb-cert\") pod \"odh-model-controller-7998c67dc4-kdl5f\" (UID: \"701ecf62-8af6-466f-b84b-f29df06b55fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-kdl5f"
Jul 27 23:25:28.938164 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.938131    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/701ecf62-8af6-466f-b84b-f29df06b55fb-cert\") pod \"odh-model-controller-7998c67dc4-kdl5f\" (UID: \"701ecf62-8af6-466f-b84b-f29df06b55fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-kdl5f"
Jul 27 23:25:28.938316 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.938224    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95464\" (UniqueName: \"kubernetes.io/projected/701ecf62-8af6-466f-b84b-f29df06b55fb-kube-api-access-95464\") pod \"odh-model-controller-7998c67dc4-kdl5f\" (UID: \"701ecf62-8af6-466f-b84b-f29df06b55fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-kdl5f"
Jul 27 23:25:28.938316 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:25:28.938281    2559 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 27 23:25:28.938419 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:25:28.938344    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/701ecf62-8af6-466f-b84b-f29df06b55fb-cert podName:701ecf62-8af6-466f-b84b-f29df06b55fb nodeName:}" failed. No retries permitted until 2026-07-27 23:25:29.438328006 +0000 UTC m=+552.382558731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/701ecf62-8af6-466f-b84b-f29df06b55fb-cert") pod "odh-model-controller-7998c67dc4-kdl5f" (UID: "701ecf62-8af6-466f-b84b-f29df06b55fb") : secret "odh-model-controller-webhook-cert" not found
Jul 27 23:25:28.946577 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:28.946554    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95464\" (UniqueName: \"kubernetes.io/projected/701ecf62-8af6-466f-b84b-f29df06b55fb-kube-api-access-95464\") pod \"odh-model-controller-7998c67dc4-kdl5f\" (UID: \"701ecf62-8af6-466f-b84b-f29df06b55fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-kdl5f"
Jul 27 23:25:29.240299 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.240269    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d6b5c0-df07-4c77-97c8-8a5cb5f70520-tls-certs\") pod \"kube-auth-proxy-5946b997db-8t42c\" (UID: \"69d6b5c0-df07-4c77-97c8-8a5cb5f70520\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c"
Jul 27 23:25:29.242620 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.242598    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d6b5c0-df07-4c77-97c8-8a5cb5f70520-tls-certs\") pod \"kube-auth-proxy-5946b997db-8t42c\" (UID: \"69d6b5c0-df07-4c77-97c8-8a5cb5f70520\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c"
Jul 27 23:25:29.404643 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.404607    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c"
Jul 27 23:25:29.442775 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.442747    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/701ecf62-8af6-466f-b84b-f29df06b55fb-cert\") pod \"odh-model-controller-7998c67dc4-kdl5f\" (UID: \"701ecf62-8af6-466f-b84b-f29df06b55fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-kdl5f"
Jul 27 23:25:29.445598 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.445578    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/701ecf62-8af6-466f-b84b-f29df06b55fb-cert\") pod \"odh-model-controller-7998c67dc4-kdl5f\" (UID: \"701ecf62-8af6-466f-b84b-f29df06b55fb\") " pod="opendatahub/odh-model-controller-7998c67dc4-kdl5f"
Jul 27 23:25:29.541690 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.541662    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dwjk9"]
Jul 27 23:25:29.544488 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.544473    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dwjk9"
Jul 27 23:25:29.547733 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.547716    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 27 23:25:29.550078 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.550056    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5946b997db-8t42c"]
Jul 27 23:25:29.554329 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:25:29.554295    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d6b5c0_df07_4c77_97c8_8a5cb5f70520.slice/crio-2043651422e902d31e5c295816be73da7d099137b4e2880eb05034cba3d56c48 WatchSource:0}: Error finding container 2043651422e902d31e5c295816be73da7d099137b4e2880eb05034cba3d56c48: Status 404 returned error can't find the container with id 2043651422e902d31e5c295816be73da7d099137b4e2880eb05034cba3d56c48
Jul 27 23:25:29.556318 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.556298    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dwjk9"]
Jul 27 23:25:29.595122 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.595095    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c" event={"ID":"69d6b5c0-df07-4c77-97c8-8a5cb5f70520","Type":"ContainerStarted","Data":"2043651422e902d31e5c295816be73da7d099137b4e2880eb05034cba3d56c48"}
Jul 27 23:25:29.632204 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.632169    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kdl5f"
Jul 27 23:25:29.644263 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.644243    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8j2k\" (UniqueName: \"kubernetes.io/projected/1ed194b2-4778-4b8e-b542-0e503143d00d-kube-api-access-v8j2k\") pod \"model-serving-api-54d89bd5fc-dwjk9\" (UID: \"1ed194b2-4778-4b8e-b542-0e503143d00d\") " pod="opendatahub/model-serving-api-54d89bd5fc-dwjk9"
Jul 27 23:25:29.644344 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.644293    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed194b2-4778-4b8e-b542-0e503143d00d-tls-certs\") pod \"model-serving-api-54d89bd5fc-dwjk9\" (UID: \"1ed194b2-4778-4b8e-b542-0e503143d00d\") " pod="opendatahub/model-serving-api-54d89bd5fc-dwjk9"
Jul 27 23:25:29.722886 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.722534    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"]
Jul 27 23:25:29.725402 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.725361    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.728162 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.727881    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 23:25:29.728162 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.727886    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 27 23:25:29.728162 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.727923    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 27 23:25:29.728162 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.727938    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 27 23:25:29.728162 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.727889    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9ljqj\""
Jul 27 23:25:29.737930 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.737296    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"]
Jul 27 23:25:29.744739 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.744718    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed194b2-4778-4b8e-b542-0e503143d00d-tls-certs\") pod \"model-serving-api-54d89bd5fc-dwjk9\" (UID: \"1ed194b2-4778-4b8e-b542-0e503143d00d\") " pod="opendatahub/model-serving-api-54d89bd5fc-dwjk9"
Jul 27 23:25:29.744866 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.744780    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8j2k\" (UniqueName: \"kubernetes.io/projected/1ed194b2-4778-4b8e-b542-0e503143d00d-kube-api-access-v8j2k\") pod \"model-serving-api-54d89bd5fc-dwjk9\" (UID: \"1ed194b2-4778-4b8e-b542-0e503143d00d\") " pod="opendatahub/model-serving-api-54d89bd5fc-dwjk9"
Jul 27 23:25:29.744927 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:25:29.744880    2559 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 27 23:25:29.744988 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:25:29.744950    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ed194b2-4778-4b8e-b542-0e503143d00d-tls-certs podName:1ed194b2-4778-4b8e-b542-0e503143d00d nodeName:}" failed. No retries permitted until 2026-07-27 23:25:30.244930047 +0000 UTC m=+553.189160785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1ed194b2-4778-4b8e-b542-0e503143d00d-tls-certs") pod "model-serving-api-54d89bd5fc-dwjk9" (UID: "1ed194b2-4778-4b8e-b542-0e503143d00d") : secret "model-serving-api-tls" not found
Jul 27 23:25:29.756454 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.756430    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8j2k\" (UniqueName: \"kubernetes.io/projected/1ed194b2-4778-4b8e-b542-0e503143d00d-kube-api-access-v8j2k\") pod \"model-serving-api-54d89bd5fc-dwjk9\" (UID: \"1ed194b2-4778-4b8e-b542-0e503143d00d\") " pod="opendatahub/model-serving-api-54d89bd5fc-dwjk9"
Jul 27 23:25:29.772311 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.772292    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kdl5f"]
Jul 27 23:25:29.774843 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:25:29.774823    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701ecf62_8af6_466f_b84b_f29df06b55fb.slice/crio-071b029ee1614df0cc62e7ff160d8be0c6d81c02f6dd53e299aa2666b430a31a WatchSource:0}: Error finding container 071b029ee1614df0cc62e7ff160d8be0c6d81c02f6dd53e299aa2666b430a31a: Status 404 returned error can't find the container with id 071b029ee1614df0cc62e7ff160d8be0c6d81c02f6dd53e299aa2666b430a31a
Jul 27 23:25:29.845430 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.845334    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/720a7f43-4d69-4c20-b7ff-91de860191b5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.845430 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.845363    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.845430 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.845417    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk8px\" (UniqueName: \"kubernetes.io/projected/720a7f43-4d69-4c20-b7ff-91de860191b5-kube-api-access-tk8px\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.845632 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.845436    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.845632 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.845463    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.845632 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.845480    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.845632 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.845567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.946434 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.946401    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.946607 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.946481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/720a7f43-4d69-4c20-b7ff-91de860191b5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.946607 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.946508    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.946607 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.946547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk8px\" (UniqueName: \"kubernetes.io/projected/720a7f43-4d69-4c20-b7ff-91de860191b5-kube-api-access-tk8px\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.946607 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.946575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.946887 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.946615    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.946887 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.946641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.947099 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.947071    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.949016 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.948983    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/720a7f43-4d69-4c20-b7ff-91de860191b5-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.949123 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.949025    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.949311 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.949297    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.949429 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.949411    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.955841 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.955821    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk8px\" (UniqueName: \"kubernetes.io/projected/720a7f43-4d69-4c20-b7ff-91de860191b5-kube-api-access-tk8px\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:29.956119 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:29.956100    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vt6j4\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:30.038183 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:30.038150    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:30.185821 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:30.185790    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"]
Jul 27 23:25:30.190550 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:25:30.190520    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720a7f43_4d69_4c20_b7ff_91de860191b5.slice/crio-a76809681040e9acbd67248a019f5f032d676f24cc7f9feac83293698675c686 WatchSource:0}: Error finding container a76809681040e9acbd67248a019f5f032d676f24cc7f9feac83293698675c686: Status 404 returned error can't find the container with id a76809681040e9acbd67248a019f5f032d676f24cc7f9feac83293698675c686
Jul 27 23:25:30.252632 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:30.252603    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed194b2-4778-4b8e-b542-0e503143d00d-tls-certs\") pod \"model-serving-api-54d89bd5fc-dwjk9\" (UID: \"1ed194b2-4778-4b8e-b542-0e503143d00d\") " pod="opendatahub/model-serving-api-54d89bd5fc-dwjk9"
Jul 27 23:25:30.255324 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:30.255300    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed194b2-4778-4b8e-b542-0e503143d00d-tls-certs\") pod \"model-serving-api-54d89bd5fc-dwjk9\" (UID: \"1ed194b2-4778-4b8e-b542-0e503143d00d\") " pod="opendatahub/model-serving-api-54d89bd5fc-dwjk9"
Jul 27 23:25:30.454874 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:30.454598    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dwjk9"
Jul 27 23:25:30.605298 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:30.605186    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kdl5f" event={"ID":"701ecf62-8af6-466f-b84b-f29df06b55fb","Type":"ContainerStarted","Data":"071b029ee1614df0cc62e7ff160d8be0c6d81c02f6dd53e299aa2666b430a31a"}
Jul 27 23:25:30.607641 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:30.607570    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4" event={"ID":"720a7f43-4d69-4c20-b7ff-91de860191b5","Type":"ContainerStarted","Data":"a76809681040e9acbd67248a019f5f032d676f24cc7f9feac83293698675c686"}
Jul 27 23:25:30.634338 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:30.634302    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dwjk9"]
Jul 27 23:25:30.637727 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:25:30.637648    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed194b2_4778_4b8e_b542_0e503143d00d.slice/crio-b6a3f3a3f1bb85810d044251ffc10d592fda7544519e22ceb850b2de15586140 WatchSource:0}: Error finding container b6a3f3a3f1bb85810d044251ffc10d592fda7544519e22ceb850b2de15586140: Status 404 returned error can't find the container with id b6a3f3a3f1bb85810d044251ffc10d592fda7544519e22ceb850b2de15586140
Jul 27 23:25:31.615010 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:31.614969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dwjk9" event={"ID":"1ed194b2-4778-4b8e-b542-0e503143d00d","Type":"ContainerStarted","Data":"b6a3f3a3f1bb85810d044251ffc10d592fda7544519e22ceb850b2de15586140"}
Jul 27 23:25:34.891863 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:34.891818    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:25:34.892183 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:34.891917    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:25:35.635735 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:35.635697    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c" event={"ID":"69d6b5c0-df07-4c77-97c8-8a5cb5f70520","Type":"ContainerStarted","Data":"0aa9c8324fb29b4778f6e9da19aff333195e006f2e474ac2087b56b83d127b0d"}
Jul 27 23:25:35.637295 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:35.637263    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kdl5f" event={"ID":"701ecf62-8af6-466f-b84b-f29df06b55fb","Type":"ContainerStarted","Data":"8e90f7f067b479cb7418ba75a182fafbaa7ef5fdeacacfc8ccab6c49569c3fad"}
Jul 27 23:25:35.637449 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:35.637319    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kdl5f"
Jul 27 23:25:35.639061 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:35.639034    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4" event={"ID":"720a7f43-4d69-4c20-b7ff-91de860191b5","Type":"ContainerStarted","Data":"4d246c3060698dd6f0e73be7e92a33abbabd78643d42fd41bc7a79ce87999ac2"}
Jul 27 23:25:35.639201 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:35.639083    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:35.640623 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:35.640599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dwjk9" event={"ID":"1ed194b2-4778-4b8e-b542-0e503143d00d","Type":"ContainerStarted","Data":"966ee113364272481bb88224524eb5b05606b5439dbddd3aa31b9538ee85d4b1"}
Jul 27 23:25:35.640761 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:35.640746    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-dwjk9"
Jul 27 23:25:35.641288 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:35.641267    2559 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-vt6j4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 23:25:35.641397 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:35.641310    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4" podUID="720a7f43-4d69-4c20-b7ff-91de860191b5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 23:25:35.650448 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:35.650411    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5946b997db-8t42c" podStartSLOduration=2.3162531 podStartE2EDuration="7.650399525s" totalImagesPullingTime="5.334146425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:25:35.649459287 +0000 UTC m=+558.593690032" watchObservedRunningTime="2026-07-27 23:25:35.650399525 +0000 UTC m=+558.594630267"
Jul 27 23:25:35.667151 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:35.666238    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4" podStartSLOduration=1.968678276 podStartE2EDuration="6.666220397s" totalImagesPullingTime="4.697542121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:25:35.665976568 +0000 UTC m=+558.610207311" watchObservedRunningTime="2026-07-27 23:25:35.666220397 +0000 UTC m=+558.610451145"
Jul 27 23:25:35.680032 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:35.679991    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-kdl5f" podStartSLOduration=2.551625655 podStartE2EDuration="7.679981446s" totalImagesPullingTime="5.128355791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:25:35.678959938 +0000 UTC m=+558.623190685" watchObservedRunningTime="2026-07-27 23:25:35.679981446 +0000 UTC m=+558.624212190"
Jul 27 23:25:35.694292 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:35.694247    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-dwjk9" podStartSLOduration=2.443133461 podStartE2EDuration="6.694231766s" totalImagesPullingTime="4.251098305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:25:35.692624381 +0000 UTC m=+558.636855130" watchObservedRunningTime="2026-07-27 23:25:35.694231766 +0000 UTC m=+558.638462512"
Jul 27 23:25:36.645637 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:36.645607    2559 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-vt6j4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 23:25:36.646003 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:36.645657    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4" podUID="720a7f43-4d69-4c20-b7ff-91de860191b5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 23:25:37.234949 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.234919    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"]
Jul 27 23:25:37.238249 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.238233    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.240624 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.240607    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mw79r\""
Jul 27 23:25:37.250485 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.250461    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"]
Jul 27 23:25:37.420977 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.420937    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqfkd\" (UniqueName: \"kubernetes.io/projected/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-kube-api-access-kqfkd\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.421171 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.420988    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.421171 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.421107    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.421279 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.421182    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.421279 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.421207    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.421279 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.421239    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.421439 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.421280    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.421439 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.421312    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.421439 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.421390    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.522299 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.522196    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.522299 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.522231    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.522299 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.522249    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.522299 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.522263    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.522654 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.522309    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.522654 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.522343    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.522654 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.522412    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqfkd\" (UniqueName: \"kubernetes.io/projected/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-kube-api-access-kqfkd\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.522654 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.522445    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.522654 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.522512    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.522896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.522768    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.523083 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.523060    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.523150 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.523091    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.523323 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.523208    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.523426 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.523296    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.524873 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.524843    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.525435 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.525408    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.530323 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.530301    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.530828 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.530807    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqfkd\" (UniqueName: \"kubernetes.io/projected/008c5090-63d0-4f0e-9f7a-ef065dfbdb6e-kube-api-access-kqfkd\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps\" (UID: \"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.548691 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.548666    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:37.694241 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:37.694206    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"]
Jul 27 23:25:37.697520 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:25:37.697491    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008c5090_63d0_4f0e_9f7a_ef065dfbdb6e.slice/crio-f8c7c2eeee710acfc465358e68ecf67afde2ef017156b3af0b14cb0839d68acf WatchSource:0}: Error finding container f8c7c2eeee710acfc465358e68ecf67afde2ef017156b3af0b14cb0839d68acf: Status 404 returned error can't find the container with id f8c7c2eeee710acfc465358e68ecf67afde2ef017156b3af0b14cb0839d68acf
Jul 27 23:25:38.654447 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:38.654414    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps" event={"ID":"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e","Type":"ContainerStarted","Data":"f8c7c2eeee710acfc465358e68ecf67afde2ef017156b3af0b14cb0839d68acf"}
Jul 27 23:25:39.563447 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.563421    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-77g9r"]
Jul 27 23:25:39.570334 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.570312    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"]
Jul 27 23:25:39.570489 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.570470    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-77g9r"
Jul 27 23:25:39.572875 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.572853    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 27 23:25:39.573261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.573244    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dxqvx\""
Jul 27 23:25:39.573643 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.573628    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"
Jul 27 23:25:39.575668 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.575646    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 27 23:25:39.575930 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.575912    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-n7486\""
Jul 27 23:25:39.581776 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.581755    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-77g9r"]
Jul 27 23:25:39.604948 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.604924    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"]
Jul 27 23:25:39.646271 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.646247    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:39.740879 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.740834    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/faf1a861-2bf2-4abf-8ac9-be78538a6360-cert\") pod \"kserve-controller-manager-d6f4b65b8-77g9r\" (UID: \"faf1a861-2bf2-4abf-8ac9-be78538a6360\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-77g9r"
Jul 27 23:25:39.740879 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.740880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnqq2\" (UniqueName: \"kubernetes.io/projected/423b58a7-f70f-4da3-8923-f4ac83305728-kube-api-access-gnqq2\") pod \"llmisvc-controller-manager-777c6bd7d5-5f5cm\" (UID: \"423b58a7-f70f-4da3-8923-f4ac83305728\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"
Jul 27 23:25:39.741107 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.740932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj2n4\" (UniqueName: \"kubernetes.io/projected/faf1a861-2bf2-4abf-8ac9-be78538a6360-kube-api-access-rj2n4\") pod \"kserve-controller-manager-d6f4b65b8-77g9r\" (UID: \"faf1a861-2bf2-4abf-8ac9-be78538a6360\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-77g9r"
Jul 27 23:25:39.741107 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.741070    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/423b58a7-f70f-4da3-8923-f4ac83305728-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5f5cm\" (UID: \"423b58a7-f70f-4da3-8923-f4ac83305728\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"
Jul 27 23:25:39.842562 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.842479    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj2n4\" (UniqueName: \"kubernetes.io/projected/faf1a861-2bf2-4abf-8ac9-be78538a6360-kube-api-access-rj2n4\") pod \"kserve-controller-manager-d6f4b65b8-77g9r\" (UID: \"faf1a861-2bf2-4abf-8ac9-be78538a6360\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-77g9r"
Jul 27 23:25:39.842740 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.842695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/423b58a7-f70f-4da3-8923-f4ac83305728-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5f5cm\" (UID: \"423b58a7-f70f-4da3-8923-f4ac83305728\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"
Jul 27 23:25:39.842815 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.842772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/faf1a861-2bf2-4abf-8ac9-be78538a6360-cert\") pod \"kserve-controller-manager-d6f4b65b8-77g9r\" (UID: \"faf1a861-2bf2-4abf-8ac9-be78538a6360\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-77g9r"
Jul 27 23:25:39.842815 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.842806    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnqq2\" (UniqueName: \"kubernetes.io/projected/423b58a7-f70f-4da3-8923-f4ac83305728-kube-api-access-gnqq2\") pod \"llmisvc-controller-manager-777c6bd7d5-5f5cm\" (UID: \"423b58a7-f70f-4da3-8923-f4ac83305728\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"
Jul 27 23:25:39.842919 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:25:39.842872    2559 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 27 23:25:39.842964 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:25:39.842933    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/423b58a7-f70f-4da3-8923-f4ac83305728-cert podName:423b58a7-f70f-4da3-8923-f4ac83305728 nodeName:}" failed. No retries permitted until 2026-07-27 23:25:40.34291316 +0000 UTC m=+563.287143898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/423b58a7-f70f-4da3-8923-f4ac83305728-cert") pod "llmisvc-controller-manager-777c6bd7d5-5f5cm" (UID: "423b58a7-f70f-4da3-8923-f4ac83305728") : secret "llmisvc-webhook-server-cert" not found
Jul 27 23:25:39.843023 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:25:39.842975    2559 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 27 23:25:39.843057 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:25:39.843022    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faf1a861-2bf2-4abf-8ac9-be78538a6360-cert podName:faf1a861-2bf2-4abf-8ac9-be78538a6360 nodeName:}" failed. No retries permitted until 2026-07-27 23:25:40.343008259 +0000 UTC m=+563.287238992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/faf1a861-2bf2-4abf-8ac9-be78538a6360-cert") pod "kserve-controller-manager-d6f4b65b8-77g9r" (UID: "faf1a861-2bf2-4abf-8ac9-be78538a6360") : secret "kserve-webhook-server-cert" not found
Jul 27 23:25:39.852565 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.852544    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnqq2\" (UniqueName: \"kubernetes.io/projected/423b58a7-f70f-4da3-8923-f4ac83305728-kube-api-access-gnqq2\") pod \"llmisvc-controller-manager-777c6bd7d5-5f5cm\" (UID: \"423b58a7-f70f-4da3-8923-f4ac83305728\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"
Jul 27 23:25:39.852694 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:39.852604    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj2n4\" (UniqueName: \"kubernetes.io/projected/faf1a861-2bf2-4abf-8ac9-be78538a6360-kube-api-access-rj2n4\") pod \"kserve-controller-manager-d6f4b65b8-77g9r\" (UID: \"faf1a861-2bf2-4abf-8ac9-be78538a6360\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-77g9r"
Jul 27 23:25:40.347747 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:40.347714    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/423b58a7-f70f-4da3-8923-f4ac83305728-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5f5cm\" (UID: \"423b58a7-f70f-4da3-8923-f4ac83305728\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"
Jul 27 23:25:40.347747 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:40.347754    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/faf1a861-2bf2-4abf-8ac9-be78538a6360-cert\") pod \"kserve-controller-manager-d6f4b65b8-77g9r\" (UID: \"faf1a861-2bf2-4abf-8ac9-be78538a6360\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-77g9r"
Jul 27 23:25:40.347992 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:25:40.347880    2559 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 27 23:25:40.347992 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:25:40.347966    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/423b58a7-f70f-4da3-8923-f4ac83305728-cert podName:423b58a7-f70f-4da3-8923-f4ac83305728 nodeName:}" failed. No retries permitted until 2026-07-27 23:25:41.347944204 +0000 UTC m=+564.292174941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/423b58a7-f70f-4da3-8923-f4ac83305728-cert") pod "llmisvc-controller-manager-777c6bd7d5-5f5cm" (UID: "423b58a7-f70f-4da3-8923-f4ac83305728") : secret "llmisvc-webhook-server-cert" not found
Jul 27 23:25:40.350123 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:40.350100    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/faf1a861-2bf2-4abf-8ac9-be78538a6360-cert\") pod \"kserve-controller-manager-d6f4b65b8-77g9r\" (UID: \"faf1a861-2bf2-4abf-8ac9-be78538a6360\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-77g9r"
Jul 27 23:25:40.373681 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:40.373658    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:25:40.373739 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:40.373732    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:25:40.373769 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:40.373757    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:25:40.486749 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:40.486720    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-77g9r"
Jul 27 23:25:40.640794 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:40.640764    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-77g9r"]
Jul 27 23:25:40.642780 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:25:40.642755    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf1a861_2bf2_4abf_8ac9_be78538a6360.slice/crio-769c451e48aab14ddfd46e0c3ed2148672329968f1d7f2f472efd300c2463c60 WatchSource:0}: Error finding container 769c451e48aab14ddfd46e0c3ed2148672329968f1d7f2f472efd300c2463c60: Status 404 returned error can't find the container with id 769c451e48aab14ddfd46e0c3ed2148672329968f1d7f2f472efd300c2463c60
Jul 27 23:25:40.665523 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:40.665475    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps" event={"ID":"008c5090-63d0-4f0e-9f7a-ef065dfbdb6e","Type":"ContainerStarted","Data":"50e59a5cdcefa4ec9f12ef695f9074f7e0c8332ac6cabc8c1af534de58c6fc36"}
Jul 27 23:25:40.666848 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:40.666826    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-77g9r" event={"ID":"faf1a861-2bf2-4abf-8ac9-be78538a6360","Type":"ContainerStarted","Data":"769c451e48aab14ddfd46e0c3ed2148672329968f1d7f2f472efd300c2463c60"}
Jul 27 23:25:40.685691 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:40.685653    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps" podStartSLOduration=1.01183578 podStartE2EDuration="3.685639117s" totalImagesPullingTime="2.673803337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:25:40.683648685 +0000 UTC m=+563.627879442" watchObservedRunningTime="2026-07-27 23:25:40.685639117 +0000 UTC m=+563.629869935"
Jul 27 23:25:41.357430 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.357391    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/423b58a7-f70f-4da3-8923-f4ac83305728-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5f5cm\" (UID: \"423b58a7-f70f-4da3-8923-f4ac83305728\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"
Jul 27 23:25:41.360322 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.360295    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/423b58a7-f70f-4da3-8923-f4ac83305728-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5f5cm\" (UID: \"423b58a7-f70f-4da3-8923-f4ac83305728\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"
Jul 27 23:25:41.393327 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.393290    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"
Jul 27 23:25:41.549803 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.549763    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:41.555088 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.555063    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:41.570644 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.570611    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"]
Jul 27 23:25:41.574540 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:25:41.574508    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod423b58a7_f70f_4da3_8923_f4ac83305728.slice/crio-4b970a2d282b2b6bfd128746a2ed4781e59e44798fa0baffc108385b070c9cb2 WatchSource:0}: Error finding container 4b970a2d282b2b6bfd128746a2ed4781e59e44798fa0baffc108385b070c9cb2: Status 404 returned error can't find the container with id 4b970a2d282b2b6bfd128746a2ed4781e59e44798fa0baffc108385b070c9cb2
Jul 27 23:25:41.676621 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.676533    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm" event={"ID":"423b58a7-f70f-4da3-8923-f4ac83305728","Type":"ContainerStarted","Data":"4b970a2d282b2b6bfd128746a2ed4781e59e44798fa0baffc108385b070c9cb2"}
Jul 27 23:25:41.677084 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.676754    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:41.677946 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.677919    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps"
Jul 27 23:25:41.681425 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.681400    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"]
Jul 27 23:25:41.681618 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.681583    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4" podUID="720a7f43-4d69-4c20-b7ff-91de860191b5" containerName="discovery" containerID="cri-o://4d246c3060698dd6f0e73be7e92a33abbabd78643d42fd41bc7a79ce87999ac2" gracePeriod=30
Jul 27 23:25:41.940481 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.940448    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:41.961218 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.961187    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-csr-dns-cert\") pod \"720a7f43-4d69-4c20-b7ff-91de860191b5\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") "
Jul 27 23:25:41.961345 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.961239    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk8px\" (UniqueName: \"kubernetes.io/projected/720a7f43-4d69-4c20-b7ff-91de860191b5-kube-api-access-tk8px\") pod \"720a7f43-4d69-4c20-b7ff-91de860191b5\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") "
Jul 27 23:25:41.961345 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.961269    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/720a7f43-4d69-4c20-b7ff-91de860191b5-local-certs\") pod \"720a7f43-4d69-4c20-b7ff-91de860191b5\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") "
Jul 27 23:25:41.961345 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.961311    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-kubeconfig\") pod \"720a7f43-4d69-4c20-b7ff-91de860191b5\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") "
Jul 27 23:25:41.961531 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.961350    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-cacerts\") pod \"720a7f43-4d69-4c20-b7ff-91de860191b5\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") "
Jul 27 23:25:41.961531 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.961411    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-token\") pod \"720a7f43-4d69-4c20-b7ff-91de860191b5\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") "
Jul 27 23:25:41.961531 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.961442    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-csr-ca-configmap\") pod \"720a7f43-4d69-4c20-b7ff-91de860191b5\" (UID: \"720a7f43-4d69-4c20-b7ff-91de860191b5\") "
Jul 27 23:25:41.962438 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.962409    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "720a7f43-4d69-4c20-b7ff-91de860191b5" (UID: "720a7f43-4d69-4c20-b7ff-91de860191b5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:25:41.965286 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.965232    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "720a7f43-4d69-4c20-b7ff-91de860191b5" (UID: "720a7f43-4d69-4c20-b7ff-91de860191b5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:25:41.965512 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.965483    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "720a7f43-4d69-4c20-b7ff-91de860191b5" (UID: "720a7f43-4d69-4c20-b7ff-91de860191b5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:25:41.965512 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.965491    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720a7f43-4d69-4c20-b7ff-91de860191b5-local-certs" (OuterVolumeSpecName: "local-certs") pod "720a7f43-4d69-4c20-b7ff-91de860191b5" (UID: "720a7f43-4d69-4c20-b7ff-91de860191b5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:25:41.965512 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.965504    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-token" (OuterVolumeSpecName: "istio-token") pod "720a7f43-4d69-4c20-b7ff-91de860191b5" (UID: "720a7f43-4d69-4c20-b7ff-91de860191b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:25:41.965805 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.965776    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720a7f43-4d69-4c20-b7ff-91de860191b5-kube-api-access-tk8px" (OuterVolumeSpecName: "kube-api-access-tk8px") pod "720a7f43-4d69-4c20-b7ff-91de860191b5" (UID: "720a7f43-4d69-4c20-b7ff-91de860191b5"). InnerVolumeSpecName "kube-api-access-tk8px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:25:41.966227 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:41.966208    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-cacerts" (OuterVolumeSpecName: "cacerts") pod "720a7f43-4d69-4c20-b7ff-91de860191b5" (UID: "720a7f43-4d69-4c20-b7ff-91de860191b5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:25:42.062512 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.062478    2559 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-kubeconfig\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:25:42.062512 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.062508    2559 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-cacerts\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:25:42.062512 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.062517    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-token\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:25:42.062965 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.062525    2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-csr-ca-configmap\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:25:42.062965 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.062534    2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/720a7f43-4d69-4c20-b7ff-91de860191b5-istio-csr-dns-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:25:42.062965 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.062542    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk8px\" (UniqueName: \"kubernetes.io/projected/720a7f43-4d69-4c20-b7ff-91de860191b5-kube-api-access-tk8px\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:25:42.062965 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.062553    2559 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/720a7f43-4d69-4c20-b7ff-91de860191b5-local-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:25:42.684665 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.684630    2559 generic.go:356] "Generic (PLEG): container finished" podID="720a7f43-4d69-4c20-b7ff-91de860191b5" containerID="4d246c3060698dd6f0e73be7e92a33abbabd78643d42fd41bc7a79ce87999ac2" exitCode=0
Jul 27 23:25:42.685272 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.685256    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"
Jul 27 23:25:42.685438 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.685410    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4" event={"ID":"720a7f43-4d69-4c20-b7ff-91de860191b5","Type":"ContainerDied","Data":"4d246c3060698dd6f0e73be7e92a33abbabd78643d42fd41bc7a79ce87999ac2"}
Jul 27 23:25:42.685541 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.685462    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4" event={"ID":"720a7f43-4d69-4c20-b7ff-91de860191b5","Type":"ContainerDied","Data":"a76809681040e9acbd67248a019f5f032d676f24cc7f9feac83293698675c686"}
Jul 27 23:25:42.685541 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.685492    2559 scope.go:117] "RemoveContainer" containerID="4d246c3060698dd6f0e73be7e92a33abbabd78643d42fd41bc7a79ce87999ac2"
Jul 27 23:25:42.709831 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.709809    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"]
Jul 27 23:25:42.713654 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.713630    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vt6j4"]
Jul 27 23:25:42.760120 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.760094    2559 scope.go:117] "RemoveContainer" containerID="4d246c3060698dd6f0e73be7e92a33abbabd78643d42fd41bc7a79ce87999ac2"
Jul 27 23:25:42.760526 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:25:42.760504    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d246c3060698dd6f0e73be7e92a33abbabd78643d42fd41bc7a79ce87999ac2\": container with ID starting with 4d246c3060698dd6f0e73be7e92a33abbabd78643d42fd41bc7a79ce87999ac2 not found: ID does not exist" containerID="4d246c3060698dd6f0e73be7e92a33abbabd78643d42fd41bc7a79ce87999ac2"
Jul 27 23:25:42.760623 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:42.760532    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d246c3060698dd6f0e73be7e92a33abbabd78643d42fd41bc7a79ce87999ac2"} err="failed to get container status \"4d246c3060698dd6f0e73be7e92a33abbabd78643d42fd41bc7a79ce87999ac2\": rpc error: code = NotFound desc = could not find container \"4d246c3060698dd6f0e73be7e92a33abbabd78643d42fd41bc7a79ce87999ac2\": container with ID starting with 4d246c3060698dd6f0e73be7e92a33abbabd78643d42fd41bc7a79ce87999ac2 not found: ID does not exist"
Jul 27 23:25:43.564163 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:43.564135    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="720a7f43-4d69-4c20-b7ff-91de860191b5" path="/var/lib/kubelet/pods/720a7f43-4d69-4c20-b7ff-91de860191b5/volumes"
Jul 27 23:25:43.692261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:43.692230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-77g9r" event={"ID":"faf1a861-2bf2-4abf-8ac9-be78538a6360","Type":"ContainerStarted","Data":"bf4e4ef8fe88ea8d90e72274d0fbb0d846914aae70982f58f2e2a2e40c4a55b1"}
Jul 27 23:25:43.692711 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:43.692316    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-77g9r"
Jul 27 23:25:43.709156 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:43.709101    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-77g9r" podStartSLOduration=1.8382409690000001 podStartE2EDuration="4.70908628s" totalImagesPullingTime="2.870845311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:25:43.707014554 +0000 UTC m=+566.651245299" watchObservedRunningTime="2026-07-27 23:25:43.70908628 +0000 UTC m=+566.653317025"
Jul 27 23:25:44.698526 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:44.698489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm" event={"ID":"423b58a7-f70f-4da3-8923-f4ac83305728","Type":"ContainerStarted","Data":"2c2832319677fa9e146d0f7688843ea1839f13233ba5fb1db3e9190b9a1e42d1"}
Jul 27 23:25:44.698956 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:44.698621    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"
Jul 27 23:25:44.716679 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:44.716632    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm" podStartSLOduration=3.013555445 podStartE2EDuration="5.716614338s" totalImagesPullingTime="2.703058893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:25:44.71410185 +0000 UTC m=+567.658332598" watchObservedRunningTime="2026-07-27 23:25:44.716614338 +0000 UTC m=+567.660845084"
Jul 27 23:25:46.649664 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:46.649630    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-dwjk9"
Jul 27 23:25:46.686846 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:25:46.686815    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-kdl5f"
Jul 27 23:26:14.703961 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:14.703931    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-77g9r"
Jul 27 23:26:15.706113 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:15.706082    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5f5cm"
Jul 27 23:26:17.457701 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:17.457682    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:26:17.462875 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:17.462859    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:26:29.593722 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:29.593690    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hf762"]
Jul 27 23:26:29.594305 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:29.594002    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="720a7f43-4d69-4c20-b7ff-91de860191b5" containerName="discovery"
Jul 27 23:26:29.594305 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:29.594013    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="720a7f43-4d69-4c20-b7ff-91de860191b5" containerName="discovery"
Jul 27 23:26:29.594305 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:29.594067    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="720a7f43-4d69-4c20-b7ff-91de860191b5" containerName="discovery"
Jul 27 23:26:29.596493 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:29.596475    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hf762"
Jul 27 23:26:29.598695 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:29.598671    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 23:26:29.598805 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:29.598671    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-655t8\""
Jul 27 23:26:29.599244 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:29.599227    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 23:26:29.605731 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:29.605704    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hf762"]
Jul 27 23:26:29.674307 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:29.674270    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqcck\" (UniqueName: \"kubernetes.io/projected/597c89b7-6ab1-4bbf-ab24-e910095bae5e-kube-api-access-xqcck\") pod \"kuadrant-operator-catalog-hf762\" (UID: \"597c89b7-6ab1-4bbf-ab24-e910095bae5e\") " pod="kuadrant-system/kuadrant-operator-catalog-hf762"
Jul 27 23:26:29.775243 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:29.775201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqcck\" (UniqueName: \"kubernetes.io/projected/597c89b7-6ab1-4bbf-ab24-e910095bae5e-kube-api-access-xqcck\") pod \"kuadrant-operator-catalog-hf762\" (UID: \"597c89b7-6ab1-4bbf-ab24-e910095bae5e\") " pod="kuadrant-system/kuadrant-operator-catalog-hf762"
Jul 27 23:26:29.790939 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:29.790911    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqcck\" (UniqueName: \"kubernetes.io/projected/597c89b7-6ab1-4bbf-ab24-e910095bae5e-kube-api-access-xqcck\") pod \"kuadrant-operator-catalog-hf762\" (UID: \"597c89b7-6ab1-4bbf-ab24-e910095bae5e\") " pod="kuadrant-system/kuadrant-operator-catalog-hf762"
Jul 27 23:26:29.907914 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:29.907756    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hf762"
Jul 27 23:26:29.963864 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:29.963829    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hf762"]
Jul 27 23:26:30.035779 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:30.035751    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hf762"]
Jul 27 23:26:30.038998 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:26:30.038963    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597c89b7_6ab1_4bbf_ab24_e910095bae5e.slice/crio-9f1c269182753d8b1b977dbdd29e56f43fd8b6af3e7e426eef36e8c1575649ff WatchSource:0}: Error finding container 9f1c269182753d8b1b977dbdd29e56f43fd8b6af3e7e426eef36e8c1575649ff: Status 404 returned error can't find the container with id 9f1c269182753d8b1b977dbdd29e56f43fd8b6af3e7e426eef36e8c1575649ff
Jul 27 23:26:30.170217 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:30.170137    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5fksn"]
Jul 27 23:26:30.172647 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:30.172631    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5fksn"
Jul 27 23:26:30.178503 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:30.178477    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq9fc\" (UniqueName: \"kubernetes.io/projected/2468e0bf-d810-4b49-9ea8-c48e2d0f9ed0-kube-api-access-nq9fc\") pod \"kuadrant-operator-catalog-5fksn\" (UID: \"2468e0bf-d810-4b49-9ea8-c48e2d0f9ed0\") " pod="kuadrant-system/kuadrant-operator-catalog-5fksn"
Jul 27 23:26:30.180247 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:30.180200    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5fksn"]
Jul 27 23:26:30.279002 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:30.278969    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq9fc\" (UniqueName: \"kubernetes.io/projected/2468e0bf-d810-4b49-9ea8-c48e2d0f9ed0-kube-api-access-nq9fc\") pod \"kuadrant-operator-catalog-5fksn\" (UID: \"2468e0bf-d810-4b49-9ea8-c48e2d0f9ed0\") " pod="kuadrant-system/kuadrant-operator-catalog-5fksn"
Jul 27 23:26:30.290431 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:30.290358    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq9fc\" (UniqueName: \"kubernetes.io/projected/2468e0bf-d810-4b49-9ea8-c48e2d0f9ed0-kube-api-access-nq9fc\") pod \"kuadrant-operator-catalog-5fksn\" (UID: \"2468e0bf-d810-4b49-9ea8-c48e2d0f9ed0\") " pod="kuadrant-system/kuadrant-operator-catalog-5fksn"
Jul 27 23:26:30.482940 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:30.482849    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5fksn"
Jul 27 23:26:30.615219 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:30.614884    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5fksn"]
Jul 27 23:26:30.657910 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:26:30.657870    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2468e0bf_d810_4b49_9ea8_c48e2d0f9ed0.slice/crio-695b9fcd4ac5d89d572665eee98cb2f54a1d79025b3d1bf205df2ae6203c1bc6 WatchSource:0}: Error finding container 695b9fcd4ac5d89d572665eee98cb2f54a1d79025b3d1bf205df2ae6203c1bc6: Status 404 returned error can't find the container with id 695b9fcd4ac5d89d572665eee98cb2f54a1d79025b3d1bf205df2ae6203c1bc6
Jul 27 23:26:30.912431 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:30.912401    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5fksn" event={"ID":"2468e0bf-d810-4b49-9ea8-c48e2d0f9ed0","Type":"ContainerStarted","Data":"695b9fcd4ac5d89d572665eee98cb2f54a1d79025b3d1bf205df2ae6203c1bc6"}
Jul 27 23:26:30.913633 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:30.913603    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hf762" event={"ID":"597c89b7-6ab1-4bbf-ab24-e910095bae5e","Type":"ContainerStarted","Data":"9f1c269182753d8b1b977dbdd29e56f43fd8b6af3e7e426eef36e8c1575649ff"}
Jul 27 23:26:32.925267 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:32.925230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5fksn" event={"ID":"2468e0bf-d810-4b49-9ea8-c48e2d0f9ed0","Type":"ContainerStarted","Data":"7e11ff4e8156acad5464117d465714411d2bec9ab1a91cc543e88ed0f4027d9a"}
Jul 27 23:26:32.926823 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:32.926795    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hf762" event={"ID":"597c89b7-6ab1-4bbf-ab24-e910095bae5e","Type":"ContainerStarted","Data":"a50909160fe5276ba04bec69da1c6cfa86df68cff75886e93c6745d4b7cb143a"}
Jul 27 23:26:32.926937 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:32.926871    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-hf762" podUID="597c89b7-6ab1-4bbf-ab24-e910095bae5e" containerName="registry-server" containerID="cri-o://a50909160fe5276ba04bec69da1c6cfa86df68cff75886e93c6745d4b7cb143a" gracePeriod=2
Jul 27 23:26:32.940336 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:32.940150    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-5fksn" podStartSLOduration=1.580680123 podStartE2EDuration="2.940134862s" totalImagesPullingTime="1.359454739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:26:32.938785138 +0000 UTC m=+615.883015897" watchObservedRunningTime="2026-07-27 23:26:32.940134862 +0000 UTC m=+615.884365609"
Jul 27 23:26:32.952339 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:32.952298    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-hf762" podStartSLOduration=2.03659867 podStartE2EDuration="3.952282116s" totalImagesPullingTime="1.915683446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:26:32.951502474 +0000 UTC m=+615.895733221" watchObservedRunningTime="2026-07-27 23:26:32.952282116 +0000 UTC m=+615.896512862"
Jul 27 23:26:33.175089 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:33.175062    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hf762"
Jul 27 23:26:33.203398 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:33.203314    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqcck\" (UniqueName: \"kubernetes.io/projected/597c89b7-6ab1-4bbf-ab24-e910095bae5e-kube-api-access-xqcck\") pod \"597c89b7-6ab1-4bbf-ab24-e910095bae5e\" (UID: \"597c89b7-6ab1-4bbf-ab24-e910095bae5e\") "
Jul 27 23:26:33.205611 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:33.205585    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597c89b7-6ab1-4bbf-ab24-e910095bae5e-kube-api-access-xqcck" (OuterVolumeSpecName: "kube-api-access-xqcck") pod "597c89b7-6ab1-4bbf-ab24-e910095bae5e" (UID: "597c89b7-6ab1-4bbf-ab24-e910095bae5e"). InnerVolumeSpecName "kube-api-access-xqcck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:26:33.304563 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:33.304534    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqcck\" (UniqueName: \"kubernetes.io/projected/597c89b7-6ab1-4bbf-ab24-e910095bae5e-kube-api-access-xqcck\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:26:33.932133 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:33.932040    2559 generic.go:356] "Generic (PLEG): container finished" podID="597c89b7-6ab1-4bbf-ab24-e910095bae5e" containerID="a50909160fe5276ba04bec69da1c6cfa86df68cff75886e93c6745d4b7cb143a" exitCode=0
Jul 27 23:26:33.932133 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:33.932083    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hf762"
Jul 27 23:26:33.932133 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:33.932123    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hf762" event={"ID":"597c89b7-6ab1-4bbf-ab24-e910095bae5e","Type":"ContainerDied","Data":"a50909160fe5276ba04bec69da1c6cfa86df68cff75886e93c6745d4b7cb143a"}
Jul 27 23:26:33.932636 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:33.932164    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hf762" event={"ID":"597c89b7-6ab1-4bbf-ab24-e910095bae5e","Type":"ContainerDied","Data":"9f1c269182753d8b1b977dbdd29e56f43fd8b6af3e7e426eef36e8c1575649ff"}
Jul 27 23:26:33.932636 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:33.932188    2559 scope.go:117] "RemoveContainer" containerID="a50909160fe5276ba04bec69da1c6cfa86df68cff75886e93c6745d4b7cb143a"
Jul 27 23:26:33.941021 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:33.941006    2559 scope.go:117] "RemoveContainer" containerID="a50909160fe5276ba04bec69da1c6cfa86df68cff75886e93c6745d4b7cb143a"
Jul 27 23:26:33.941269 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:26:33.941254    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50909160fe5276ba04bec69da1c6cfa86df68cff75886e93c6745d4b7cb143a\": container with ID starting with a50909160fe5276ba04bec69da1c6cfa86df68cff75886e93c6745d4b7cb143a not found: ID does not exist" containerID="a50909160fe5276ba04bec69da1c6cfa86df68cff75886e93c6745d4b7cb143a"
Jul 27 23:26:33.941306 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:33.941278    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50909160fe5276ba04bec69da1c6cfa86df68cff75886e93c6745d4b7cb143a"} err="failed to get container status \"a50909160fe5276ba04bec69da1c6cfa86df68cff75886e93c6745d4b7cb143a\": rpc error: code = NotFound desc = could not find container \"a50909160fe5276ba04bec69da1c6cfa86df68cff75886e93c6745d4b7cb143a\": container with ID starting with a50909160fe5276ba04bec69da1c6cfa86df68cff75886e93c6745d4b7cb143a not found: ID does not exist"
Jul 27 23:26:33.945603 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:33.945582    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hf762"]
Jul 27 23:26:33.947553 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:33.947534    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hf762"]
Jul 27 23:26:35.562075 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:35.562041    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="597c89b7-6ab1-4bbf-ab24-e910095bae5e" path="/var/lib/kubelet/pods/597c89b7-6ab1-4bbf-ab24-e910095bae5e/volumes"
Jul 27 23:26:40.483271 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:40.483186    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-5fksn"
Jul 27 23:26:40.483271 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:40.483243    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-5fksn"
Jul 27 23:26:40.505293 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:40.505267    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-5fksn"
Jul 27 23:26:40.982035 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:26:40.982009    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-5fksn"
Jul 27 23:27:00.718398 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:00.718337    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"]
Jul 27 23:27:00.718883 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:00.718696    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="597c89b7-6ab1-4bbf-ab24-e910095bae5e" containerName="registry-server"
Jul 27 23:27:00.718883 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:00.718708    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="597c89b7-6ab1-4bbf-ab24-e910095bae5e" containerName="registry-server"
Jul 27 23:27:00.718883 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:00.718770    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="597c89b7-6ab1-4bbf-ab24-e910095bae5e" containerName="registry-server"
Jul 27 23:27:00.724824 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:00.724808    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:00.726897 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:00.726856    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-cw579\""
Jul 27 23:27:00.729259 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:00.729236    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"]
Jul 27 23:27:00.837077 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:00.837047    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snkzz\" (UniqueName: \"kubernetes.io/projected/2a1061da-06b5-44c7-90f0-5a4537fc48b4-kube-api-access-snkzz\") pod \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" (UID: \"2a1061da-06b5-44c7-90f0-5a4537fc48b4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:00.938268 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:00.938224    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snkzz\" (UniqueName: \"kubernetes.io/projected/2a1061da-06b5-44c7-90f0-5a4537fc48b4-kube-api-access-snkzz\") pod \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" (UID: \"2a1061da-06b5-44c7-90f0-5a4537fc48b4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:00.951252 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:00.951214    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snkzz\" (UniqueName: \"kubernetes.io/projected/2a1061da-06b5-44c7-90f0-5a4537fc48b4-kube-api-access-snkzz\") pod \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" (UID: \"2a1061da-06b5-44c7-90f0-5a4537fc48b4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:01.036281 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:01.036251    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:01.167539 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:01.167512    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"]
Jul 27 23:27:01.170549 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:27:01.170515    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1061da_06b5_44c7_90f0_5a4537fc48b4.slice/crio-203196c2d1d3df586d95711ada725a82f345899b8aa17a996c071a7508cd011d WatchSource:0}: Error finding container 203196c2d1d3df586d95711ada725a82f345899b8aa17a996c071a7508cd011d: Status 404 returned error can't find the container with id 203196c2d1d3df586d95711ada725a82f345899b8aa17a996c071a7508cd011d
Jul 27 23:27:02.055297 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:02.055255    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk" event={"ID":"2a1061da-06b5-44c7-90f0-5a4537fc48b4","Type":"ContainerStarted","Data":"203196c2d1d3df586d95711ada725a82f345899b8aa17a996c071a7508cd011d"}
Jul 27 23:27:04.065951 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:04.065912    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk" event={"ID":"2a1061da-06b5-44c7-90f0-5a4537fc48b4","Type":"ContainerStarted","Data":"9470b1e9dff4698610375efea70f993255350ec984061e004176f2d9fb1f5e6b"}
Jul 27 23:27:04.066330 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:04.066008    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:04.080747 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:04.080704    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk" podStartSLOduration=2.170270575 podStartE2EDuration="4.080687851s" totalImagesPullingTime="1.910417276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:27:04.078810547 +0000 UTC m=+647.023041295" watchObservedRunningTime="2026-07-27 23:27:04.080687851 +0000 UTC m=+647.024918595"
Jul 27 23:27:05.648427 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:05.648385    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"]
Jul 27 23:27:05.651303 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:05.651287    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:05.653642 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:05.653619    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6nx7v\""
Jul 27 23:27:05.662508 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:05.662487    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"]
Jul 27 23:27:05.781552 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:05.781507    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7xbc\" (UniqueName: \"kubernetes.io/projected/df2863d0-96b0-4ead-bdeb-29fec8321207-kube-api-access-t7xbc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" (UID: \"df2863d0-96b0-4ead-bdeb-29fec8321207\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:05.781744 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:05.781698    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df2863d0-96b0-4ead-bdeb-29fec8321207-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" (UID: \"df2863d0-96b0-4ead-bdeb-29fec8321207\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:05.883112 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:05.883075    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df2863d0-96b0-4ead-bdeb-29fec8321207-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" (UID: \"df2863d0-96b0-4ead-bdeb-29fec8321207\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:05.883112 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:05.883114    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7xbc\" (UniqueName: \"kubernetes.io/projected/df2863d0-96b0-4ead-bdeb-29fec8321207-kube-api-access-t7xbc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" (UID: \"df2863d0-96b0-4ead-bdeb-29fec8321207\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:05.883504 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:05.883480    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df2863d0-96b0-4ead-bdeb-29fec8321207-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" (UID: \"df2863d0-96b0-4ead-bdeb-29fec8321207\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:05.892313 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:05.892286    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7xbc\" (UniqueName: \"kubernetes.io/projected/df2863d0-96b0-4ead-bdeb-29fec8321207-kube-api-access-t7xbc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" (UID: \"df2863d0-96b0-4ead-bdeb-29fec8321207\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:05.961568 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:05.961479    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:06.098047 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:06.098022    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"]
Jul 27 23:27:06.101014 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:27:06.100990    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2863d0_96b0_4ead_bdeb_29fec8321207.slice/crio-6fc04f5f07e7b8b9adefb118cd39519c995de23e27310bbd1c906db53f88cf15 WatchSource:0}: Error finding container 6fc04f5f07e7b8b9adefb118cd39519c995de23e27310bbd1c906db53f88cf15: Status 404 returned error can't find the container with id 6fc04f5f07e7b8b9adefb118cd39519c995de23e27310bbd1c906db53f88cf15
Jul 27 23:27:07.081255 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:07.081217    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8" event={"ID":"df2863d0-96b0-4ead-bdeb-29fec8321207","Type":"ContainerStarted","Data":"6fc04f5f07e7b8b9adefb118cd39519c995de23e27310bbd1c906db53f88cf15"}
Jul 27 23:27:08.933824 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:08.933792    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqv9h"]
Jul 27 23:27:08.937138 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:08.937121    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqv9h"
Jul 27 23:27:08.939343 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:08.939320    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 27 23:27:08.939472 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:08.939446    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cbdvg\""
Jul 27 23:27:08.948676 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:08.948654    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqv9h"]
Jul 27 23:27:09.109487 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:09.109453    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w8jg\" (UniqueName: \"kubernetes.io/projected/605263d2-8841-417a-974e-640173936e64-kube-api-access-7w8jg\") pod \"dns-operator-controller-manager-864dd78c4c-qqv9h\" (UID: \"605263d2-8841-417a-974e-640173936e64\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqv9h"
Jul 27 23:27:09.210526 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:09.210441    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w8jg\" (UniqueName: \"kubernetes.io/projected/605263d2-8841-417a-974e-640173936e64-kube-api-access-7w8jg\") pod \"dns-operator-controller-manager-864dd78c4c-qqv9h\" (UID: \"605263d2-8841-417a-974e-640173936e64\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqv9h"
Jul 27 23:27:09.220647 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:09.220618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w8jg\" (UniqueName: \"kubernetes.io/projected/605263d2-8841-417a-974e-640173936e64-kube-api-access-7w8jg\") pod \"dns-operator-controller-manager-864dd78c4c-qqv9h\" (UID: \"605263d2-8841-417a-974e-640173936e64\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqv9h"
Jul 27 23:27:09.247563 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:09.247540    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqv9h"
Jul 27 23:27:09.387662 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:09.387635    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqv9h"]
Jul 27 23:27:09.389944 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:27:09.389913    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605263d2_8841_417a_974e_640173936e64.slice/crio-fb37ee019d75d5b4c57386a28560f5ddfb6eaafcc8ab9107aa6755e5b609195c WatchSource:0}: Error finding container fb37ee019d75d5b4c57386a28560f5ddfb6eaafcc8ab9107aa6755e5b609195c: Status 404 returned error can't find the container with id fb37ee019d75d5b4c57386a28560f5ddfb6eaafcc8ab9107aa6755e5b609195c
Jul 27 23:27:10.096815 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:10.096784    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqv9h" event={"ID":"605263d2-8841-417a-974e-640173936e64","Type":"ContainerStarted","Data":"fb37ee019d75d5b4c57386a28560f5ddfb6eaafcc8ab9107aa6755e5b609195c"}
Jul 27 23:27:12.323431 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:12.323393    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nsdgv"]
Jul 27 23:27:12.327249 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:12.327220    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nsdgv"
Jul 27 23:27:12.329653 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:12.329632    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hbzht\""
Jul 27 23:27:12.337035 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:12.337011    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nsdgv"]
Jul 27 23:27:12.442262 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:12.442235    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46792\" (UniqueName: \"kubernetes.io/projected/fb75458a-d253-4bea-a666-344d36bee972-kube-api-access-46792\") pod \"authorino-operator-75cc8d5cb7-nsdgv\" (UID: \"fb75458a-d253-4bea-a666-344d36bee972\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nsdgv"
Jul 27 23:27:12.543742 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:12.543703    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46792\" (UniqueName: \"kubernetes.io/projected/fb75458a-d253-4bea-a666-344d36bee972-kube-api-access-46792\") pod \"authorino-operator-75cc8d5cb7-nsdgv\" (UID: \"fb75458a-d253-4bea-a666-344d36bee972\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nsdgv"
Jul 27 23:27:12.559939 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:12.559906    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46792\" (UniqueName: \"kubernetes.io/projected/fb75458a-d253-4bea-a666-344d36bee972-kube-api-access-46792\") pod \"authorino-operator-75cc8d5cb7-nsdgv\" (UID: \"fb75458a-d253-4bea-a666-344d36bee972\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nsdgv"
Jul 27 23:27:12.641293 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:12.641204    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nsdgv"
Jul 27 23:27:12.791096 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:12.791049    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nsdgv"]
Jul 27 23:27:12.795023 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:27:12.794990    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb75458a_d253_4bea_a666_344d36bee972.slice/crio-60938af6d02599e76cee7e1df6481642252d1841b26922ae914cb8563d753f9c WatchSource:0}: Error finding container 60938af6d02599e76cee7e1df6481642252d1841b26922ae914cb8563d753f9c: Status 404 returned error can't find the container with id 60938af6d02599e76cee7e1df6481642252d1841b26922ae914cb8563d753f9c
Jul 27 23:27:13.114325 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:13.114294    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqv9h" event={"ID":"605263d2-8841-417a-974e-640173936e64","Type":"ContainerStarted","Data":"64ca7a89c9245646ce5d05913337760fcb5804ae0e408fdc09f16e094c25c064"}
Jul 27 23:27:13.114540 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:13.114418    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqv9h"
Jul 27 23:27:13.115739 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:13.115716    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nsdgv" event={"ID":"fb75458a-d253-4bea-a666-344d36bee972","Type":"ContainerStarted","Data":"60938af6d02599e76cee7e1df6481642252d1841b26922ae914cb8563d753f9c"}
Jul 27 23:27:13.117313 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:13.117294    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8" event={"ID":"df2863d0-96b0-4ead-bdeb-29fec8321207","Type":"ContainerStarted","Data":"b33583a5bf6cfd726e3650e0a0002d0d62983cdf80e08a1c056f5acad0816d77"}
Jul 27 23:27:13.117439 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:13.117422    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:13.134783 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:13.134727    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqv9h" podStartSLOduration=2.126578062 podStartE2EDuration="5.134705632s" totalImagesPullingTime="3.00812757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:27:13.132946855 +0000 UTC m=+656.077177601" watchObservedRunningTime="2026-07-27 23:27:13.134705632 +0000 UTC m=+656.078936378"
Jul 27 23:27:13.150818 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:13.150780    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8" podStartSLOduration=2.331677028 podStartE2EDuration="8.150746165s" totalImagesPullingTime="5.819069137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:27:13.149811795 +0000 UTC m=+656.094042543" watchObservedRunningTime="2026-07-27 23:27:13.150746165 +0000 UTC m=+656.094976909"
Jul 27 23:27:15.073434 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:15.073397    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:16.134816 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:16.134774    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nsdgv" event={"ID":"fb75458a-d253-4bea-a666-344d36bee972","Type":"ContainerStarted","Data":"db2700a5c3056616e8d02b95b5c1746f07dc8df43b7dec590ed0f6df7e8150d6"}
Jul 27 23:27:16.135206 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:16.134850    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nsdgv"
Jul 27 23:27:16.149953 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:16.149908    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nsdgv" podStartSLOduration=1.8162644060000002 podStartE2EDuration="4.149894076s" totalImagesPullingTime="2.33362967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:27:16.149705719 +0000 UTC m=+659.093936468" watchObservedRunningTime="2026-07-27 23:27:16.149894076 +0000 UTC m=+659.094124886"
Jul 27 23:27:24.126003 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:24.125965    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qqv9h"
Jul 27 23:27:24.126421 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:24.126130    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:25.870905 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.870875    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"]
Jul 27 23:27:25.871304 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.871048    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" containerName="manager" containerID="cri-o://b33583a5bf6cfd726e3650e0a0002d0d62983cdf80e08a1c056f5acad0816d77" gracePeriod=2
Jul 27 23:27:25.877989 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.877960    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"]
Jul 27 23:27:25.891257 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.890830    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"]
Jul 27 23:27:25.891257 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.891075    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" containerName="manager" containerID="cri-o://9470b1e9dff4698610375efea70f993255350ec984061e004176f2d9fb1f5e6b" gracePeriod=2
Jul 27 23:27:25.892872 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.892843    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"]
Jul 27 23:27:25.894522 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.893356    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" containerName="manager"
Jul 27 23:27:25.894522 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.893394    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" containerName="manager"
Jul 27 23:27:25.894522 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.893476    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" containerName="manager"
Jul 27 23:27:25.894522 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:25.893601    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:25.895604 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:25.895575    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:25.896610 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.896592    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"]
Jul 27 23:27:25.896718 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.896680    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"
Jul 27 23:27:25.898395 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:25.898346    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:25.911121 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.911096    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"]
Jul 27 23:27:25.913216 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.913195    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k46lx"]
Jul 27 23:27:25.913700 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.913681    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" containerName="manager"
Jul 27 23:27:25.913772 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.913704    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" containerName="manager"
Jul 27 23:27:25.913827 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.913796    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" containerName="manager"
Jul 27 23:27:25.917113 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.917095    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k46lx"
Jul 27 23:27:25.925010 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:25.924976    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:25.926560 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:25.926523    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:25.927714 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.927688    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k46lx"]
Jul 27 23:27:25.951893 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:25.951849    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:25.959411 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.959362    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6prf8\" (UniqueName: \"kubernetes.io/projected/999737a5-5e30-445a-a605-c0e6d423f77d-kube-api-access-6prf8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jgzmd\" (UID: \"999737a5-5e30-445a-a605-c0e6d423f77d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"
Jul 27 23:27:25.959511 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.959465    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/999737a5-5e30-445a-a605-c0e6d423f77d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jgzmd\" (UID: \"999737a5-5e30-445a-a605-c0e6d423f77d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"
Jul 27 23:27:25.959511 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:25.959493    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw9jf\" (UniqueName: \"kubernetes.io/projected/36f61d75-c765-4434-a582-344825e5cd38-kube-api-access-lw9jf\") pod \"limitador-operator-controller-manager-859d7b78bd-k46lx\" (UID: \"36f61d75-c765-4434-a582-344825e5cd38\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k46lx"
Jul 27 23:27:26.061003 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.060966    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/999737a5-5e30-445a-a605-c0e6d423f77d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jgzmd\" (UID: \"999737a5-5e30-445a-a605-c0e6d423f77d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"
Jul 27 23:27:26.061171 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.061042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw9jf\" (UniqueName: \"kubernetes.io/projected/36f61d75-c765-4434-a582-344825e5cd38-kube-api-access-lw9jf\") pod \"limitador-operator-controller-manager-859d7b78bd-k46lx\" (UID: \"36f61d75-c765-4434-a582-344825e5cd38\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k46lx"
Jul 27 23:27:26.061244 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.061167    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6prf8\" (UniqueName: \"kubernetes.io/projected/999737a5-5e30-445a-a605-c0e6d423f77d-kube-api-access-6prf8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jgzmd\" (UID: \"999737a5-5e30-445a-a605-c0e6d423f77d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"
Jul 27 23:27:26.061293 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.061278    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/999737a5-5e30-445a-a605-c0e6d423f77d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jgzmd\" (UID: \"999737a5-5e30-445a-a605-c0e6d423f77d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"
Jul 27 23:27:26.070002 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.069969    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6prf8\" (UniqueName: \"kubernetes.io/projected/999737a5-5e30-445a-a605-c0e6d423f77d-kube-api-access-6prf8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jgzmd\" (UID: \"999737a5-5e30-445a-a605-c0e6d423f77d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"
Jul 27 23:27:26.070129 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.070044    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw9jf\" (UniqueName: \"kubernetes.io/projected/36f61d75-c765-4434-a582-344825e5cd38-kube-api-access-lw9jf\") pod \"limitador-operator-controller-manager-859d7b78bd-k46lx\" (UID: \"36f61d75-c765-4434-a582-344825e5cd38\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k46lx"
Jul 27 23:27:26.133350 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.133321    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:26.135133 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.135106    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:26.135477 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.135462    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:26.136448 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.136430    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:26.137760 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.137742    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:26.138986 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.138968    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:26.162470 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.162444    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7xbc\" (UniqueName: \"kubernetes.io/projected/df2863d0-96b0-4ead-bdeb-29fec8321207-kube-api-access-t7xbc\") pod \"df2863d0-96b0-4ead-bdeb-29fec8321207\" (UID: \"df2863d0-96b0-4ead-bdeb-29fec8321207\") "
Jul 27 23:27:26.162624 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.162477    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df2863d0-96b0-4ead-bdeb-29fec8321207-extensions-socket-volume\") pod \"df2863d0-96b0-4ead-bdeb-29fec8321207\" (UID: \"df2863d0-96b0-4ead-bdeb-29fec8321207\") "
Jul 27 23:27:26.162624 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.162521    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snkzz\" (UniqueName: \"kubernetes.io/projected/2a1061da-06b5-44c7-90f0-5a4537fc48b4-kube-api-access-snkzz\") pod \"2a1061da-06b5-44c7-90f0-5a4537fc48b4\" (UID: \"2a1061da-06b5-44c7-90f0-5a4537fc48b4\") "
Jul 27 23:27:26.162930 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.162902    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2863d0-96b0-4ead-bdeb-29fec8321207-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "df2863d0-96b0-4ead-bdeb-29fec8321207" (UID: "df2863d0-96b0-4ead-bdeb-29fec8321207"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:27:26.164784 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.164765    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2863d0-96b0-4ead-bdeb-29fec8321207-kube-api-access-t7xbc" (OuterVolumeSpecName: "kube-api-access-t7xbc") pod "df2863d0-96b0-4ead-bdeb-29fec8321207" (UID: "df2863d0-96b0-4ead-bdeb-29fec8321207"). InnerVolumeSpecName "kube-api-access-t7xbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:27:26.164838 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.164823    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1061da-06b5-44c7-90f0-5a4537fc48b4-kube-api-access-snkzz" (OuterVolumeSpecName: "kube-api-access-snkzz") pod "2a1061da-06b5-44c7-90f0-5a4537fc48b4" (UID: "2a1061da-06b5-44c7-90f0-5a4537fc48b4"). InnerVolumeSpecName "kube-api-access-snkzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:27:26.185074 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.185042    2559 generic.go:356] "Generic (PLEG): container finished" podID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" containerID="9470b1e9dff4698610375efea70f993255350ec984061e004176f2d9fb1f5e6b" exitCode=0
Jul 27 23:27:26.185247 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.185087    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:26.185247 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.185150    2559 scope.go:117] "RemoveContainer" containerID="9470b1e9dff4698610375efea70f993255350ec984061e004176f2d9fb1f5e6b"
Jul 27 23:27:26.186503 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.186479    2559 generic.go:356] "Generic (PLEG): container finished" podID="df2863d0-96b0-4ead-bdeb-29fec8321207" containerID="b33583a5bf6cfd726e3650e0a0002d0d62983cdf80e08a1c056f5acad0816d77" exitCode=0
Jul 27 23:27:26.186610 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.186519    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:26.187113 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.187089    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:26.188589 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.188564    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:26.189897 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.189877    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:26.191251 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.191222    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:26.194340 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.194324    2559 scope.go:117] "RemoveContainer" containerID="9470b1e9dff4698610375efea70f993255350ec984061e004176f2d9fb1f5e6b"
Jul 27 23:27:26.194633 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.194612    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9470b1e9dff4698610375efea70f993255350ec984061e004176f2d9fb1f5e6b\": container with ID starting with 9470b1e9dff4698610375efea70f993255350ec984061e004176f2d9fb1f5e6b not found: ID does not exist" containerID="9470b1e9dff4698610375efea70f993255350ec984061e004176f2d9fb1f5e6b"
Jul 27 23:27:26.194710 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.194641    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9470b1e9dff4698610375efea70f993255350ec984061e004176f2d9fb1f5e6b"} err="failed to get container status \"9470b1e9dff4698610375efea70f993255350ec984061e004176f2d9fb1f5e6b\": rpc error: code = NotFound desc = could not find container \"9470b1e9dff4698610375efea70f993255350ec984061e004176f2d9fb1f5e6b\": container with ID starting with 9470b1e9dff4698610375efea70f993255350ec984061e004176f2d9fb1f5e6b not found: ID does not exist"
Jul 27 23:27:26.194710 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.194667    2559 scope.go:117] "RemoveContainer" containerID="b33583a5bf6cfd726e3650e0a0002d0d62983cdf80e08a1c056f5acad0816d77"
Jul 27 23:27:26.198093 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.198073    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:26.199622 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.199599    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:26.201049 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.201021    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:26.202392 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.202299    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:26.202452 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.202438    2559 scope.go:117] "RemoveContainer" containerID="b33583a5bf6cfd726e3650e0a0002d0d62983cdf80e08a1c056f5acad0816d77"
Jul 27 23:27:26.202705 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.202687    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33583a5bf6cfd726e3650e0a0002d0d62983cdf80e08a1c056f5acad0816d77\": container with ID starting with b33583a5bf6cfd726e3650e0a0002d0d62983cdf80e08a1c056f5acad0816d77 not found: ID does not exist" containerID="b33583a5bf6cfd726e3650e0a0002d0d62983cdf80e08a1c056f5acad0816d77"
Jul 27 23:27:26.202812 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.202710    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33583a5bf6cfd726e3650e0a0002d0d62983cdf80e08a1c056f5acad0816d77"} err="failed to get container status \"b33583a5bf6cfd726e3650e0a0002d0d62983cdf80e08a1c056f5acad0816d77\": rpc error: code = NotFound desc = could not find container \"b33583a5bf6cfd726e3650e0a0002d0d62983cdf80e08a1c056f5acad0816d77\": container with ID starting with b33583a5bf6cfd726e3650e0a0002d0d62983cdf80e08a1c056f5acad0816d77 not found: ID does not exist"
Jul 27 23:27:26.263282 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.263259    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7xbc\" (UniqueName: \"kubernetes.io/projected/df2863d0-96b0-4ead-bdeb-29fec8321207-kube-api-access-t7xbc\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:27:26.263282 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.263283    2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df2863d0-96b0-4ead-bdeb-29fec8321207-extensions-socket-volume\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:27:26.263469 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.263294    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snkzz\" (UniqueName: \"kubernetes.io/projected/2a1061da-06b5-44c7-90f0-5a4537fc48b4-kube-api-access-snkzz\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:27:26.327233 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.327206    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"
Jul 27 23:27:26.335698 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.335680    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k46lx"
Jul 27 23:27:26.493853 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.493829    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"]
Jul 27 23:27:26.496410 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:27:26.496357    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999737a5_5e30_445a_a605_c0e6d423f77d.slice/crio-48c7b8068898bcf3b9de0a8580904a7f5fabe70150811dd48197ec0499de871e WatchSource:0}: Error finding container 48c7b8068898bcf3b9de0a8580904a7f5fabe70150811dd48197ec0499de871e: Status 404 returned error can't find the container with id 48c7b8068898bcf3b9de0a8580904a7f5fabe70150811dd48197ec0499de871e
Jul 27 23:27:26.508007 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.507983    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k46lx"]
Jul 27 23:27:26.510258 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:27:26.510230    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f61d75_c765_4434_a582_344825e5cd38.slice/crio-d084564a74983753d4f1bb906d7007fb2d73ee38cd6391c08a7ae50bdee8c22f WatchSource:0}: Error finding container d084564a74983753d4f1bb906d7007fb2d73ee38cd6391c08a7ae50bdee8c22f: Status 404 returned error can't find the container with id d084564a74983753d4f1bb906d7007fb2d73ee38cd6391c08a7ae50bdee8c22f
Jul 27 23:27:26.779703 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.779672    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"]
Jul 27 23:27:26.783080 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.783061    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"
Jul 27 23:27:26.784941 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.784914    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:26.786460 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:26.786437    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:26.797806 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.797779    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"]
Jul 27 23:27:26.868725 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.868686    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kglpp\" (UniqueName: \"kubernetes.io/projected/24ddd5ac-46a3-4c58-8414-0da283830da0-kube-api-access-kglpp\") pod \"kuadrant-operator-controller-manager-86878c769f-mg64d\" (UID: \"24ddd5ac-46a3-4c58-8414-0da283830da0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"
Jul 27 23:27:26.868889 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.868756    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24ddd5ac-46a3-4c58-8414-0da283830da0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mg64d\" (UID: \"24ddd5ac-46a3-4c58-8414-0da283830da0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"
Jul 27 23:27:26.969396 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.969333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kglpp\" (UniqueName: \"kubernetes.io/projected/24ddd5ac-46a3-4c58-8414-0da283830da0-kube-api-access-kglpp\") pod \"kuadrant-operator-controller-manager-86878c769f-mg64d\" (UID: \"24ddd5ac-46a3-4c58-8414-0da283830da0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"
Jul 27 23:27:26.969830 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.969455    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24ddd5ac-46a3-4c58-8414-0da283830da0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mg64d\" (UID: \"24ddd5ac-46a3-4c58-8414-0da283830da0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"
Jul 27 23:27:26.969941 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.969918    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24ddd5ac-46a3-4c58-8414-0da283830da0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mg64d\" (UID: \"24ddd5ac-46a3-4c58-8414-0da283830da0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"
Jul 27 23:27:26.981482 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:26.981455    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kglpp\" (UniqueName: \"kubernetes.io/projected/24ddd5ac-46a3-4c58-8414-0da283830da0-kube-api-access-kglpp\") pod \"kuadrant-operator-controller-manager-86878c769f-mg64d\" (UID: \"24ddd5ac-46a3-4c58-8414-0da283830da0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"
Jul 27 23:27:27.093361 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:27.093284    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"
Jul 27 23:27:27.142256 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:27.142224    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nsdgv"
Jul 27 23:27:27.163523 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:27.163487    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:27.164936 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:27.164905    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:27.196829 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:27.196794    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd" event={"ID":"999737a5-5e30-445a-a605-c0e6d423f77d","Type":"ContainerStarted","Data":"27624357a7929d2885ce7420222e93dc09c3cab0c446824ca7ec0f27ab5fca2a"}
Jul 27 23:27:27.197013 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:27.196855    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd" event={"ID":"999737a5-5e30-445a-a605-c0e6d423f77d","Type":"ContainerStarted","Data":"48c7b8068898bcf3b9de0a8580904a7f5fabe70150811dd48197ec0499de871e"}
Jul 27 23:27:27.197277 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:27.197256    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"
Jul 27 23:27:27.198588 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:27.198448    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:27.200042 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:27.199938    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:27.200177 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:27.200121    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k46lx" event={"ID":"36f61d75-c765-4434-a582-344825e5cd38","Type":"ContainerStarted","Data":"43593fdc24634ab65a25c429544c1fe9ee8755dfc586b9d6d24eb5f36b790a45"}
Jul 27 23:27:27.200177 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:27.200156    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k46lx" event={"ID":"36f61d75-c765-4434-a582-344825e5cd38","Type":"ContainerStarted","Data":"d084564a74983753d4f1bb906d7007fb2d73ee38cd6391c08a7ae50bdee8c22f"}
Jul 27 23:27:27.200323 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:27.200308    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k46lx"
Jul 27 23:27:27.220175 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:27.220045    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:27.221284 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:27.221222    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd" podStartSLOduration=2.221206708 podStartE2EDuration="2.221206708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:27:27.218350101 +0000 UTC m=+670.162580847" watchObservedRunningTime="2026-07-27 23:27:27.221206708 +0000 UTC m=+670.165437453"
Jul 27 23:27:27.221493 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:27.221411    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:27.237785 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:27.237764    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"]
Jul 27 23:27:27.240071 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:27.240028    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k46lx" podStartSLOduration=2.240012992 podStartE2EDuration="2.240012992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:27:27.238574752 +0000 UTC m=+670.182805497" watchObservedRunningTime="2026-07-27 23:27:27.240012992 +0000 UTC m=+670.184243736"
Jul 27 23:27:27.241444 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:27:27.241422    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ddd5ac_46a3_4c58_8414_0da283830da0.slice/crio-ff59ea24747ec43d85a72ba50557b8f9c3df7b05614d44600f06c490cb334782 WatchSource:0}: Error finding container ff59ea24747ec43d85a72ba50557b8f9c3df7b05614d44600f06c490cb334782: Status 404 returned error can't find the container with id ff59ea24747ec43d85a72ba50557b8f9c3df7b05614d44600f06c490cb334782
Jul 27 23:27:27.563110 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:27.563079    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" path="/var/lib/kubelet/pods/2a1061da-06b5-44c7-90f0-5a4537fc48b4/volumes"
Jul 27 23:27:27.563460 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:27.563447    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" path="/var/lib/kubelet/pods/df2863d0-96b0-4ead-bdeb-29fec8321207/volumes"
Jul 27 23:27:27.565356 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:27.565323    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fsqsk\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="2a1061da-06b5-44c7-90f0-5a4537fc48b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fsqsk"
Jul 27 23:27:27.567202 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:27.567181    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l6fh8\" is forbidden: User \"system:node:ip-10-0-142-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-53.ec2.internal' and this object" podUID="df2863d0-96b0-4ead-bdeb-29fec8321207" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l6fh8"
Jul 27 23:27:28.207045 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:28.207008    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d" event={"ID":"24ddd5ac-46a3-4c58-8414-0da283830da0","Type":"ContainerStarted","Data":"a2f73e21645032fbe649da1832e258f18621025fbb325741c2ef8ca7227644ac"}
Jul 27 23:27:28.207454 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:28.207067    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d" event={"ID":"24ddd5ac-46a3-4c58-8414-0da283830da0","Type":"ContainerStarted","Data":"ff59ea24747ec43d85a72ba50557b8f9c3df7b05614d44600f06c490cb334782"}
Jul 27 23:27:28.207454 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:28.207122    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"
Jul 27 23:27:28.231061 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:28.230970    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d" podStartSLOduration=2.230918 podStartE2EDuration="2.230918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:27:28.229731757 +0000 UTC m=+671.173962505" watchObservedRunningTime="2026-07-27 23:27:28.230918 +0000 UTC m=+671.175148748"
Jul 27 23:27:38.209318 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:38.209285    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k46lx"
Jul 27 23:27:39.213961 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:39.213931    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"
Jul 27 23:27:39.214433 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:39.214071    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"
Jul 27 23:27:39.283407 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:39.283348    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"]
Jul 27 23:27:39.283765 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:39.283734    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd" podUID="999737a5-5e30-445a-a605-c0e6d423f77d" containerName="manager" containerID="cri-o://27624357a7929d2885ce7420222e93dc09c3cab0c446824ca7ec0f27ab5fca2a" gracePeriod=10
Jul 27 23:27:39.543629 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:39.543603    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"
Jul 27 23:27:39.675703 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:39.675666    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6prf8\" (UniqueName: \"kubernetes.io/projected/999737a5-5e30-445a-a605-c0e6d423f77d-kube-api-access-6prf8\") pod \"999737a5-5e30-445a-a605-c0e6d423f77d\" (UID: \"999737a5-5e30-445a-a605-c0e6d423f77d\") "
Jul 27 23:27:39.675703 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:39.675702    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/999737a5-5e30-445a-a605-c0e6d423f77d-extensions-socket-volume\") pod \"999737a5-5e30-445a-a605-c0e6d423f77d\" (UID: \"999737a5-5e30-445a-a605-c0e6d423f77d\") "
Jul 27 23:27:39.676196 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:39.676169    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999737a5-5e30-445a-a605-c0e6d423f77d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "999737a5-5e30-445a-a605-c0e6d423f77d" (UID: "999737a5-5e30-445a-a605-c0e6d423f77d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:27:39.677992 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:39.677966    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999737a5-5e30-445a-a605-c0e6d423f77d-kube-api-access-6prf8" (OuterVolumeSpecName: "kube-api-access-6prf8") pod "999737a5-5e30-445a-a605-c0e6d423f77d" (UID: "999737a5-5e30-445a-a605-c0e6d423f77d"). InnerVolumeSpecName "kube-api-access-6prf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:27:39.777560 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:39.777467    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6prf8\" (UniqueName: \"kubernetes.io/projected/999737a5-5e30-445a-a605-c0e6d423f77d-kube-api-access-6prf8\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:27:39.777560 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:39.777501    2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/999737a5-5e30-445a-a605-c0e6d423f77d-extensions-socket-volume\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:27:40.266447 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:40.266412    2559 generic.go:356] "Generic (PLEG): container finished" podID="999737a5-5e30-445a-a605-c0e6d423f77d" containerID="27624357a7929d2885ce7420222e93dc09c3cab0c446824ca7ec0f27ab5fca2a" exitCode=0
Jul 27 23:27:40.266936 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:40.266459    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"
Jul 27 23:27:40.266936 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:40.266502    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd" event={"ID":"999737a5-5e30-445a-a605-c0e6d423f77d","Type":"ContainerDied","Data":"27624357a7929d2885ce7420222e93dc09c3cab0c446824ca7ec0f27ab5fca2a"}
Jul 27 23:27:40.266936 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:40.266565    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd" event={"ID":"999737a5-5e30-445a-a605-c0e6d423f77d","Type":"ContainerDied","Data":"48c7b8068898bcf3b9de0a8580904a7f5fabe70150811dd48197ec0499de871e"}
Jul 27 23:27:40.266936 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:40.266600    2559 scope.go:117] "RemoveContainer" containerID="27624357a7929d2885ce7420222e93dc09c3cab0c446824ca7ec0f27ab5fca2a"
Jul 27 23:27:40.275521 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:40.275502    2559 scope.go:117] "RemoveContainer" containerID="27624357a7929d2885ce7420222e93dc09c3cab0c446824ca7ec0f27ab5fca2a"
Jul 27 23:27:40.275767 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:27:40.275751    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27624357a7929d2885ce7420222e93dc09c3cab0c446824ca7ec0f27ab5fca2a\": container with ID starting with 27624357a7929d2885ce7420222e93dc09c3cab0c446824ca7ec0f27ab5fca2a not found: ID does not exist" containerID="27624357a7929d2885ce7420222e93dc09c3cab0c446824ca7ec0f27ab5fca2a"
Jul 27 23:27:40.275803 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:40.275776    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27624357a7929d2885ce7420222e93dc09c3cab0c446824ca7ec0f27ab5fca2a"} err="failed to get container status \"27624357a7929d2885ce7420222e93dc09c3cab0c446824ca7ec0f27ab5fca2a\": rpc error: code = NotFound desc = could not find container \"27624357a7929d2885ce7420222e93dc09c3cab0c446824ca7ec0f27ab5fca2a\": container with ID starting with 27624357a7929d2885ce7420222e93dc09c3cab0c446824ca7ec0f27ab5fca2a not found: ID does not exist"
Jul 27 23:27:40.289117 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:40.289095    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"]
Jul 27 23:27:40.295830 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:40.295808    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jgzmd"]
Jul 27 23:27:41.562203 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:41.562170    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="999737a5-5e30-445a-a605-c0e6d423f77d" path="/var/lib/kubelet/pods/999737a5-5e30-445a-a605-c0e6d423f77d/volumes"
Jul 27 23:27:55.514296 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.514264    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"]
Jul 27 23:27:55.514756 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.514742    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="999737a5-5e30-445a-a605-c0e6d423f77d" containerName="manager"
Jul 27 23:27:55.514756 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.514757    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="999737a5-5e30-445a-a605-c0e6d423f77d" containerName="manager"
Jul 27 23:27:55.514863 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.514853    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="999737a5-5e30-445a-a605-c0e6d423f77d" containerName="manager"
Jul 27 23:27:55.519834 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.519812    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.521785 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.521768    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pk544\""
Jul 27 23:27:55.529341 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.529321    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"]
Jul 27 23:27:55.620928 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.620897    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg9zw\" (UniqueName: \"kubernetes.io/projected/d706cd9a-be1b-4544-9740-12f63c54c73e-kube-api-access-rg9zw\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.621103 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.620942    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d706cd9a-be1b-4544-9740-12f63c54c73e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.621103 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.620963    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.621103 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.620998    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.621103 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.621071    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d706cd9a-be1b-4544-9740-12f63c54c73e-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.621268 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.621151    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.621268 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.621189    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.621268 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.621234    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.621395 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.621265    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d706cd9a-be1b-4544-9740-12f63c54c73e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.722459 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.722424    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d706cd9a-be1b-4544-9740-12f63c54c73e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.722635 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.722472    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg9zw\" (UniqueName: \"kubernetes.io/projected/d706cd9a-be1b-4544-9740-12f63c54c73e-kube-api-access-rg9zw\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.722635 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.722511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d706cd9a-be1b-4544-9740-12f63c54c73e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.722635 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.722538    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.722635 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.722566    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.722635 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.722593    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d706cd9a-be1b-4544-9740-12f63c54c73e-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.722913 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.722652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.722913 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.722687    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.722913 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.722729    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.723132 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.723077    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.723132 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.723100    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.723332 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.723162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.723332 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.723264    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.723512 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.723496    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d706cd9a-be1b-4544-9740-12f63c54c73e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.725614 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.725594    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d706cd9a-be1b-4544-9740-12f63c54c73e-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.725740 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.725723    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d706cd9a-be1b-4544-9740-12f63c54c73e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.729428 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.729410    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg9zw\" (UniqueName: \"kubernetes.io/projected/d706cd9a-be1b-4544-9740-12f63c54c73e-kube-api-access-rg9zw\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.729654 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.729633    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d706cd9a-be1b-4544-9740-12f63c54c73e-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hbnzn\" (UID: \"d706cd9a-be1b-4544-9740-12f63c54c73e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.830730 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.830702    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:55.973570 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.973547    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"]
Jul 27 23:27:55.975753 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:27:55.975726    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd706cd9a_be1b_4544_9740_12f63c54c73e.slice/crio-e01fdd84b5b59247059b0cfae62ac6431a63819a1356271f15e479d1bae24eb7 WatchSource:0}: Error finding container e01fdd84b5b59247059b0cfae62ac6431a63819a1356271f15e479d1bae24eb7: Status 404 returned error can't find the container with id e01fdd84b5b59247059b0cfae62ac6431a63819a1356271f15e479d1bae24eb7
Jul 27 23:27:55.977967 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.977918    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:27:55.978047 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.977994    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:27:55.978047 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:55.978024    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:27:56.352742 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:56.352707    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn" event={"ID":"d706cd9a-be1b-4544-9740-12f63c54c73e","Type":"ContainerStarted","Data":"6e841d9245b55c7889005b76ccf5830c0c7c56960a9947041cf236df61f6461f"}
Jul 27 23:27:56.352918 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:56.352775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn" event={"ID":"d706cd9a-be1b-4544-9740-12f63c54c73e","Type":"ContainerStarted","Data":"e01fdd84b5b59247059b0cfae62ac6431a63819a1356271f15e479d1bae24eb7"}
Jul 27 23:27:56.371803 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:56.371756    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn" podStartSLOduration=1.371741835 podStartE2EDuration="1.371741835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:27:56.370072825 +0000 UTC m=+699.314303583" watchObservedRunningTime="2026-07-27 23:27:56.371741835 +0000 UTC m=+699.315972582"
Jul 27 23:27:56.831049 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:56.831005    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:57.836867 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:57.836840    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:58.077054 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:58.077024    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-rv9qd"]
Jul 27 23:27:58.079791 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:58.079771    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rv9qd"
Jul 27 23:27:58.081867 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:58.081851    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nx6j8\""
Jul 27 23:27:58.086026 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:58.086005    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rv9qd"]
Jul 27 23:27:58.245715 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:58.245683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdd8k\" (UniqueName: \"kubernetes.io/projected/69a56444-afee-4c94-8589-e8e0acaf2c3b-kube-api-access-rdd8k\") pod \"authorino-58cd955675-rv9qd\" (UID: \"69a56444-afee-4c94-8589-e8e0acaf2c3b\") " pod="kuadrant-system/authorino-58cd955675-rv9qd"
Jul 27 23:27:58.346168 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:58.346084    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdd8k\" (UniqueName: \"kubernetes.io/projected/69a56444-afee-4c94-8589-e8e0acaf2c3b-kube-api-access-rdd8k\") pod \"authorino-58cd955675-rv9qd\" (UID: \"69a56444-afee-4c94-8589-e8e0acaf2c3b\") " pod="kuadrant-system/authorino-58cd955675-rv9qd"
Jul 27 23:27:58.353513 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:58.353481    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdd8k\" (UniqueName: \"kubernetes.io/projected/69a56444-afee-4c94-8589-e8e0acaf2c3b-kube-api-access-rdd8k\") pod \"authorino-58cd955675-rv9qd\" (UID: \"69a56444-afee-4c94-8589-e8e0acaf2c3b\") " pod="kuadrant-system/authorino-58cd955675-rv9qd"
Jul 27 23:27:58.364422 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:58.364400    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:58.365222 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:58.365203    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:27:58.391444 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:58.391408    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rv9qd"
Jul 27 23:27:58.555899 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:58.555863    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rv9qd"]
Jul 27 23:27:58.558558 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:27:58.558530    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69a56444_afee_4c94_8589_e8e0acaf2c3b.slice/crio-9c7753f7ec4c6402e183b2c2398c0908b2307ac84104918ed6d488c1fac1b2da WatchSource:0}: Error finding container 9c7753f7ec4c6402e183b2c2398c0908b2307ac84104918ed6d488c1fac1b2da: Status 404 returned error can't find the container with id 9c7753f7ec4c6402e183b2c2398c0908b2307ac84104918ed6d488c1fac1b2da
Jul 27 23:27:59.374544 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:27:59.374511    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rv9qd" event={"ID":"69a56444-afee-4c94-8589-e8e0acaf2c3b","Type":"ContainerStarted","Data":"9c7753f7ec4c6402e183b2c2398c0908b2307ac84104918ed6d488c1fac1b2da"}
Jul 27 23:28:02.392400 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:02.392351    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rv9qd" event={"ID":"69a56444-afee-4c94-8589-e8e0acaf2c3b","Type":"ContainerStarted","Data":"a27a7901c1f45de9896ca07664657a95b8c48edb12447c4fe6554abbb2e1c519"}
Jul 27 23:28:02.405354 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:02.405312    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-rv9qd" podStartSLOduration=1.32016535 podStartE2EDuration="4.40529732s" totalImagesPullingTime="3.08513197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:28:02.404008947 +0000 UTC m=+705.348239692" watchObservedRunningTime="2026-07-27 23:28:02.40529732 +0000 UTC m=+705.349528064"
Jul 27 23:28:02.865599 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:02.865566    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rv9qd"]
Jul 27 23:28:04.401646 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:04.401606    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-rv9qd" podUID="69a56444-afee-4c94-8589-e8e0acaf2c3b" containerName="authorino" containerID="cri-o://a27a7901c1f45de9896ca07664657a95b8c48edb12447c4fe6554abbb2e1c519" gracePeriod=30
Jul 27 23:28:04.645268 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:04.645243    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rv9qd"
Jul 27 23:28:04.809765 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:04.809742    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdd8k\" (UniqueName: \"kubernetes.io/projected/69a56444-afee-4c94-8589-e8e0acaf2c3b-kube-api-access-rdd8k\") pod \"69a56444-afee-4c94-8589-e8e0acaf2c3b\" (UID: \"69a56444-afee-4c94-8589-e8e0acaf2c3b\") "
Jul 27 23:28:04.811956 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:04.811933    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a56444-afee-4c94-8589-e8e0acaf2c3b-kube-api-access-rdd8k" (OuterVolumeSpecName: "kube-api-access-rdd8k") pod "69a56444-afee-4c94-8589-e8e0acaf2c3b" (UID: "69a56444-afee-4c94-8589-e8e0acaf2c3b"). InnerVolumeSpecName "kube-api-access-rdd8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:28:04.910480 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:04.910455    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdd8k\" (UniqueName: \"kubernetes.io/projected/69a56444-afee-4c94-8589-e8e0acaf2c3b-kube-api-access-rdd8k\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:28:05.408023 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:05.407990    2559 generic.go:356] "Generic (PLEG): container finished" podID="69a56444-afee-4c94-8589-e8e0acaf2c3b" containerID="a27a7901c1f45de9896ca07664657a95b8c48edb12447c4fe6554abbb2e1c519" exitCode=0
Jul 27 23:28:05.408417 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:05.408038    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rv9qd"
Jul 27 23:28:05.408417 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:05.408078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rv9qd" event={"ID":"69a56444-afee-4c94-8589-e8e0acaf2c3b","Type":"ContainerDied","Data":"a27a7901c1f45de9896ca07664657a95b8c48edb12447c4fe6554abbb2e1c519"}
Jul 27 23:28:05.408417 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:05.408120    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rv9qd" event={"ID":"69a56444-afee-4c94-8589-e8e0acaf2c3b","Type":"ContainerDied","Data":"9c7753f7ec4c6402e183b2c2398c0908b2307ac84104918ed6d488c1fac1b2da"}
Jul 27 23:28:05.408417 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:05.408138    2559 scope.go:117] "RemoveContainer" containerID="a27a7901c1f45de9896ca07664657a95b8c48edb12447c4fe6554abbb2e1c519"
Jul 27 23:28:05.418889 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:05.418867    2559 scope.go:117] "RemoveContainer" containerID="a27a7901c1f45de9896ca07664657a95b8c48edb12447c4fe6554abbb2e1c519"
Jul 27 23:28:05.419333 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:28:05.419313    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27a7901c1f45de9896ca07664657a95b8c48edb12447c4fe6554abbb2e1c519\": container with ID starting with a27a7901c1f45de9896ca07664657a95b8c48edb12447c4fe6554abbb2e1c519 not found: ID does not exist" containerID="a27a7901c1f45de9896ca07664657a95b8c48edb12447c4fe6554abbb2e1c519"
Jul 27 23:28:05.419460 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:05.419353    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27a7901c1f45de9896ca07664657a95b8c48edb12447c4fe6554abbb2e1c519"} err="failed to get container status \"a27a7901c1f45de9896ca07664657a95b8c48edb12447c4fe6554abbb2e1c519\": rpc error: code = NotFound desc = could not find container \"a27a7901c1f45de9896ca07664657a95b8c48edb12447c4fe6554abbb2e1c519\": container with ID starting with a27a7901c1f45de9896ca07664657a95b8c48edb12447c4fe6554abbb2e1c519 not found: ID does not exist"
Jul 27 23:28:05.445391 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:05.427658    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rv9qd"]
Jul 27 23:28:05.450256 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:05.450233    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-rv9qd"]
Jul 27 23:28:05.562242 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:05.562216    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a56444-afee-4c94-8589-e8e0acaf2c3b" path="/var/lib/kubelet/pods/69a56444-afee-4c94-8589-e8e0acaf2c3b/volumes"
Jul 27 23:28:18.875479 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:18.875445    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj"]
Jul 27 23:28:18.875973 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:18.875796    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a56444-afee-4c94-8589-e8e0acaf2c3b" containerName="authorino"
Jul 27 23:28:18.875973 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:18.875806    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a56444-afee-4c94-8589-e8e0acaf2c3b" containerName="authorino"
Jul 27 23:28:18.875973 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:18.875864    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a56444-afee-4c94-8589-e8e0acaf2c3b" containerName="authorino"
Jul 27 23:28:18.880430 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:18.880408    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj"
Jul 27 23:28:18.882672 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:18.882649    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 27 23:28:18.882809 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:18.882653    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 23:28:18.882809 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:18.882754    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bfp8l\""
Jul 27 23:28:18.882927 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:18.882830    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 23:28:18.887954 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:18.887933    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj"]
Jul 27 23:28:18.931449 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:18.931420    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a0a555a0-6df4-448d-9a64-4d5aae82b5f4-data\") pod \"postgres-69fb4594cf-k7xmj\" (UID: \"a0a555a0-6df4-448d-9a64-4d5aae82b5f4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj"
Jul 27 23:28:18.931568 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:18.931476    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m8dg\" (UniqueName: \"kubernetes.io/projected/a0a555a0-6df4-448d-9a64-4d5aae82b5f4-kube-api-access-7m8dg\") pod \"postgres-69fb4594cf-k7xmj\" (UID: \"a0a555a0-6df4-448d-9a64-4d5aae82b5f4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj"
Jul 27 23:28:19.032728 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:19.032682    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m8dg\" (UniqueName: \"kubernetes.io/projected/a0a555a0-6df4-448d-9a64-4d5aae82b5f4-kube-api-access-7m8dg\") pod \"postgres-69fb4594cf-k7xmj\" (UID: \"a0a555a0-6df4-448d-9a64-4d5aae82b5f4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj"
Jul 27 23:28:19.032880 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:19.032807    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a0a555a0-6df4-448d-9a64-4d5aae82b5f4-data\") pod \"postgres-69fb4594cf-k7xmj\" (UID: \"a0a555a0-6df4-448d-9a64-4d5aae82b5f4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj"
Jul 27 23:28:19.033183 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:19.033164    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a0a555a0-6df4-448d-9a64-4d5aae82b5f4-data\") pod \"postgres-69fb4594cf-k7xmj\" (UID: \"a0a555a0-6df4-448d-9a64-4d5aae82b5f4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj"
Jul 27 23:28:19.041580 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:19.041557    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m8dg\" (UniqueName: \"kubernetes.io/projected/a0a555a0-6df4-448d-9a64-4d5aae82b5f4-kube-api-access-7m8dg\") pod \"postgres-69fb4594cf-k7xmj\" (UID: \"a0a555a0-6df4-448d-9a64-4d5aae82b5f4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj"
Jul 27 23:28:19.194919 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:19.194846    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj"
Jul 27 23:28:19.324945 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:19.324824    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj"]
Jul 27 23:28:19.327554 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:28:19.327523    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a555a0_6df4_448d_9a64_4d5aae82b5f4.slice/crio-fd0360bbe740cbb0b00a2cbf922127c2f041fa038aee3766057680a4e670e62d WatchSource:0}: Error finding container fd0360bbe740cbb0b00a2cbf922127c2f041fa038aee3766057680a4e670e62d: Status 404 returned error can't find the container with id fd0360bbe740cbb0b00a2cbf922127c2f041fa038aee3766057680a4e670e62d
Jul 27 23:28:19.473600 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:19.473519    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj" event={"ID":"a0a555a0-6df4-448d-9a64-4d5aae82b5f4","Type":"ContainerStarted","Data":"fd0360bbe740cbb0b00a2cbf922127c2f041fa038aee3766057680a4e670e62d"}
Jul 27 23:28:24.504943 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:24.504902    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj" event={"ID":"a0a555a0-6df4-448d-9a64-4d5aae82b5f4","Type":"ContainerStarted","Data":"7a62739419ddfbf529bab767c7eb218d7199eb393f83b08572904d27e44bf4f9"}
Jul 27 23:28:24.505354 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:24.505039    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj"
Jul 27 23:28:24.520458 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:24.520415    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj" podStartSLOduration=1.825648831 podStartE2EDuration="6.520398404s" totalImagesPullingTime="4.694749573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:28:24.519773971 +0000 UTC m=+727.464004718" watchObservedRunningTime="2026-07-27 23:28:24.520398404 +0000 UTC m=+727.464629151"
Jul 27 23:28:30.537854 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:30.537824    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k7xmj"
Jul 27 23:28:31.045854 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.045819    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hrvlv"]
Jul 27 23:28:31.052321 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.052300    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hrvlv"
Jul 27 23:28:31.054435 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.054413    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nx6j8\""
Jul 27 23:28:31.055445 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.055421    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hrvlv"]
Jul 27 23:28:31.139648 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.139612    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8c6h\" (UniqueName: \"kubernetes.io/projected/0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97-kube-api-access-m8c6h\") pod \"authorino-5d4b67c446-hrvlv\" (UID: \"0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97\") " pod="kuadrant-system/authorino-5d4b67c446-hrvlv"
Jul 27 23:28:31.240583 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.240550    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8c6h\" (UniqueName: \"kubernetes.io/projected/0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97-kube-api-access-m8c6h\") pod \"authorino-5d4b67c446-hrvlv\" (UID: \"0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97\") " pod="kuadrant-system/authorino-5d4b67c446-hrvlv"
Jul 27 23:28:31.251523 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.251495    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8c6h\" (UniqueName: \"kubernetes.io/projected/0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97-kube-api-access-m8c6h\") pod \"authorino-5d4b67c446-hrvlv\" (UID: \"0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97\") " pod="kuadrant-system/authorino-5d4b67c446-hrvlv"
Jul 27 23:28:31.271758 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.271726    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hrvlv"]
Jul 27 23:28:31.272008 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.271997    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hrvlv"
Jul 27 23:28:31.297069 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.297003    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57dd84b75b-bh5gn"]
Jul 27 23:28:31.302606 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.302582    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57dd84b75b-bh5gn"
Jul 27 23:28:31.306249 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.306224    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57dd84b75b-bh5gn"]
Jul 27 23:28:31.416162 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.416137    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hrvlv"]
Jul 27 23:28:31.418623 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:28:31.418589    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0201c0d6_a8cd_4ec3_ac6c_9bdaefc27a97.slice/crio-f2906ef03826dd03620ab2114984ecaad3a66327cd40f95ebf98769335e9729d WatchSource:0}: Error finding container f2906ef03826dd03620ab2114984ecaad3a66327cd40f95ebf98769335e9729d: Status 404 returned error can't find the container with id f2906ef03826dd03620ab2114984ecaad3a66327cd40f95ebf98769335e9729d
Jul 27 23:28:31.442510 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.442483    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxg7k\" (UniqueName: \"kubernetes.io/projected/703ccbc9-a7a4-4b2e-93ba-3a586d43a108-kube-api-access-nxg7k\") pod \"authorino-57dd84b75b-bh5gn\" (UID: \"703ccbc9-a7a4-4b2e-93ba-3a586d43a108\") " pod="kuadrant-system/authorino-57dd84b75b-bh5gn"
Jul 27 23:28:31.537909 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.537872    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-hrvlv" event={"ID":"0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97","Type":"ContainerStarted","Data":"f2906ef03826dd03620ab2114984ecaad3a66327cd40f95ebf98769335e9729d"}
Jul 27 23:28:31.543363 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.543336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxg7k\" (UniqueName: \"kubernetes.io/projected/703ccbc9-a7a4-4b2e-93ba-3a586d43a108-kube-api-access-nxg7k\") pod \"authorino-57dd84b75b-bh5gn\" (UID: \"703ccbc9-a7a4-4b2e-93ba-3a586d43a108\") " pod="kuadrant-system/authorino-57dd84b75b-bh5gn"
Jul 27 23:28:31.551172 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.551114    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxg7k\" (UniqueName: \"kubernetes.io/projected/703ccbc9-a7a4-4b2e-93ba-3a586d43a108-kube-api-access-nxg7k\") pod \"authorino-57dd84b75b-bh5gn\" (UID: \"703ccbc9-a7a4-4b2e-93ba-3a586d43a108\") " pod="kuadrant-system/authorino-57dd84b75b-bh5gn"
Jul 27 23:28:31.614186 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.614159    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57dd84b75b-bh5gn"
Jul 27 23:28:31.680082 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.680050    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57dd84b75b-bh5gn"]
Jul 27 23:28:31.706834 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.706806    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cdb5477c6-6b9j5"]
Jul 27 23:28:31.714068 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.714046    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cdb5477c6-6b9j5"
Jul 27 23:28:31.716256 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.716231    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 27 23:28:31.717616 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.717334    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cdb5477c6-6b9j5"]
Jul 27 23:28:31.761443 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.761417    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57dd84b75b-bh5gn"]
Jul 27 23:28:31.765570 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:28:31.765529    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod703ccbc9_a7a4_4b2e_93ba_3a586d43a108.slice/crio-1a12fe3d775413571abf78cf16412c9a51c1678e8728d8f26ddeb040987e7183 WatchSource:0}: Error finding container 1a12fe3d775413571abf78cf16412c9a51c1678e8728d8f26ddeb040987e7183: Status 404 returned error can't find the container with id 1a12fe3d775413571abf78cf16412c9a51c1678e8728d8f26ddeb040987e7183
Jul 27 23:28:31.846914 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.846833    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aa3f98b0-3bd2-4e67-89d2-ebfde735ae85-tls-cert\") pod \"authorino-7cdb5477c6-6b9j5\" (UID: \"aa3f98b0-3bd2-4e67-89d2-ebfde735ae85\") " pod="kuadrant-system/authorino-7cdb5477c6-6b9j5"
Jul 27 23:28:31.846914 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.846886    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p54w8\" (UniqueName: \"kubernetes.io/projected/aa3f98b0-3bd2-4e67-89d2-ebfde735ae85-kube-api-access-p54w8\") pod \"authorino-7cdb5477c6-6b9j5\" (UID: \"aa3f98b0-3bd2-4e67-89d2-ebfde735ae85\") " pod="kuadrant-system/authorino-7cdb5477c6-6b9j5"
Jul 27 23:28:31.947672 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.947649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aa3f98b0-3bd2-4e67-89d2-ebfde735ae85-tls-cert\") pod \"authorino-7cdb5477c6-6b9j5\" (UID: \"aa3f98b0-3bd2-4e67-89d2-ebfde735ae85\") " pod="kuadrant-system/authorino-7cdb5477c6-6b9j5"
Jul 27 23:28:31.947814 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.947683    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p54w8\" (UniqueName: \"kubernetes.io/projected/aa3f98b0-3bd2-4e67-89d2-ebfde735ae85-kube-api-access-p54w8\") pod \"authorino-7cdb5477c6-6b9j5\" (UID: \"aa3f98b0-3bd2-4e67-89d2-ebfde735ae85\") " pod="kuadrant-system/authorino-7cdb5477c6-6b9j5"
Jul 27 23:28:31.950980 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.950954    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aa3f98b0-3bd2-4e67-89d2-ebfde735ae85-tls-cert\") pod \"authorino-7cdb5477c6-6b9j5\" (UID: \"aa3f98b0-3bd2-4e67-89d2-ebfde735ae85\") " pod="kuadrant-system/authorino-7cdb5477c6-6b9j5"
Jul 27 23:28:31.955104 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:31.955081    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p54w8\" (UniqueName: \"kubernetes.io/projected/aa3f98b0-3bd2-4e67-89d2-ebfde735ae85-kube-api-access-p54w8\") pod \"authorino-7cdb5477c6-6b9j5\" (UID: \"aa3f98b0-3bd2-4e67-89d2-ebfde735ae85\") " pod="kuadrant-system/authorino-7cdb5477c6-6b9j5"
Jul 27 23:28:32.027130 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.027090    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cdb5477c6-6b9j5"
Jul 27 23:28:32.173952 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.173927    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cdb5477c6-6b9j5"]
Jul 27 23:28:32.175719 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:28:32.175696    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa3f98b0_3bd2_4e67_89d2_ebfde735ae85.slice/crio-0a8f1448dc17a4cf2dc88de5201d44c6b53be35f8bb07a72c7fae4bc0bfe48ab WatchSource:0}: Error finding container 0a8f1448dc17a4cf2dc88de5201d44c6b53be35f8bb07a72c7fae4bc0bfe48ab: Status 404 returned error can't find the container with id 0a8f1448dc17a4cf2dc88de5201d44c6b53be35f8bb07a72c7fae4bc0bfe48ab
Jul 27 23:28:32.543951 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.543911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cdb5477c6-6b9j5" event={"ID":"aa3f98b0-3bd2-4e67-89d2-ebfde735ae85","Type":"ContainerStarted","Data":"0a8f1448dc17a4cf2dc88de5201d44c6b53be35f8bb07a72c7fae4bc0bfe48ab"}
Jul 27 23:28:32.545334 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.545307    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57dd84b75b-bh5gn" event={"ID":"703ccbc9-a7a4-4b2e-93ba-3a586d43a108","Type":"ContainerStarted","Data":"1ebe298fde4b198c89860719d6195dddd53e75041c4bd276901c16c2fac949b6"}
Jul 27 23:28:32.545484 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.545347    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57dd84b75b-bh5gn" event={"ID":"703ccbc9-a7a4-4b2e-93ba-3a586d43a108","Type":"ContainerStarted","Data":"1a12fe3d775413571abf78cf16412c9a51c1678e8728d8f26ddeb040987e7183"}
Jul 27 23:28:32.545484 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.545345    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-57dd84b75b-bh5gn" podUID="703ccbc9-a7a4-4b2e-93ba-3a586d43a108" containerName="authorino" containerID="cri-o://1ebe298fde4b198c89860719d6195dddd53e75041c4bd276901c16c2fac949b6" gracePeriod=30
Jul 27 23:28:32.546901 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.546874    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-hrvlv" event={"ID":"0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97","Type":"ContainerStarted","Data":"d218210bec163f5cce59de1b812bdf2527dc7b290131cf182ce74b05e9250c27"}
Jul 27 23:28:32.547002 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.546959    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-hrvlv" podUID="0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97" containerName="authorino" containerID="cri-o://d218210bec163f5cce59de1b812bdf2527dc7b290131cf182ce74b05e9250c27" gracePeriod=30
Jul 27 23:28:32.560233 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.560189    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-57dd84b75b-bh5gn" podStartSLOduration=1.18594683 podStartE2EDuration="1.560173297s" totalImagesPullingTime="374.226467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:28:32.557857899 +0000 UTC m=+735.502088646" watchObservedRunningTime="2026-07-27 23:28:32.560173297 +0000 UTC m=+735.504404062"
Jul 27 23:28:32.570927 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.570887    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-hrvlv" podStartSLOduration=1.102708746 podStartE2EDuration="1.570872794s" totalImagesPullingTime="468.164048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:28:32.570011971 +0000 UTC m=+735.514242718" watchObservedRunningTime="2026-07-27 23:28:32.570872794 +0000 UTC m=+735.515103540"
Jul 27 23:28:32.857986 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.857946    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57dd84b75b-bh5gn"
Jul 27 23:28:32.862981 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.862959    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hrvlv"
Jul 27 23:28:32.956286 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.956254    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxg7k\" (UniqueName: \"kubernetes.io/projected/703ccbc9-a7a4-4b2e-93ba-3a586d43a108-kube-api-access-nxg7k\") pod \"703ccbc9-a7a4-4b2e-93ba-3a586d43a108\" (UID: \"703ccbc9-a7a4-4b2e-93ba-3a586d43a108\") "
Jul 27 23:28:32.956476 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.956295    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8c6h\" (UniqueName: \"kubernetes.io/projected/0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97-kube-api-access-m8c6h\") pod \"0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97\" (UID: \"0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97\") "
Jul 27 23:28:32.958496 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.958464    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97-kube-api-access-m8c6h" (OuterVolumeSpecName: "kube-api-access-m8c6h") pod "0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97" (UID: "0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97"). InnerVolumeSpecName "kube-api-access-m8c6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:28:32.958608 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:32.958495    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703ccbc9-a7a4-4b2e-93ba-3a586d43a108-kube-api-access-nxg7k" (OuterVolumeSpecName: "kube-api-access-nxg7k") pod "703ccbc9-a7a4-4b2e-93ba-3a586d43a108" (UID: "703ccbc9-a7a4-4b2e-93ba-3a586d43a108"). InnerVolumeSpecName "kube-api-access-nxg7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:28:33.057749 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.057717    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxg7k\" (UniqueName: \"kubernetes.io/projected/703ccbc9-a7a4-4b2e-93ba-3a586d43a108-kube-api-access-nxg7k\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:28:33.057749 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.057743    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8c6h\" (UniqueName: \"kubernetes.io/projected/0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97-kube-api-access-m8c6h\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:28:33.552876 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.552841    2559 generic.go:356] "Generic (PLEG): container finished" podID="703ccbc9-a7a4-4b2e-93ba-3a586d43a108" containerID="1ebe298fde4b198c89860719d6195dddd53e75041c4bd276901c16c2fac949b6" exitCode=0
Jul 27 23:28:33.553344 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.552904    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57dd84b75b-bh5gn"
Jul 27 23:28:33.553344 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.552933    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57dd84b75b-bh5gn" event={"ID":"703ccbc9-a7a4-4b2e-93ba-3a586d43a108","Type":"ContainerDied","Data":"1ebe298fde4b198c89860719d6195dddd53e75041c4bd276901c16c2fac949b6"}
Jul 27 23:28:33.553344 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.552981    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57dd84b75b-bh5gn" event={"ID":"703ccbc9-a7a4-4b2e-93ba-3a586d43a108","Type":"ContainerDied","Data":"1a12fe3d775413571abf78cf16412c9a51c1678e8728d8f26ddeb040987e7183"}
Jul 27 23:28:33.553344 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.553003    2559 scope.go:117] "RemoveContainer" containerID="1ebe298fde4b198c89860719d6195dddd53e75041c4bd276901c16c2fac949b6"
Jul 27 23:28:33.554234 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.554156    2559 generic.go:356] "Generic (PLEG): container finished" podID="0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97" containerID="d218210bec163f5cce59de1b812bdf2527dc7b290131cf182ce74b05e9250c27" exitCode=0
Jul 27 23:28:33.554234 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.554216    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hrvlv"
Jul 27 23:28:33.554335 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.554234    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-hrvlv" event={"ID":"0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97","Type":"ContainerDied","Data":"d218210bec163f5cce59de1b812bdf2527dc7b290131cf182ce74b05e9250c27"}
Jul 27 23:28:33.554335 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.554272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-hrvlv" event={"ID":"0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97","Type":"ContainerDied","Data":"f2906ef03826dd03620ab2114984ecaad3a66327cd40f95ebf98769335e9729d"}
Jul 27 23:28:33.556064 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.556040    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cdb5477c6-6b9j5" event={"ID":"aa3f98b0-3bd2-4e67-89d2-ebfde735ae85","Type":"ContainerStarted","Data":"8887073d6b82f77815fd74f83a94db8a2628cfbdb13e83e380ae6cc7b7d05d3e"}
Jul 27 23:28:33.562963 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.562905    2559 scope.go:117] "RemoveContainer" containerID="1ebe298fde4b198c89860719d6195dddd53e75041c4bd276901c16c2fac949b6"
Jul 27 23:28:33.563345 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:28:33.563190    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ebe298fde4b198c89860719d6195dddd53e75041c4bd276901c16c2fac949b6\": container with ID starting with 1ebe298fde4b198c89860719d6195dddd53e75041c4bd276901c16c2fac949b6 not found: ID does not exist" containerID="1ebe298fde4b198c89860719d6195dddd53e75041c4bd276901c16c2fac949b6"
Jul 27 23:28:33.563345 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.563236    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ebe298fde4b198c89860719d6195dddd53e75041c4bd276901c16c2fac949b6"} err="failed to get container status \"1ebe298fde4b198c89860719d6195dddd53e75041c4bd276901c16c2fac949b6\": rpc error: code = NotFound desc = could not find container \"1ebe298fde4b198c89860719d6195dddd53e75041c4bd276901c16c2fac949b6\": container with ID starting with 1ebe298fde4b198c89860719d6195dddd53e75041c4bd276901c16c2fac949b6 not found: ID does not exist"
Jul 27 23:28:33.563345 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.563263    2559 scope.go:117] "RemoveContainer" containerID="d218210bec163f5cce59de1b812bdf2527dc7b290131cf182ce74b05e9250c27"
Jul 27 23:28:33.572114 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.572088    2559 scope.go:117] "RemoveContainer" containerID="d218210bec163f5cce59de1b812bdf2527dc7b290131cf182ce74b05e9250c27"
Jul 27 23:28:33.572452 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:28:33.572430    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d218210bec163f5cce59de1b812bdf2527dc7b290131cf182ce74b05e9250c27\": container with ID starting with d218210bec163f5cce59de1b812bdf2527dc7b290131cf182ce74b05e9250c27 not found: ID does not exist" containerID="d218210bec163f5cce59de1b812bdf2527dc7b290131cf182ce74b05e9250c27"
Jul 27 23:28:33.572514 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.572463    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d218210bec163f5cce59de1b812bdf2527dc7b290131cf182ce74b05e9250c27"} err="failed to get container status \"d218210bec163f5cce59de1b812bdf2527dc7b290131cf182ce74b05e9250c27\": rpc error: code = NotFound desc = could not find container \"d218210bec163f5cce59de1b812bdf2527dc7b290131cf182ce74b05e9250c27\": container with ID starting with d218210bec163f5cce59de1b812bdf2527dc7b290131cf182ce74b05e9250c27 not found: ID does not exist"
Jul 27 23:28:33.572718 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.572685    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cdb5477c6-6b9j5" podStartSLOduration=2.074841738 podStartE2EDuration="2.572673332s" totalImagesPullingTime="497.831594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:28:33.571303574 +0000 UTC m=+736.515534415" watchObservedRunningTime="2026-07-27 23:28:33.572673332 +0000 UTC m=+736.516904078"
Jul 27 23:28:33.585989 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.585961    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hrvlv"]
Jul 27 23:28:33.590196 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.590172    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hrvlv"]
Jul 27 23:28:33.604954 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.604923    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57dd84b75b-bh5gn"]
Jul 27 23:28:33.611670 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:33.611641    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57dd84b75b-bh5gn"]
Jul 27 23:28:35.563384 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:35.563347    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97" path="/var/lib/kubelet/pods/0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97/volumes"
Jul 27 23:28:35.563979 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:28:35.563838    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="703ccbc9-a7a4-4b2e-93ba-3a586d43a108" path="/var/lib/kubelet/pods/703ccbc9-a7a4-4b2e-93ba-3a586d43a108/volumes"
Jul 27 23:29:20.785532 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:20.785499    2559 generic.go:356] "Generic (PLEG): container finished" podID="d706cd9a-be1b-4544-9740-12f63c54c73e" containerID="6e841d9245b55c7889005b76ccf5830c0c7c56960a9947041cf236df61f6461f" exitCode=0
Jul 27 23:29:20.786053 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:20.785576    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn" event={"ID":"d706cd9a-be1b-4544-9740-12f63c54c73e","Type":"ContainerDied","Data":"6e841d9245b55c7889005b76ccf5830c0c7c56960a9947041cf236df61f6461f"}
Jul 27 23:29:20.786053 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:20.786049    2559 scope.go:117] "RemoveContainer" containerID="6e841d9245b55c7889005b76ccf5830c0c7c56960a9947041cf236df61f6461f"
Jul 27 23:29:20.787207 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:20.787178    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:29:20.787322 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:20.787252    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:29:20.787322 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:20.787293    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:29:21.794220 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:21.794179    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn" event={"ID":"d706cd9a-be1b-4544-9740-12f63c54c73e","Type":"ContainerStarted","Data":"37d3722d5d58e3b19ce68ea35143639cb22257204d84966b9bc099858191cd0d"}
Jul 27 23:29:21.831847 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:21.831811    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:29:21.832035 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:21.831996    2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hbnzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 27 23:29:21.832078 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:21.832042    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn" podUID="d706cd9a-be1b-4544-9740-12f63c54c73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 27 23:29:22.831925 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:22.831877    2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hbnzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 27 23:29:22.832289 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:22.831945    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn" podUID="d706cd9a-be1b-4544-9740-12f63c54c73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 27 23:29:23.831597 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:23.831566    2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hbnzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 27 23:29:23.831756 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:23.831623    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn" podUID="d706cd9a-be1b-4544-9740-12f63c54c73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 27 23:29:24.831300 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:24.831258    2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hbnzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 27 23:29:24.831726 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:24.831328    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn" podUID="d706cd9a-be1b-4544-9740-12f63c54c73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 27 23:29:25.835992 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:25.835963    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:29:25.836471 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:25.836325    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:29:25.837208 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:25.837181    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hbnzn"
Jul 27 23:29:51.739797 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.739720    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"]
Jul 27 23:29:51.740257 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.740117    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97" containerName="authorino"
Jul 27 23:29:51.740257 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.740135    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97" containerName="authorino"
Jul 27 23:29:51.740257 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.740153    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="703ccbc9-a7a4-4b2e-93ba-3a586d43a108" containerName="authorino"
Jul 27 23:29:51.740257 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.740158    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="703ccbc9-a7a4-4b2e-93ba-3a586d43a108" containerName="authorino"
Jul 27 23:29:51.740257 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.740230    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0201c0d6-a8cd-4ec3-ac6c-9bdaefc27a97" containerName="authorino"
Jul 27 23:29:51.740257 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.740240    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="703ccbc9-a7a4-4b2e-93ba-3a586d43a108" containerName="authorino"
Jul 27 23:29:51.743249 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.743227    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.745671 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.745650    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 23:29:51.745792 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.745705    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 23:29:51.746305 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.746286    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 27 23:29:51.746416 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.746288    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dz654\""
Jul 27 23:29:51.753580 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.753559    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"]
Jul 27 23:29:51.852682 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.852631    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zgm9\" (UniqueName: \"kubernetes.io/projected/6224811f-02b3-4951-8227-fc2329e9971a-kube-api-access-4zgm9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.852682 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.852685    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6224811f-02b3-4951-8227-fc2329e9971a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.852898 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.852712    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6224811f-02b3-4951-8227-fc2329e9971a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.852898 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.852755    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6224811f-02b3-4951-8227-fc2329e9971a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.852898 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.852812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6224811f-02b3-4951-8227-fc2329e9971a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.852898 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.852860    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6224811f-02b3-4951-8227-fc2329e9971a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.954324 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.954286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zgm9\" (UniqueName: \"kubernetes.io/projected/6224811f-02b3-4951-8227-fc2329e9971a-kube-api-access-4zgm9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.954324 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.954330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6224811f-02b3-4951-8227-fc2329e9971a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.954588 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.954354    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6224811f-02b3-4951-8227-fc2329e9971a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.954588 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.954402    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6224811f-02b3-4951-8227-fc2329e9971a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.954588 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.954426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6224811f-02b3-4951-8227-fc2329e9971a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.954588 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.954462    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6224811f-02b3-4951-8227-fc2329e9971a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.954794 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.954735    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6224811f-02b3-4951-8227-fc2329e9971a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.954794 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.954760    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6224811f-02b3-4951-8227-fc2329e9971a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.954906 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.954844    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6224811f-02b3-4951-8227-fc2329e9971a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.956977 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.956949    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6224811f-02b3-4951-8227-fc2329e9971a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.957103 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.957027    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6224811f-02b3-4951-8227-fc2329e9971a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:51.962219 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:51.962199    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zgm9\" (UniqueName: \"kubernetes.io/projected/6224811f-02b3-4951-8227-fc2329e9971a-kube-api-access-4zgm9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glx4q\" (UID: \"6224811f-02b3-4951-8227-fc2329e9971a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:52.055289 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:52.055257    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:29:52.194266 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:52.194219    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"]
Jul 27 23:29:52.196678 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:29:52.196648    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6224811f_02b3_4951_8227_fc2329e9971a.slice/crio-93c1c8e939b2986b006e7b37fcc3eea2ff363929f934bdd48af3123f9a0780bf WatchSource:0}: Error finding container 93c1c8e939b2986b006e7b37fcc3eea2ff363929f934bdd48af3123f9a0780bf: Status 404 returned error can't find the container with id 93c1c8e939b2986b006e7b37fcc3eea2ff363929f934bdd48af3123f9a0780bf
Jul 27 23:29:52.949168 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:52.949128    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q" event={"ID":"6224811f-02b3-4951-8227-fc2329e9971a","Type":"ContainerStarted","Data":"93c1c8e939b2986b006e7b37fcc3eea2ff363929f934bdd48af3123f9a0780bf"}
Jul 27 23:29:53.955679 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:53.955640    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q" event={"ID":"6224811f-02b3-4951-8227-fc2329e9971a","Type":"ContainerStarted","Data":"87e2a11bdeaa453709ebf1164323def2a6c2cfaa631f9d26571c336a9e852fa4"}
Jul 27 23:29:53.972422 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:29:53.972352    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q" podStartSLOduration=1.826900725 podStartE2EDuration="2.972338205s" totalImagesPullingTime="1.14543748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:29:53.971001045 +0000 UTC m=+816.915231792" watchObservedRunningTime="2026-07-27 23:29:53.972338205 +0000 UTC m=+816.916568949"
Jul 27 23:30:02.055512 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:02.055451    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:30:02.056027 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:02.055551    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:30:02.067768 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:02.067743    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:30:03.010617 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:03.010589    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glx4q"
Jul 27 23:30:10.432790 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.432755    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"]
Jul 27 23:30:10.438830 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.438792    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.440760 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.440739    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 27 23:30:10.446830 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.446726    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"]
Jul 27 23:30:10.532366 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.532335    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/419fad95-9938-4b45-aa29-3081ef8a8b4e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.532563 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.532428    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc9gf\" (UniqueName: \"kubernetes.io/projected/419fad95-9938-4b45-aa29-3081ef8a8b4e-kube-api-access-zc9gf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.532563 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.532478    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/419fad95-9938-4b45-aa29-3081ef8a8b4e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.532563 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.532512    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/419fad95-9938-4b45-aa29-3081ef8a8b4e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.532699 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.532564    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/419fad95-9938-4b45-aa29-3081ef8a8b4e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.532699 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.532595    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419fad95-9938-4b45-aa29-3081ef8a8b4e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.634063 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.634021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/419fad95-9938-4b45-aa29-3081ef8a8b4e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.634063 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.634055    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc9gf\" (UniqueName: \"kubernetes.io/projected/419fad95-9938-4b45-aa29-3081ef8a8b4e-kube-api-access-zc9gf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.634319 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.634095    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/419fad95-9938-4b45-aa29-3081ef8a8b4e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.634319 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.634153    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/419fad95-9938-4b45-aa29-3081ef8a8b4e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.634319 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.634182    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/419fad95-9938-4b45-aa29-3081ef8a8b4e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.634319 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.634206    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419fad95-9938-4b45-aa29-3081ef8a8b4e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.634588 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.634564    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/419fad95-9938-4b45-aa29-3081ef8a8b4e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.634635 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.634614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/419fad95-9938-4b45-aa29-3081ef8a8b4e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.634705 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.634686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419fad95-9938-4b45-aa29-3081ef8a8b4e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.636758 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.636733    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/419fad95-9938-4b45-aa29-3081ef8a8b4e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.637028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.637007    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/419fad95-9938-4b45-aa29-3081ef8a8b4e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.641455 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.641432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc9gf\" (UniqueName: \"kubernetes.io/projected/419fad95-9938-4b45-aa29-3081ef8a8b4e-kube-api-access-zc9gf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s\" (UID: \"419fad95-9938-4b45-aa29-3081ef8a8b4e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.751042 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.750962    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:10.886038 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.886012    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"]
Jul 27 23:30:10.888261 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:30:10.888228    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419fad95_9938_4b45_aa29_3081ef8a8b4e.slice/crio-a67c90de0404564dfb7ad8de3bfbe1658cf8ee762fd961a1898e3b6c02a10a36 WatchSource:0}: Error finding container a67c90de0404564dfb7ad8de3bfbe1658cf8ee762fd961a1898e3b6c02a10a36: Status 404 returned error can't find the container with id a67c90de0404564dfb7ad8de3bfbe1658cf8ee762fd961a1898e3b6c02a10a36
Jul 27 23:30:10.889852 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:10.889833    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 23:30:11.037997 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:11.037945    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s" event={"ID":"419fad95-9938-4b45-aa29-3081ef8a8b4e","Type":"ContainerStarted","Data":"a67c90de0404564dfb7ad8de3bfbe1658cf8ee762fd961a1898e3b6c02a10a36"}
Jul 27 23:30:12.045575 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:12.045451    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s" event={"ID":"419fad95-9938-4b45-aa29-3081ef8a8b4e","Type":"ContainerStarted","Data":"f89269372b06df1221f3a2951eedbcb645dcb46ba197f1bc7a09380fc7faeb9f"}
Jul 27 23:30:12.061817 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:12.061767    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s" podStartSLOduration=1.876743735 podStartE2EDuration="2.061751007s" totalImagesPullingTime="185.007272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:30:12.060567464 +0000 UTC m=+835.004798210" watchObservedRunningTime="2026-07-27 23:30:12.061751007 +0000 UTC m=+835.005981752"
Jul 27 23:30:20.751741 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:20.751710    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:20.752229 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:20.751782    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:20.764015 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:20.763990    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:21.103386 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:21.103340    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s"
Jul 27 23:30:37.437718 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.437686    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"]
Jul 27 23:30:37.442459 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.442437    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.444835 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.444814    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 27 23:30:37.453906 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.453884    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"]
Jul 27 23:30:37.461261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.461238    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0a63841-fa2a-482b-a58f-1933106becbe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.461365 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.461281    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0a63841-fa2a-482b-a58f-1933106becbe-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.461365 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.461300    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0a63841-fa2a-482b-a58f-1933106becbe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.461495 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.461393    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0a63841-fa2a-482b-a58f-1933106becbe-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.461533 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.461493    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2djv2\" (UniqueName: \"kubernetes.io/projected/d0a63841-fa2a-482b-a58f-1933106becbe-kube-api-access-2djv2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.461577 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.461543    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0a63841-fa2a-482b-a58f-1933106becbe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.562552 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.562513    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0a63841-fa2a-482b-a58f-1933106becbe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.562745 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.562668    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0a63841-fa2a-482b-a58f-1933106becbe-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.562885 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.562863    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2djv2\" (UniqueName: \"kubernetes.io/projected/d0a63841-fa2a-482b-a58f-1933106becbe-kube-api-access-2djv2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.562945 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.562927    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0a63841-fa2a-482b-a58f-1933106becbe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.563006 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.562989    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0a63841-fa2a-482b-a58f-1933106becbe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.563061 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.563045    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0a63841-fa2a-482b-a58f-1933106becbe-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.563116 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.563065    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0a63841-fa2a-482b-a58f-1933106becbe-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.563410 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.563357    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0a63841-fa2a-482b-a58f-1933106becbe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.563662 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.563637    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0a63841-fa2a-482b-a58f-1933106becbe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.565231 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.565214    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0a63841-fa2a-482b-a58f-1933106becbe-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.565305 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.565286    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0a63841-fa2a-482b-a58f-1933106becbe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.570506 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.570485    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2djv2\" (UniqueName: \"kubernetes.io/projected/d0a63841-fa2a-482b-a58f-1933106becbe-kube-api-access-2djv2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t\" (UID: \"d0a63841-fa2a-482b-a58f-1933106becbe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.751280 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.751187    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:37.889600 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:37.889566    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"]
Jul 27 23:30:37.891079 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:30:37.891050    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0a63841_fa2a_482b_a58f_1933106becbe.slice/crio-6895a25636570b354034d09fe6747bfb62fffaa08875ad3829e81eb1671fc9f9 WatchSource:0}: Error finding container 6895a25636570b354034d09fe6747bfb62fffaa08875ad3829e81eb1671fc9f9: Status 404 returned error can't find the container with id 6895a25636570b354034d09fe6747bfb62fffaa08875ad3829e81eb1671fc9f9
Jul 27 23:30:38.173330 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:38.173296    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t" event={"ID":"d0a63841-fa2a-482b-a58f-1933106becbe","Type":"ContainerStarted","Data":"e4f7f0f9729137c7681359949ef1dcc8f99105edd812e2a02ed09cd800e308a5"}
Jul 27 23:30:38.173491 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:38.173345    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t" event={"ID":"d0a63841-fa2a-482b-a58f-1933106becbe","Type":"ContainerStarted","Data":"6895a25636570b354034d09fe6747bfb62fffaa08875ad3829e81eb1671fc9f9"}
Jul 27 23:30:38.189648 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:38.189604    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t" podStartSLOduration=0.99450349 podStartE2EDuration="1.189588673s" totalImagesPullingTime="195.085183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:30:38.188390411 +0000 UTC m=+861.132621160" watchObservedRunningTime="2026-07-27 23:30:38.189588673 +0000 UTC m=+861.133819418"
Jul 27 23:30:47.751947 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:47.751912    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:47.752392 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:47.752199    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:47.764523 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:47.764498    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:30:48.238225 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:30:48.238198    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t"
Jul 27 23:31:17.489167 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:17.489080    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:31:17.492010 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:17.491989    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:31:29.946329 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:29.944988    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cdb5477c6-6b9j5"]
Jul 27 23:31:29.946329 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:29.945239    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7cdb5477c6-6b9j5" podUID="aa3f98b0-3bd2-4e67-89d2-ebfde735ae85" containerName="authorino" containerID="cri-o://8887073d6b82f77815fd74f83a94db8a2628cfbdb13e83e380ae6cc7b7d05d3e" gracePeriod=30
Jul 27 23:31:30.209944 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.209878    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cdb5477c6-6b9j5"
Jul 27 23:31:30.239989 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.239959    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aa3f98b0-3bd2-4e67-89d2-ebfde735ae85-tls-cert\") pod \"aa3f98b0-3bd2-4e67-89d2-ebfde735ae85\" (UID: \"aa3f98b0-3bd2-4e67-89d2-ebfde735ae85\") "
Jul 27 23:31:30.240172 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.240024    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p54w8\" (UniqueName: \"kubernetes.io/projected/aa3f98b0-3bd2-4e67-89d2-ebfde735ae85-kube-api-access-p54w8\") pod \"aa3f98b0-3bd2-4e67-89d2-ebfde735ae85\" (UID: \"aa3f98b0-3bd2-4e67-89d2-ebfde735ae85\") "
Jul 27 23:31:30.242231 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.242198    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3f98b0-3bd2-4e67-89d2-ebfde735ae85-kube-api-access-p54w8" (OuterVolumeSpecName: "kube-api-access-p54w8") pod "aa3f98b0-3bd2-4e67-89d2-ebfde735ae85" (UID: "aa3f98b0-3bd2-4e67-89d2-ebfde735ae85"). InnerVolumeSpecName "kube-api-access-p54w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:31:30.250207 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.250179    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa3f98b0-3bd2-4e67-89d2-ebfde735ae85-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "aa3f98b0-3bd2-4e67-89d2-ebfde735ae85" (UID: "aa3f98b0-3bd2-4e67-89d2-ebfde735ae85"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:31:30.340710 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.340680    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p54w8\" (UniqueName: \"kubernetes.io/projected/aa3f98b0-3bd2-4e67-89d2-ebfde735ae85-kube-api-access-p54w8\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:31:30.340710 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.340708    2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aa3f98b0-3bd2-4e67-89d2-ebfde735ae85-tls-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:31:30.434210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.434167    2559 generic.go:356] "Generic (PLEG): container finished" podID="aa3f98b0-3bd2-4e67-89d2-ebfde735ae85" containerID="8887073d6b82f77815fd74f83a94db8a2628cfbdb13e83e380ae6cc7b7d05d3e" exitCode=0
Jul 27 23:31:30.434434 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.434257    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cdb5477c6-6b9j5"
Jul 27 23:31:30.434434 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.434265    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cdb5477c6-6b9j5" event={"ID":"aa3f98b0-3bd2-4e67-89d2-ebfde735ae85","Type":"ContainerDied","Data":"8887073d6b82f77815fd74f83a94db8a2628cfbdb13e83e380ae6cc7b7d05d3e"}
Jul 27 23:31:30.434434 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.434322    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cdb5477c6-6b9j5" event={"ID":"aa3f98b0-3bd2-4e67-89d2-ebfde735ae85","Type":"ContainerDied","Data":"0a8f1448dc17a4cf2dc88de5201d44c6b53be35f8bb07a72c7fae4bc0bfe48ab"}
Jul 27 23:31:30.434434 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.434346    2559 scope.go:117] "RemoveContainer" containerID="8887073d6b82f77815fd74f83a94db8a2628cfbdb13e83e380ae6cc7b7d05d3e"
Jul 27 23:31:30.443775 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.443758    2559 scope.go:117] "RemoveContainer" containerID="8887073d6b82f77815fd74f83a94db8a2628cfbdb13e83e380ae6cc7b7d05d3e"
Jul 27 23:31:30.444043 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:31:30.444024    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8887073d6b82f77815fd74f83a94db8a2628cfbdb13e83e380ae6cc7b7d05d3e\": container with ID starting with 8887073d6b82f77815fd74f83a94db8a2628cfbdb13e83e380ae6cc7b7d05d3e not found: ID does not exist" containerID="8887073d6b82f77815fd74f83a94db8a2628cfbdb13e83e380ae6cc7b7d05d3e"
Jul 27 23:31:30.444092 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.444051    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8887073d6b82f77815fd74f83a94db8a2628cfbdb13e83e380ae6cc7b7d05d3e"} err="failed to get container status \"8887073d6b82f77815fd74f83a94db8a2628cfbdb13e83e380ae6cc7b7d05d3e\": rpc error: code = NotFound desc = could not find container \"8887073d6b82f77815fd74f83a94db8a2628cfbdb13e83e380ae6cc7b7d05d3e\": container with ID starting with 8887073d6b82f77815fd74f83a94db8a2628cfbdb13e83e380ae6cc7b7d05d3e not found: ID does not exist"
Jul 27 23:31:30.454235 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.454208    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cdb5477c6-6b9j5"]
Jul 27 23:31:30.459297 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:30.459276    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7cdb5477c6-6b9j5"]
Jul 27 23:31:31.562877 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:31:31.562846    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa3f98b0-3bd2-4e67-89d2-ebfde735ae85" path="/var/lib/kubelet/pods/aa3f98b0-3bd2-4e67-89d2-ebfde735ae85/volumes"
Jul 27 23:36:17.530930 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:36:17.530899    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:36:17.531497 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:36:17.530899    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:38:30.435153 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:30.435111    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"]
Jul 27 23:38:30.496841 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:30.496765    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d" podUID="24ddd5ac-46a3-4c58-8414-0da283830da0" containerName="manager" containerID="cri-o://a2f73e21645032fbe649da1832e258f18621025fbb325741c2ef8ca7227644ac" gracePeriod=10
Jul 27 23:38:34.462859 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.462835    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"
Jul 27 23:38:34.512766 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.512739    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kglpp\" (UniqueName: \"kubernetes.io/projected/24ddd5ac-46a3-4c58-8414-0da283830da0-kube-api-access-kglpp\") pod \"24ddd5ac-46a3-4c58-8414-0da283830da0\" (UID: \"24ddd5ac-46a3-4c58-8414-0da283830da0\") "
Jul 27 23:38:34.512927 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.512828    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24ddd5ac-46a3-4c58-8414-0da283830da0-extensions-socket-volume\") pod \"24ddd5ac-46a3-4c58-8414-0da283830da0\" (UID: \"24ddd5ac-46a3-4c58-8414-0da283830da0\") "
Jul 27 23:38:34.513203 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.513180    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ddd5ac-46a3-4c58-8414-0da283830da0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "24ddd5ac-46a3-4c58-8414-0da283830da0" (UID: "24ddd5ac-46a3-4c58-8414-0da283830da0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:38:34.515081 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.515055    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ddd5ac-46a3-4c58-8414-0da283830da0-kube-api-access-kglpp" (OuterVolumeSpecName: "kube-api-access-kglpp") pod "24ddd5ac-46a3-4c58-8414-0da283830da0" (UID: "24ddd5ac-46a3-4c58-8414-0da283830da0"). InnerVolumeSpecName "kube-api-access-kglpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:38:34.519428 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.519396    2559 generic.go:356] "Generic (PLEG): container finished" podID="24ddd5ac-46a3-4c58-8414-0da283830da0" containerID="a2f73e21645032fbe649da1832e258f18621025fbb325741c2ef8ca7227644ac" exitCode=0
Jul 27 23:38:34.519542 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.519456    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"
Jul 27 23:38:34.519542 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.519478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d" event={"ID":"24ddd5ac-46a3-4c58-8414-0da283830da0","Type":"ContainerDied","Data":"a2f73e21645032fbe649da1832e258f18621025fbb325741c2ef8ca7227644ac"}
Jul 27 23:38:34.519542 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.519531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d" event={"ID":"24ddd5ac-46a3-4c58-8414-0da283830da0","Type":"ContainerDied","Data":"ff59ea24747ec43d85a72ba50557b8f9c3df7b05614d44600f06c490cb334782"}
Jul 27 23:38:34.519654 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.519563    2559 scope.go:117] "RemoveContainer" containerID="a2f73e21645032fbe649da1832e258f18621025fbb325741c2ef8ca7227644ac"
Jul 27 23:38:34.529581 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.529523    2559 scope.go:117] "RemoveContainer" containerID="a2f73e21645032fbe649da1832e258f18621025fbb325741c2ef8ca7227644ac"
Jul 27 23:38:34.529888 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:38:34.529866    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f73e21645032fbe649da1832e258f18621025fbb325741c2ef8ca7227644ac\": container with ID starting with a2f73e21645032fbe649da1832e258f18621025fbb325741c2ef8ca7227644ac not found: ID does not exist" containerID="a2f73e21645032fbe649da1832e258f18621025fbb325741c2ef8ca7227644ac"
Jul 27 23:38:34.529953 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.529897    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f73e21645032fbe649da1832e258f18621025fbb325741c2ef8ca7227644ac"} err="failed to get container status \"a2f73e21645032fbe649da1832e258f18621025fbb325741c2ef8ca7227644ac\": rpc error: code = NotFound desc = could not find container \"a2f73e21645032fbe649da1832e258f18621025fbb325741c2ef8ca7227644ac\": container with ID starting with a2f73e21645032fbe649da1832e258f18621025fbb325741c2ef8ca7227644ac not found: ID does not exist"
Jul 27 23:38:34.541914 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.541849    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"]
Jul 27 23:38:34.545398 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.545355    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mg64d"]
Jul 27 23:38:34.614186 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.614147    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kglpp\" (UniqueName: \"kubernetes.io/projected/24ddd5ac-46a3-4c58-8414-0da283830da0-kube-api-access-kglpp\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:38:34.614186 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:34.614176    2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24ddd5ac-46a3-4c58-8414-0da283830da0-extensions-socket-volume\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:38:35.563565 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:38:35.563533    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24ddd5ac-46a3-4c58-8414-0da283830da0" path="/var/lib/kubelet/pods/24ddd5ac-46a3-4c58-8414-0da283830da0/volumes"
Jul 27 23:39:31.639056 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.639022    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7"]
Jul 27 23:39:31.639494 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.639463    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ddd5ac-46a3-4c58-8414-0da283830da0" containerName="manager"
Jul 27 23:39:31.639494 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.639477    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ddd5ac-46a3-4c58-8414-0da283830da0" containerName="manager"
Jul 27 23:39:31.639494 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.639490    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3f98b0-3bd2-4e67-89d2-ebfde735ae85" containerName="authorino"
Jul 27 23:39:31.639596 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.639496    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3f98b0-3bd2-4e67-89d2-ebfde735ae85" containerName="authorino"
Jul 27 23:39:31.639596 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.639569    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa3f98b0-3bd2-4e67-89d2-ebfde735ae85" containerName="authorino"
Jul 27 23:39:31.639596 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.639578    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ddd5ac-46a3-4c58-8414-0da283830da0" containerName="manager"
Jul 27 23:39:31.643790 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.643774    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7"
Jul 27 23:39:31.645871 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.645846    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6nx7v\""
Jul 27 23:39:31.656297 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.656272    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7"]
Jul 27 23:39:31.720207 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.720175    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t49r8\" (UniqueName: \"kubernetes.io/projected/a536fc25-5bcf-4327-8659-a8b1867dbe93-kube-api-access-t49r8\") pod \"kuadrant-operator-controller-manager-86878c769f-q87m7\" (UID: \"a536fc25-5bcf-4327-8659-a8b1867dbe93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7"
Jul 27 23:39:31.720385 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.720221    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a536fc25-5bcf-4327-8659-a8b1867dbe93-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q87m7\" (UID: \"a536fc25-5bcf-4327-8659-a8b1867dbe93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7"
Jul 27 23:39:31.820916 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.820871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t49r8\" (UniqueName: \"kubernetes.io/projected/a536fc25-5bcf-4327-8659-a8b1867dbe93-kube-api-access-t49r8\") pod \"kuadrant-operator-controller-manager-86878c769f-q87m7\" (UID: \"a536fc25-5bcf-4327-8659-a8b1867dbe93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7"
Jul 27 23:39:31.821114 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.820944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a536fc25-5bcf-4327-8659-a8b1867dbe93-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q87m7\" (UID: \"a536fc25-5bcf-4327-8659-a8b1867dbe93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7"
Jul 27 23:39:31.821427 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.821384    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a536fc25-5bcf-4327-8659-a8b1867dbe93-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q87m7\" (UID: \"a536fc25-5bcf-4327-8659-a8b1867dbe93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7"
Jul 27 23:39:31.829980 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.829951    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t49r8\" (UniqueName: \"kubernetes.io/projected/a536fc25-5bcf-4327-8659-a8b1867dbe93-kube-api-access-t49r8\") pod \"kuadrant-operator-controller-manager-86878c769f-q87m7\" (UID: \"a536fc25-5bcf-4327-8659-a8b1867dbe93\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7"
Jul 27 23:39:31.954361 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:31.954270    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7"
Jul 27 23:39:32.100401 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:32.100308    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7"]
Jul 27 23:39:32.103030 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:39:32.102990    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda536fc25_5bcf_4327_8659_a8b1867dbe93.slice/crio-8b06ce5ce20d4793099c4ca09fd68a6e8e3805420455791827451e6dba7d3c0e WatchSource:0}: Error finding container 8b06ce5ce20d4793099c4ca09fd68a6e8e3805420455791827451e6dba7d3c0e: Status 404 returned error can't find the container with id 8b06ce5ce20d4793099c4ca09fd68a6e8e3805420455791827451e6dba7d3c0e
Jul 27 23:39:32.809802 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:32.809763    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7" event={"ID":"a536fc25-5bcf-4327-8659-a8b1867dbe93","Type":"ContainerStarted","Data":"3920994c10810522eb3cdd209980abf3607569f3f57446aa485f41ddfd2a7129"}
Jul 27 23:39:32.810221 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:32.809829    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7" event={"ID":"a536fc25-5bcf-4327-8659-a8b1867dbe93","Type":"ContainerStarted","Data":"8b06ce5ce20d4793099c4ca09fd68a6e8e3805420455791827451e6dba7d3c0e"}
Jul 27 23:39:32.810221 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:32.809859    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7"
Jul 27 23:39:32.841683 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:32.841616    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7" podStartSLOduration=1.841594951 podStartE2EDuration="1.841594951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:39:32.839683925 +0000 UTC m=+1395.783914672" watchObservedRunningTime="2026-07-27 23:39:32.841594951 +0000 UTC m=+1395.785825698"
Jul 27 23:39:43.817836 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:39:43.817800    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q87m7"
Jul 27 23:41:17.562637 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:41:17.562599    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:41:17.565087 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:41:17.562985    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:42:12.957910 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:12.957858    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"]
Jul 27 23:42:12.961556 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:12.961534    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:12.963684 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:12.963657    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3dfbadf5-kserve-self-signed-certs\""
Jul 27 23:42:12.975089 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:12.975057    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"]
Jul 27 23:42:13.043364 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.043324    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c03f4de0-b07c-45bd-913a-bd302a7862f5-tls-certs\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.043574 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.043404    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-home\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.043574 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.043426    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-tmp-dir\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.043574 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.043457    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-dshm\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.043683 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.043556    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-model-cache\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.043683 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.043602    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8nl2\" (UniqueName: \"kubernetes.io/projected/c03f4de0-b07c-45bd-913a-bd302a7862f5-kube-api-access-d8nl2\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.144763 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.144718    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-model-cache\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.144763 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.144761    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8nl2\" (UniqueName: \"kubernetes.io/projected/c03f4de0-b07c-45bd-913a-bd302a7862f5-kube-api-access-d8nl2\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.145017 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.144794    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c03f4de0-b07c-45bd-913a-bd302a7862f5-tls-certs\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.145017 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.144830    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-home\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.145017 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.144844    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-tmp-dir\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.145017 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.144876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-dshm\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.145239 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.145218    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-model-cache\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.145299 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.145239    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-home\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.145299 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.145287    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-tmp-dir\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.147452 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.147417    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-dshm\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.147784 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.147762    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c03f4de0-b07c-45bd-913a-bd302a7862f5-tls-certs\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.153958 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.153911    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8nl2\" (UniqueName: \"kubernetes.io/projected/c03f4de0-b07c-45bd-913a-bd302a7862f5-kube-api-access-d8nl2\") pod \"e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") " pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.270882 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.270759    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:13.420809 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.420780    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"]
Jul 27 23:42:13.423259 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:42:13.423226    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc03f4de0_b07c_45bd_913a_bd302a7862f5.slice/crio-e81a3a6a99cd29ac8251db2cd5a0632284b8d06568bfb88cadacf1d1e6e60cf8 WatchSource:0}: Error finding container e81a3a6a99cd29ac8251db2cd5a0632284b8d06568bfb88cadacf1d1e6e60cf8: Status 404 returned error can't find the container with id e81a3a6a99cd29ac8251db2cd5a0632284b8d06568bfb88cadacf1d1e6e60cf8
Jul 27 23:42:13.620583 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.620542    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k" event={"ID":"c03f4de0-b07c-45bd-913a-bd302a7862f5","Type":"ContainerStarted","Data":"7698a263d59a5e44c08aabd5679e291b0edc737869a72feae93081eb48b0812f"}
Jul 27 23:42:13.620583 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.620590    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k" event={"ID":"c03f4de0-b07c-45bd-913a-bd302a7862f5","Type":"ContainerStarted","Data":"e81a3a6a99cd29ac8251db2cd5a0632284b8d06568bfb88cadacf1d1e6e60cf8"}
Jul 27 23:42:13.636152 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:13.636097    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k" podStartSLOduration=1.636078726 podStartE2EDuration="1.636078726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:42:13.634877157 +0000 UTC m=+1556.579107902" watchObservedRunningTime="2026-07-27 23:42:13.636078726 +0000 UTC m=+1556.580309472"
Jul 27 23:42:23.271085 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:23.270978    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:23.271085 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:23.271040    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:23.283851 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:23.283823    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:23.682442 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:23.682413    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:42:45.130983 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:42:45.130946    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"]
Jul 27 23:43:00.151436 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.151338    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k" podUID="c03f4de0-b07c-45bd-913a-bd302a7862f5" containerName="main" containerID="cri-o://7698a263d59a5e44c08aabd5679e291b0edc737869a72feae93081eb48b0812f" gracePeriod=45
Jul 27 23:43:00.418521 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.418497    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:43:00.500765 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.500727    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-tmp-dir\") pod \"c03f4de0-b07c-45bd-913a-bd302a7862f5\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") "
Jul 27 23:43:00.500933 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.500795    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-home\") pod \"c03f4de0-b07c-45bd-913a-bd302a7862f5\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") "
Jul 27 23:43:00.500933 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.500879    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c03f4de0-b07c-45bd-913a-bd302a7862f5-tls-certs\") pod \"c03f4de0-b07c-45bd-913a-bd302a7862f5\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") "
Jul 27 23:43:00.500933 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.500922    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8nl2\" (UniqueName: \"kubernetes.io/projected/c03f4de0-b07c-45bd-913a-bd302a7862f5-kube-api-access-d8nl2\") pod \"c03f4de0-b07c-45bd-913a-bd302a7862f5\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") "
Jul 27 23:43:00.501086 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.500958    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-dshm\") pod \"c03f4de0-b07c-45bd-913a-bd302a7862f5\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") "
Jul 27 23:43:00.501086 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.500959    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c03f4de0-b07c-45bd-913a-bd302a7862f5" (UID: "c03f4de0-b07c-45bd-913a-bd302a7862f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:43:00.501086 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.500990    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-model-cache\") pod \"c03f4de0-b07c-45bd-913a-bd302a7862f5\" (UID: \"c03f4de0-b07c-45bd-913a-bd302a7862f5\") "
Jul 27 23:43:00.501086 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.501068    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-home" (OuterVolumeSpecName: "home") pod "c03f4de0-b07c-45bd-913a-bd302a7862f5" (UID: "c03f4de0-b07c-45bd-913a-bd302a7862f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:43:00.501665 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.501295    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-tmp-dir\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:43:00.501665 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.501313    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-home\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:43:00.501665 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.501328    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "c03f4de0-b07c-45bd-913a-bd302a7862f5" (UID: "c03f4de0-b07c-45bd-913a-bd302a7862f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:43:00.503410 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.503359    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03f4de0-b07c-45bd-913a-bd302a7862f5-kube-api-access-d8nl2" (OuterVolumeSpecName: "kube-api-access-d8nl2") pod "c03f4de0-b07c-45bd-913a-bd302a7862f5" (UID: "c03f4de0-b07c-45bd-913a-bd302a7862f5"). InnerVolumeSpecName "kube-api-access-d8nl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:43:00.503539 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.503403    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c03f4de0-b07c-45bd-913a-bd302a7862f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c03f4de0-b07c-45bd-913a-bd302a7862f5" (UID: "c03f4de0-b07c-45bd-913a-bd302a7862f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:43:00.503801 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.503785    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-dshm" (OuterVolumeSpecName: "dshm") pod "c03f4de0-b07c-45bd-913a-bd302a7862f5" (UID: "c03f4de0-b07c-45bd-913a-bd302a7862f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:43:00.602435 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.602396    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c03f4de0-b07c-45bd-913a-bd302a7862f5-tls-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:43:00.602435 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.602432    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8nl2\" (UniqueName: \"kubernetes.io/projected/c03f4de0-b07c-45bd-913a-bd302a7862f5-kube-api-access-d8nl2\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:43:00.602636 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.602446    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-dshm\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:43:00.602636 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.602456    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c03f4de0-b07c-45bd-913a-bd302a7862f5-model-cache\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:43:00.865675 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.865638    2559 generic.go:356] "Generic (PLEG): container finished" podID="c03f4de0-b07c-45bd-913a-bd302a7862f5" containerID="7698a263d59a5e44c08aabd5679e291b0edc737869a72feae93081eb48b0812f" exitCode=0
Jul 27 23:43:00.865858 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.865683    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"
Jul 27 23:43:00.865858 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.865713    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k" event={"ID":"c03f4de0-b07c-45bd-913a-bd302a7862f5","Type":"ContainerDied","Data":"7698a263d59a5e44c08aabd5679e291b0edc737869a72feae93081eb48b0812f"}
Jul 27 23:43:00.865858 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.865755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k" event={"ID":"c03f4de0-b07c-45bd-913a-bd302a7862f5","Type":"ContainerDied","Data":"e81a3a6a99cd29ac8251db2cd5a0632284b8d06568bfb88cadacf1d1e6e60cf8"}
Jul 27 23:43:00.865858 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.865772    2559 scope.go:117] "RemoveContainer" containerID="7698a263d59a5e44c08aabd5679e291b0edc737869a72feae93081eb48b0812f"
Jul 27 23:43:00.875121 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.875105    2559 scope.go:117] "RemoveContainer" containerID="7698a263d59a5e44c08aabd5679e291b0edc737869a72feae93081eb48b0812f"
Jul 27 23:43:00.875421 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:43:00.875401    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7698a263d59a5e44c08aabd5679e291b0edc737869a72feae93081eb48b0812f\": container with ID starting with 7698a263d59a5e44c08aabd5679e291b0edc737869a72feae93081eb48b0812f not found: ID does not exist" containerID="7698a263d59a5e44c08aabd5679e291b0edc737869a72feae93081eb48b0812f"
Jul 27 23:43:00.875479 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.875433    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7698a263d59a5e44c08aabd5679e291b0edc737869a72feae93081eb48b0812f"} err="failed to get container status \"7698a263d59a5e44c08aabd5679e291b0edc737869a72feae93081eb48b0812f\": rpc error: code = NotFound desc = could not find container \"7698a263d59a5e44c08aabd5679e291b0edc737869a72feae93081eb48b0812f\": container with ID starting with 7698a263d59a5e44c08aabd5679e291b0edc737869a72feae93081eb48b0812f not found: ID does not exist"
Jul 27 23:43:00.884830 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.884808    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"]
Jul 27 23:43:00.887860 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:00.887838    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3dfbadf5-kserve-c48c4594c-srj2k"]
Jul 27 23:43:01.564461 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:43:01.564421    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c03f4de0-b07c-45bd-913a-bd302a7862f5" path="/var/lib/kubelet/pods/c03f4de0-b07c-45bd-913a-bd302a7862f5/volumes"
Jul 27 23:46:17.598040 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:17.597916    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:46:17.602532 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:17.600165    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:46:26.937890 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:26.937835    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"]
Jul 27 23:46:26.938516 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:26.938483    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c03f4de0-b07c-45bd-913a-bd302a7862f5" containerName="main"
Jul 27 23:46:26.938516 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:26.938508    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c03f4de0-b07c-45bd-913a-bd302a7862f5" containerName="main"
Jul 27 23:46:26.938666 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:26.938648    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c03f4de0-b07c-45bd-913a-bd302a7862f5" containerName="main"
Jul 27 23:46:26.941513 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:26.941494    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:26.943634 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:26.943613    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-17ce21b6-openshift-default-dockercfg-m9bld\""
Jul 27 23:46:26.951219 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:26.951195    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"]
Jul 27 23:46:27.025474 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.025442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-credential-socket\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.025657 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.025497    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-workload-certs\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.025657 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.025547    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-data\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.025657 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.025602    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-podinfo\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.025657 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.025624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e834bf-8e75-44e8-967a-64528a4c6bcb-istiod-ca-cert\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.025657 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.025649    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzj68\" (UniqueName: \"kubernetes.io/projected/43e834bf-8e75-44e8-967a-64528a4c6bcb-kube-api-access-nzj68\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.025859 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.025703    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-workload-socket\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.025859 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.025773    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-envoy\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.025859 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.025807    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-token\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.127123 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.127081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-workload-certs\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.127123 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.127126    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-data\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.127416 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.127173    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-podinfo\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.127416 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.127198    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e834bf-8e75-44e8-967a-64528a4c6bcb-istiod-ca-cert\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.127416 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.127219    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzj68\" (UniqueName: \"kubernetes.io/projected/43e834bf-8e75-44e8-967a-64528a4c6bcb-kube-api-access-nzj68\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.127416 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.127246    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-workload-socket\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.127416 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.127315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-envoy\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.127416 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.127354    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-token\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.127726 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.127435    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-credential-socket\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.127726 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.127601    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-workload-certs\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.127726 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.127642    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-data\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.127870 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.127785    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-workload-socket\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.127870 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.127807    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-credential-socket\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.128029 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.127979    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e834bf-8e75-44e8-967a-64528a4c6bcb-istiod-ca-cert\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.129822 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.129800    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-envoy\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.129990 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.129968    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-podinfo\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.134924 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.134900    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-token\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.135095 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.135073    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzj68\" (UniqueName: \"kubernetes.io/projected/43e834bf-8e75-44e8-967a-64528a4c6bcb-kube-api-access-nzj68\") pod \"e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") " pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.251924 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.251822    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:27.394366 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.394339    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"]
Jul 27 23:46:27.397880 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:46:27.397845    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e834bf_8e75_44e8_967a_64528a4c6bcb.slice/crio-02275688beb463be28b8267c77c5d8649d1a5cafa908aa72eb1ac4d3ff157051 WatchSource:0}: Error finding container 02275688beb463be28b8267c77c5d8649d1a5cafa908aa72eb1ac4d3ff157051: Status 404 returned error can't find the container with id 02275688beb463be28b8267c77c5d8649d1a5cafa908aa72eb1ac4d3ff157051
Jul 27 23:46:27.400214 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.400175    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:46:27.400382 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.400265    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:46:27.400382 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.400309    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:46:27.891673 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.891631    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn" event={"ID":"43e834bf-8e75-44e8-967a-64528a4c6bcb","Type":"ContainerStarted","Data":"491a1df7751576d9ea591508e0709b449535fd70c0a29e3418c6c29bb3d9b240"}
Jul 27 23:46:27.891869 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.891699    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn" event={"ID":"43e834bf-8e75-44e8-967a-64528a4c6bcb","Type":"ContainerStarted","Data":"02275688beb463be28b8267c77c5d8649d1a5cafa908aa72eb1ac4d3ff157051"}
Jul 27 23:46:27.911117 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:27.911062    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn" podStartSLOduration=1.911045105 podStartE2EDuration="1.911045105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:46:27.909103074 +0000 UTC m=+1810.853333821" watchObservedRunningTime="2026-07-27 23:46:27.911045105 +0000 UTC m=+1810.855275851"
Jul 27 23:46:28.252485 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:28.252390    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:28.258424 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:28.258397    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:28.897428 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:28.897399    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:28.898439 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:28.898420    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:41.965834 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:41.965730    2559 generic.go:356] "Generic (PLEG): container finished" podID="43e834bf-8e75-44e8-967a-64528a4c6bcb" containerID="491a1df7751576d9ea591508e0709b449535fd70c0a29e3418c6c29bb3d9b240" exitCode=0
Jul 27 23:46:41.965834 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:41.965803    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn" event={"ID":"43e834bf-8e75-44e8-967a-64528a4c6bcb","Type":"ContainerDied","Data":"491a1df7751576d9ea591508e0709b449535fd70c0a29e3418c6c29bb3d9b240"}
Jul 27 23:46:41.966260 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:41.966238    2559 scope.go:117] "RemoveContainer" containerID="491a1df7751576d9ea591508e0709b449535fd70c0a29e3418c6c29bb3d9b240"
Jul 27 23:46:41.967387 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:41.967333    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:46:41.967668 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:41.967434    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:46:41.967668 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:41.967469    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:46:42.972799 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:42.972762    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn" event={"ID":"43e834bf-8e75-44e8-967a-64528a4c6bcb","Type":"ContainerStarted","Data":"c77a7035970345061d5092cc432e4ff033a055c3bde53ea3371853c1109e8c83"}
Jul 27 23:46:43.252679 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:43.252582    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:43.253680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:43.253653    2559 patch_prober.go:28] interesting pod/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 27 23:46:43.253803 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:43.253707    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn" podUID="43e834bf-8e75-44e8-967a-64528a4c6bcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 27 23:46:43.897730 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:43.897682    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:44.252896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:44.252801    2559 patch_prober.go:28] interesting pod/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 27 23:46:44.252896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:44.252864    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn" podUID="43e834bf-8e75-44e8-967a-64528a4c6bcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 27 23:46:45.252844 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:45.252801    2559 patch_prober.go:28] interesting pod/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 27 23:46:45.253062 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:45.252866    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn" podUID="43e834bf-8e75-44e8-967a-64528a4c6bcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 27 23:46:46.252959 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:46.252919    2559 patch_prober.go:28] interesting pod/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 27 23:46:46.253217 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:46.252978    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn" podUID="43e834bf-8e75-44e8-967a-64528a4c6bcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 27 23:46:47.256854 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:47.256819    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:46:47.257817 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:46:47.257797    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:47:10.132679 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:10.132644    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"]
Jul 27 23:47:10.133170 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:10.132921    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn" podUID="43e834bf-8e75-44e8-967a-64528a4c6bcb" containerName="istio-proxy" containerID="cri-o://c77a7035970345061d5092cc432e4ff033a055c3bde53ea3371853c1109e8c83" gracePeriod=30
Jul 27 23:47:15.403254 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.403194    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:47:15.497174 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497136    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-credential-socket\") pod \"43e834bf-8e75-44e8-967a-64528a4c6bcb\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") "
Jul 27 23:47:15.497174 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497175    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzj68\" (UniqueName: \"kubernetes.io/projected/43e834bf-8e75-44e8-967a-64528a4c6bcb-kube-api-access-nzj68\") pod \"43e834bf-8e75-44e8-967a-64528a4c6bcb\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") "
Jul 27 23:47:15.497475 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497192    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-token\") pod \"43e834bf-8e75-44e8-967a-64528a4c6bcb\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") "
Jul 27 23:47:15.497475 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497213    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-workload-socket\") pod \"43e834bf-8e75-44e8-967a-64528a4c6bcb\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") "
Jul 27 23:47:15.497475 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497249    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e834bf-8e75-44e8-967a-64528a4c6bcb-istiod-ca-cert\") pod \"43e834bf-8e75-44e8-967a-64528a4c6bcb\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") "
Jul 27 23:47:15.497475 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497303    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-workload-certs\") pod \"43e834bf-8e75-44e8-967a-64528a4c6bcb\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") "
Jul 27 23:47:15.497475 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497341    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-podinfo\") pod \"43e834bf-8e75-44e8-967a-64528a4c6bcb\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") "
Jul 27 23:47:15.497475 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497427    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-data\") pod \"43e834bf-8e75-44e8-967a-64528a4c6bcb\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") "
Jul 27 23:47:15.497475 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497459    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-envoy\") pod \"43e834bf-8e75-44e8-967a-64528a4c6bcb\" (UID: \"43e834bf-8e75-44e8-967a-64528a4c6bcb\") "
Jul 27 23:47:15.497839 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497539    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "43e834bf-8e75-44e8-967a-64528a4c6bcb" (UID: "43e834bf-8e75-44e8-967a-64528a4c6bcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:47:15.497839 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497533    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "43e834bf-8e75-44e8-967a-64528a4c6bcb" (UID: "43e834bf-8e75-44e8-967a-64528a4c6bcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:47:15.497839 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497693    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "43e834bf-8e75-44e8-967a-64528a4c6bcb" (UID: "43e834bf-8e75-44e8-967a-64528a4c6bcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:47:15.497992 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497851    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-credential-socket\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:47:15.497992 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497872    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-workload-socket\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:47:15.497992 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.497894    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-workload-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:47:15.498319 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.498183    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e834bf-8e75-44e8-967a-64528a4c6bcb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "43e834bf-8e75-44e8-967a-64528a4c6bcb" (UID: "43e834bf-8e75-44e8-967a-64528a4c6bcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:47:15.498561 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.498535    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-data" (OuterVolumeSpecName: "istio-data") pod "43e834bf-8e75-44e8-967a-64528a4c6bcb" (UID: "43e834bf-8e75-44e8-967a-64528a4c6bcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:47:15.499830 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.499797    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "43e834bf-8e75-44e8-967a-64528a4c6bcb" (UID: "43e834bf-8e75-44e8-967a-64528a4c6bcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 23:47:15.499923 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.499805    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e834bf-8e75-44e8-967a-64528a4c6bcb-kube-api-access-nzj68" (OuterVolumeSpecName: "kube-api-access-nzj68") pod "43e834bf-8e75-44e8-967a-64528a4c6bcb" (UID: "43e834bf-8e75-44e8-967a-64528a4c6bcb"). InnerVolumeSpecName "kube-api-access-nzj68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:47:15.500264 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.500240    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "43e834bf-8e75-44e8-967a-64528a4c6bcb" (UID: "43e834bf-8e75-44e8-967a-64528a4c6bcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:47:15.500364 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.500278    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-token" (OuterVolumeSpecName: "istio-token") pod "43e834bf-8e75-44e8-967a-64528a4c6bcb" (UID: "43e834bf-8e75-44e8-967a-64528a4c6bcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:47:15.598756 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.598722    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzj68\" (UniqueName: \"kubernetes.io/projected/43e834bf-8e75-44e8-967a-64528a4c6bcb-kube-api-access-nzj68\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:47:15.598756 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.598750    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-token\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:47:15.598756 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.598759    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e834bf-8e75-44e8-967a-64528a4c6bcb-istiod-ca-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:47:15.598975 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.598776    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-podinfo\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:47:15.598975 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.598787    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-data\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:47:15.598975 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:15.598796    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e834bf-8e75-44e8-967a-64528a4c6bcb-istio-envoy\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:47:16.146102 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:16.146069    2559 generic.go:356] "Generic (PLEG): container finished" podID="43e834bf-8e75-44e8-967a-64528a4c6bcb" containerID="c77a7035970345061d5092cc432e4ff033a055c3bde53ea3371853c1109e8c83" exitCode=0
Jul 27 23:47:16.146280 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:16.146131    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn" event={"ID":"43e834bf-8e75-44e8-967a-64528a4c6bcb","Type":"ContainerDied","Data":"c77a7035970345061d5092cc432e4ff033a055c3bde53ea3371853c1109e8c83"}
Jul 27 23:47:16.146280 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:16.146145    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"
Jul 27 23:47:16.146280 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:16.146193    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn" event={"ID":"43e834bf-8e75-44e8-967a-64528a4c6bcb","Type":"ContainerDied","Data":"02275688beb463be28b8267c77c5d8649d1a5cafa908aa72eb1ac4d3ff157051"}
Jul 27 23:47:16.146280 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:16.146218    2559 scope.go:117] "RemoveContainer" containerID="c77a7035970345061d5092cc432e4ff033a055c3bde53ea3371853c1109e8c83"
Jul 27 23:47:16.155690 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:16.155667    2559 scope.go:117] "RemoveContainer" containerID="491a1df7751576d9ea591508e0709b449535fd70c0a29e3418c6c29bb3d9b240"
Jul 27 23:47:16.163567 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:16.163547    2559 scope.go:117] "RemoveContainer" containerID="c77a7035970345061d5092cc432e4ff033a055c3bde53ea3371853c1109e8c83"
Jul 27 23:47:16.163859 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:47:16.163833    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c77a7035970345061d5092cc432e4ff033a055c3bde53ea3371853c1109e8c83\": container with ID starting with c77a7035970345061d5092cc432e4ff033a055c3bde53ea3371853c1109e8c83 not found: ID does not exist" containerID="c77a7035970345061d5092cc432e4ff033a055c3bde53ea3371853c1109e8c83"
Jul 27 23:47:16.163989 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:16.163868    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c77a7035970345061d5092cc432e4ff033a055c3bde53ea3371853c1109e8c83"} err="failed to get container status \"c77a7035970345061d5092cc432e4ff033a055c3bde53ea3371853c1109e8c83\": rpc error: code = NotFound desc = could not find container \"c77a7035970345061d5092cc432e4ff033a055c3bde53ea3371853c1109e8c83\": container with ID starting with c77a7035970345061d5092cc432e4ff033a055c3bde53ea3371853c1109e8c83 not found: ID does not exist"
Jul 27 23:47:16.163989 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:16.163885    2559 scope.go:117] "RemoveContainer" containerID="491a1df7751576d9ea591508e0709b449535fd70c0a29e3418c6c29bb3d9b240"
Jul 27 23:47:16.164213 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:47:16.164193    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491a1df7751576d9ea591508e0709b449535fd70c0a29e3418c6c29bb3d9b240\": container with ID starting with 491a1df7751576d9ea591508e0709b449535fd70c0a29e3418c6c29bb3d9b240 not found: ID does not exist" containerID="491a1df7751576d9ea591508e0709b449535fd70c0a29e3418c6c29bb3d9b240"
Jul 27 23:47:16.164280 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:16.164224    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491a1df7751576d9ea591508e0709b449535fd70c0a29e3418c6c29bb3d9b240"} err="failed to get container status \"491a1df7751576d9ea591508e0709b449535fd70c0a29e3418c6c29bb3d9b240\": rpc error: code = NotFound desc = could not find container \"491a1df7751576d9ea591508e0709b449535fd70c0a29e3418c6c29bb3d9b240\": container with ID starting with 491a1df7751576d9ea591508e0709b449535fd70c0a29e3418c6c29bb3d9b240 not found: ID does not exist"
Jul 27 23:47:16.166721 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:16.166699    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"]
Jul 27 23:47:16.171791 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:16.171767    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-17ce21b6-openshift-default-596cb46c8-nldbn"]
Jul 27 23:47:17.564146 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:17.564107    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43e834bf-8e75-44e8-967a-64528a4c6bcb" path="/var/lib/kubelet/pods/43e834bf-8e75-44e8-967a-64528a4c6bcb/volumes"
Jul 27 23:47:41.121774 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:41.121687    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w"]
Jul 27 23:47:41.122321 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:41.122261    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e834bf-8e75-44e8-967a-64528a4c6bcb" containerName="istio-proxy"
Jul 27 23:47:41.122321 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:41.122282    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e834bf-8e75-44e8-967a-64528a4c6bcb" containerName="istio-proxy"
Jul 27 23:47:41.122487 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:41.122405    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e834bf-8e75-44e8-967a-64528a4c6bcb" containerName="istio-proxy"
Jul 27 23:47:41.122487 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:41.122419    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e834bf-8e75-44e8-967a-64528a4c6bcb" containerName="istio-proxy"
Jul 27 23:47:41.124594 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:41.124574    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w"
Jul 27 23:47:41.126747 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:41.126728    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 27 23:47:41.132922 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:41.132902    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w"]
Jul 27 23:47:41.233000 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:41.232958    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c38ca-cd31-4e60-881a-ebbbb637d777-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w\" (UID: \"ea0c38ca-cd31-4e60-881a-ebbbb637d777\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w"
Jul 27 23:47:41.334096 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:41.334058    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c38ca-cd31-4e60-881a-ebbbb637d777-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w\" (UID: \"ea0c38ca-cd31-4e60-881a-ebbbb637d777\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w"
Jul 27 23:47:41.334714 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:41.334693    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c38ca-cd31-4e60-881a-ebbbb637d777-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w\" (UID: \"ea0c38ca-cd31-4e60-881a-ebbbb637d777\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w"
Jul 27 23:47:41.435899 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:41.435810    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w"
Jul 27 23:47:41.572547 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:41.572512    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w"]
Jul 27 23:47:41.574869 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:47:41.574825    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0c38ca_cd31_4e60_881a_ebbbb637d777.slice/crio-db89903c8c058a8873074a2b9f5393043be959976423846f9e71cc253b46262a WatchSource:0}: Error finding container db89903c8c058a8873074a2b9f5393043be959976423846f9e71cc253b46262a: Status 404 returned error can't find the container with id db89903c8c058a8873074a2b9f5393043be959976423846f9e71cc253b46262a
Jul 27 23:47:41.576725 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:41.576705    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 23:47:42.275290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:42.275231    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w" event={"ID":"ea0c38ca-cd31-4e60-881a-ebbbb637d777","Type":"ContainerStarted","Data":"db89903c8c058a8873074a2b9f5393043be959976423846f9e71cc253b46262a"}
Jul 27 23:47:44.288436 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:44.288393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w" event={"ID":"ea0c38ca-cd31-4e60-881a-ebbbb637d777","Type":"ContainerStarted","Data":"de81841c8b372b789f155f4473c6076e65413a65cdbe2f5315b96120c81ff26f"}
Jul 27 23:47:44.302576 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:44.302529    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w" podStartSLOduration=1.308266522 podStartE2EDuration="3.302513269s" totalImagesPullingTime="1.994246747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:47:44.300428153 +0000 UTC m=+1887.244658900" watchObservedRunningTime="2026-07-27 23:47:44.302513269 +0000 UTC m=+1887.246744015"
Jul 27 23:47:45.295931 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:45.295892    2559 generic.go:356] "Generic (PLEG): container finished" podID="ea0c38ca-cd31-4e60-881a-ebbbb637d777" containerID="de81841c8b372b789f155f4473c6076e65413a65cdbe2f5315b96120c81ff26f" exitCode=0
Jul 27 23:47:45.296308 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:45.295956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w" event={"ID":"ea0c38ca-cd31-4e60-881a-ebbbb637d777","Type":"ContainerDied","Data":"de81841c8b372b789f155f4473c6076e65413a65cdbe2f5315b96120c81ff26f"}
Jul 27 23:47:46.437271 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:46.437244    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w"
Jul 27 23:47:46.586619 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:46.586534    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c38ca-cd31-4e60-881a-ebbbb637d777-maas-api-service-ca\") pod \"ea0c38ca-cd31-4e60-881a-ebbbb637d777\" (UID: \"ea0c38ca-cd31-4e60-881a-ebbbb637d777\") "
Jul 27 23:47:46.586922 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:46.586900    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0c38ca-cd31-4e60-881a-ebbbb637d777-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea0c38ca-cd31-4e60-881a-ebbbb637d777" (UID: "ea0c38ca-cd31-4e60-881a-ebbbb637d777"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:47:46.687357 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:46.687318    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c38ca-cd31-4e60-881a-ebbbb637d777-maas-api-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:47:47.309350 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:47.309321    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w"
Jul 27 23:47:47.309350 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:47.309327    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w" event={"ID":"ea0c38ca-cd31-4e60-881a-ebbbb637d777","Type":"ContainerDied","Data":"db89903c8c058a8873074a2b9f5393043be959976423846f9e71cc253b46262a"}
Jul 27 23:47:47.309637 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:47.309405    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db89903c8c058a8873074a2b9f5393043be959976423846f9e71cc253b46262a"
Jul 27 23:47:56.195100 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:56.195058    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w"]
Jul 27 23:47:56.197045 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:56.197019    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5c2a4b0b-b8l2w"]
Jul 27 23:47:57.051608 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.051575    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"]
Jul 27 23:47:57.052048 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.052032    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea0c38ca-cd31-4e60-881a-ebbbb637d777" containerName="revoke-keys"
Jul 27 23:47:57.052145 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.052051    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea0c38ca-cd31-4e60-881a-ebbbb637d777" containerName="revoke-keys"
Jul 27 23:47:57.052145 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.052070    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e834bf-8e75-44e8-967a-64528a4c6bcb" containerName="istio-proxy"
Jul 27 23:47:57.052145 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.052077    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e834bf-8e75-44e8-967a-64528a4c6bcb" containerName="istio-proxy"
Jul 27 23:47:57.052302 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.052168    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea0c38ca-cd31-4e60-881a-ebbbb637d777" containerName="revoke-keys"
Jul 27 23:47:57.055979 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.055958    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.057905 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.057887    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7355578b-openshift-default-dockercfg-jsnxd\""
Jul 27 23:47:57.065713 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.065687    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"]
Jul 27 23:47:57.084591 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.084566    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-workload-certs\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.084720 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.084601    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-token\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.084720 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.084621    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-data\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.084720 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.084665    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-workload-socket\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.084720 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.084683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-podinfo\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.084720 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.084709    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istiod-ca-cert\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.084916 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.084806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-envoy\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.084916 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.084829    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-credential-socket\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.084916 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.084861    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88rcq\" (UniqueName: \"kubernetes.io/projected/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-kube-api-access-88rcq\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.185673 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.185635    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-workload-certs\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.185865 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.185684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-token\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.185865 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.185704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-data\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.185865 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.185732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-workload-socket\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.185865 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.185758    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-podinfo\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.185865 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.185788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istiod-ca-cert\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.186128 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.185871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-envoy\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.186128 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.185894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-credential-socket\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.186128 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.185937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88rcq\" (UniqueName: \"kubernetes.io/projected/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-kube-api-access-88rcq\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.186290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.186118    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-workload-certs\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.186290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.186157    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-data\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.186290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.186227    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-workload-socket\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.186495 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.186455    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-credential-socket\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.186708 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.186687    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istiod-ca-cert\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.188449 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.188429    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-envoy\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.188561 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.188537    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-podinfo\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.193728 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.193705    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-token\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.193827 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.193793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88rcq\" (UniqueName: \"kubernetes.io/projected/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-kube-api-access-88rcq\") pod \"e2e-ait-7355578b-openshift-default-96df69c84-46d6q\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") " pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.367175 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.367090    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:57.506833 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.506806    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"]
Jul 27 23:47:57.509520 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:47:57.509496    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39e6c14_5c7f_47e4_b3fb_f5891c6fb9a2.slice/crio-8cfd96c2c7ed1493d1338558e57aed401eccaf0dcb9b7960847fa072fe7bc906 WatchSource:0}: Error finding container 8cfd96c2c7ed1493d1338558e57aed401eccaf0dcb9b7960847fa072fe7bc906: Status 404 returned error can't find the container with id 8cfd96c2c7ed1493d1338558e57aed401eccaf0dcb9b7960847fa072fe7bc906
Jul 27 23:47:57.512013 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.511981    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:47:57.512105 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.512050    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:47:57.512105 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.512079    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:47:57.564139 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:57.564113    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea0c38ca-cd31-4e60-881a-ebbbb637d777" path="/var/lib/kubelet/pods/ea0c38ca-cd31-4e60-881a-ebbbb637d777/volumes"
Jul 27 23:47:58.366141 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:58.366103    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q" event={"ID":"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2","Type":"ContainerStarted","Data":"7284eefb2190779307c1a7a31cb892086dd398801a698c5b23fc197934bfdc4d"}
Jul 27 23:47:58.366328 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:58.366196    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q" event={"ID":"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2","Type":"ContainerStarted","Data":"8cfd96c2c7ed1493d1338558e57aed401eccaf0dcb9b7960847fa072fe7bc906"}
Jul 27 23:47:58.367398 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:58.367360    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:58.372504 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:58.372479    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:58.385093 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:58.385048    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q" podStartSLOduration=1.385033612 podStartE2EDuration="1.385033612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:47:58.382877981 +0000 UTC m=+1901.327108727" watchObservedRunningTime="2026-07-27 23:47:58.385033612 +0000 UTC m=+1901.329264361"
Jul 27 23:47:59.371816 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:59.371783    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:47:59.372928 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:47:59.372905    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:48:14.373563 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:14.373527    2559 patch_prober.go:28] interesting pod/e2e-ait-7355578b-openshift-default-96df69c84-46d6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 27 23:48:14.374022 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:14.373582    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q" podUID="d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 27 23:48:14.452199 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:14.452162    2559 generic.go:356] "Generic (PLEG): container finished" podID="d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" containerID="7284eefb2190779307c1a7a31cb892086dd398801a698c5b23fc197934bfdc4d" exitCode=0
Jul 27 23:48:14.452380 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:14.452237    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q" event={"ID":"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2","Type":"ContainerDied","Data":"7284eefb2190779307c1a7a31cb892086dd398801a698c5b23fc197934bfdc4d"}
Jul 27 23:48:14.452685 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:14.452666    2559 scope.go:117] "RemoveContainer" containerID="7284eefb2190779307c1a7a31cb892086dd398801a698c5b23fc197934bfdc4d"
Jul 27 23:48:14.453747 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:14.453718    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:48:14.453805 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:14.453794    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:48:14.453840 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:14.453828    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:48:15.459432 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:15.459398    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q" event={"ID":"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2","Type":"ContainerStarted","Data":"bab9c44cab2e5fbb4e409c3000fdc4b267b9323c78b7a3f1c3fe7d383393795b"}
Jul 27 23:48:16.367617 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:16.367581    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:48:16.367932 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:16.367906    2559 patch_prober.go:28] interesting pod/e2e-ait-7355578b-openshift-default-96df69c84-46d6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 27 23:48:16.367984 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:16.367964    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q" podUID="d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 27 23:48:17.367704 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:17.367661    2559 patch_prober.go:28] interesting pod/e2e-ait-7355578b-openshift-default-96df69c84-46d6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 27 23:48:17.368109 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:17.367725    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q" podUID="d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 27 23:48:18.368554 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:18.368512    2559 patch_prober.go:28] interesting pod/e2e-ait-7355578b-openshift-default-96df69c84-46d6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 27 23:48:18.368977 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:18.368585    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q" podUID="d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 27 23:48:19.394599 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:19.394566    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:48:19.395076 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:19.394864    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:48:19.395623 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:19.395607    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:48:41.683040 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:41.683002    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"]
Jul 27 23:48:41.683494 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:41.683211    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q" podUID="d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" containerName="istio-proxy" containerID="cri-o://bab9c44cab2e5fbb4e409c3000fdc4b267b9323c78b7a3f1c3fe7d383393795b" gracePeriod=30
Jul 27 23:48:46.935913 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:46.935882    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:48:47.071065 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071027    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-workload-socket\") pod \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") "
Jul 27 23:48:47.071239 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071079    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-podinfo\") pod \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") "
Jul 27 23:48:47.071239 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071121    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88rcq\" (UniqueName: \"kubernetes.io/projected/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-kube-api-access-88rcq\") pod \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") "
Jul 27 23:48:47.071239 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071169    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-workload-certs\") pod \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") "
Jul 27 23:48:47.071239 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071208    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-data\") pod \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") "
Jul 27 23:48:47.071502 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071324    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-token\") pod \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") "
Jul 27 23:48:47.071502 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071344    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" (UID: "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:48:47.071502 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071348    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istiod-ca-cert\") pod \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") "
Jul 27 23:48:47.071502 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071443    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-envoy\") pod \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") "
Jul 27 23:48:47.071502 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071470    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-credential-socket\") pod \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\" (UID: \"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2\") "
Jul 27 23:48:47.071771 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071591    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" (UID: "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:48:47.071771 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071726    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" (UID: "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:48:47.071874 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071802    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" (UID: "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:48:47.071874 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071851    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istiod-ca-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:48:47.071994 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071871    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-workload-socket\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:48:47.071994 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.071885    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-workload-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:48:47.072728 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.072694    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-data" (OuterVolumeSpecName: "istio-data") pod "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" (UID: "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:48:47.073751 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.073721    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-kube-api-access-88rcq" (OuterVolumeSpecName: "kube-api-access-88rcq") pod "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" (UID: "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2"). InnerVolumeSpecName "kube-api-access-88rcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:48:47.073878 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.073849    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-token" (OuterVolumeSpecName: "istio-token") pod "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" (UID: "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:48:47.074113 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.074092    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" (UID: "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 23:48:47.074113 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.074102    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" (UID: "d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:48:47.172788 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.172748    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-token\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:48:47.172788 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.172781    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-envoy\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:48:47.172788 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.172790    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-credential-socket\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:48:47.172788 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.172800    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-podinfo\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:48:47.173091 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.172808    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88rcq\" (UniqueName: \"kubernetes.io/projected/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-kube-api-access-88rcq\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:48:47.173091 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.172817    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2-istio-data\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:48:47.620026 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.619994    2559 generic.go:356] "Generic (PLEG): container finished" podID="d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" containerID="bab9c44cab2e5fbb4e409c3000fdc4b267b9323c78b7a3f1c3fe7d383393795b" exitCode=0
Jul 27 23:48:47.620177 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.620048    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"
Jul 27 23:48:47.620177 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.620082    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q" event={"ID":"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2","Type":"ContainerDied","Data":"bab9c44cab2e5fbb4e409c3000fdc4b267b9323c78b7a3f1c3fe7d383393795b"}
Jul 27 23:48:47.620177 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.620139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q" event={"ID":"d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2","Type":"ContainerDied","Data":"8cfd96c2c7ed1493d1338558e57aed401eccaf0dcb9b7960847fa072fe7bc906"}
Jul 27 23:48:47.620177 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.620170    2559 scope.go:117] "RemoveContainer" containerID="bab9c44cab2e5fbb4e409c3000fdc4b267b9323c78b7a3f1c3fe7d383393795b"
Jul 27 23:48:47.629525 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.629477    2559 scope.go:117] "RemoveContainer" containerID="7284eefb2190779307c1a7a31cb892086dd398801a698c5b23fc197934bfdc4d"
Jul 27 23:48:47.635531 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.635504    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"]
Jul 27 23:48:47.638444 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.638423    2559 scope.go:117] "RemoveContainer" containerID="bab9c44cab2e5fbb4e409c3000fdc4b267b9323c78b7a3f1c3fe7d383393795b"
Jul 27 23:48:47.638804 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:48:47.638750    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab9c44cab2e5fbb4e409c3000fdc4b267b9323c78b7a3f1c3fe7d383393795b\": container with ID starting with bab9c44cab2e5fbb4e409c3000fdc4b267b9323c78b7a3f1c3fe7d383393795b not found: ID does not exist" containerID="bab9c44cab2e5fbb4e409c3000fdc4b267b9323c78b7a3f1c3fe7d383393795b"
Jul 27 23:48:47.638896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.638806    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab9c44cab2e5fbb4e409c3000fdc4b267b9323c78b7a3f1c3fe7d383393795b"} err="failed to get container status \"bab9c44cab2e5fbb4e409c3000fdc4b267b9323c78b7a3f1c3fe7d383393795b\": rpc error: code = NotFound desc = could not find container \"bab9c44cab2e5fbb4e409c3000fdc4b267b9323c78b7a3f1c3fe7d383393795b\": container with ID starting with bab9c44cab2e5fbb4e409c3000fdc4b267b9323c78b7a3f1c3fe7d383393795b not found: ID does not exist"
Jul 27 23:48:47.638896 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.638843    2559 scope.go:117] "RemoveContainer" containerID="7284eefb2190779307c1a7a31cb892086dd398801a698c5b23fc197934bfdc4d"
Jul 27 23:48:47.639315 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:48:47.639229    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7284eefb2190779307c1a7a31cb892086dd398801a698c5b23fc197934bfdc4d\": container with ID starting with 7284eefb2190779307c1a7a31cb892086dd398801a698c5b23fc197934bfdc4d not found: ID does not exist" containerID="7284eefb2190779307c1a7a31cb892086dd398801a698c5b23fc197934bfdc4d"
Jul 27 23:48:47.639315 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.639267    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7284eefb2190779307c1a7a31cb892086dd398801a698c5b23fc197934bfdc4d"} err="failed to get container status \"7284eefb2190779307c1a7a31cb892086dd398801a698c5b23fc197934bfdc4d\": rpc error: code = NotFound desc = could not find container \"7284eefb2190779307c1a7a31cb892086dd398801a698c5b23fc197934bfdc4d\": container with ID starting with 7284eefb2190779307c1a7a31cb892086dd398801a698c5b23fc197934bfdc4d not found: ID does not exist"
Jul 27 23:48:47.641259 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:47.641234    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7355578b-openshift-default-96df69c84-46d6q"]
Jul 27 23:48:49.563215 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:48:49.563177    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" path="/var/lib/kubelet/pods/d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2/volumes"
Jul 27 23:49:09.337338 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:09.337250    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g"]
Jul 27 23:49:09.337744 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:09.337649    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" containerName="istio-proxy"
Jul 27 23:49:09.337744 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:09.337662    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" containerName="istio-proxy"
Jul 27 23:49:09.337744 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:09.337738    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" containerName="istio-proxy"
Jul 27 23:49:09.337744 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:09.337746    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" containerName="istio-proxy"
Jul 27 23:49:09.341255 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:09.341235    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g"
Jul 27 23:49:09.343201 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:09.343178    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 27 23:49:09.347227 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:09.347200    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g"]
Jul 27 23:49:09.385754 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:09.385718    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d213fdf4-4455-4096-a317-7ab1b6de8080-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-21351db5-bdw7g\" (UID: \"d213fdf4-4455-4096-a317-7ab1b6de8080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g"
Jul 27 23:49:09.486801 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:09.486756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d213fdf4-4455-4096-a317-7ab1b6de8080-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-21351db5-bdw7g\" (UID: \"d213fdf4-4455-4096-a317-7ab1b6de8080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g"
Jul 27 23:49:09.487427 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:09.487407    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d213fdf4-4455-4096-a317-7ab1b6de8080-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-21351db5-bdw7g\" (UID: \"d213fdf4-4455-4096-a317-7ab1b6de8080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g"
Jul 27 23:49:09.654952 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:09.654863    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g"
Jul 27 23:49:09.796447 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:09.796396    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g"]
Jul 27 23:49:09.799645 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:49:09.799622    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd213fdf4_4455_4096_a317_7ab1b6de8080.slice/crio-187ee37f6727f417c9bac63cfc50825e92dc68f949b05c6d6392f94e6686bc9d WatchSource:0}: Error finding container 187ee37f6727f417c9bac63cfc50825e92dc68f949b05c6d6392f94e6686bc9d: Status 404 returned error can't find the container with id 187ee37f6727f417c9bac63cfc50825e92dc68f949b05c6d6392f94e6686bc9d
Jul 27 23:49:10.743531 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:10.743442    2559 generic.go:356] "Generic (PLEG): container finished" podID="d213fdf4-4455-4096-a317-7ab1b6de8080" containerID="fd9354cfdcd5b67b9f3d7f2fb47981f8a59d9c28f999d3f8569441b087d1be37" exitCode=0
Jul 27 23:49:10.743531 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:10.743517    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g" event={"ID":"d213fdf4-4455-4096-a317-7ab1b6de8080","Type":"ContainerDied","Data":"fd9354cfdcd5b67b9f3d7f2fb47981f8a59d9c28f999d3f8569441b087d1be37"}
Jul 27 23:49:10.743950 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:10.743556    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g" event={"ID":"d213fdf4-4455-4096-a317-7ab1b6de8080","Type":"ContainerStarted","Data":"187ee37f6727f417c9bac63cfc50825e92dc68f949b05c6d6392f94e6686bc9d"}
Jul 27 23:49:11.887214 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:11.887184    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g"
Jul 27 23:49:11.910190 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:11.910155    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d213fdf4-4455-4096-a317-7ab1b6de8080-maas-api-service-ca\") pod \"d213fdf4-4455-4096-a317-7ab1b6de8080\" (UID: \"d213fdf4-4455-4096-a317-7ab1b6de8080\") "
Jul 27 23:49:11.910554 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:11.910531    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d213fdf4-4455-4096-a317-7ab1b6de8080-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d213fdf4-4455-4096-a317-7ab1b6de8080" (UID: "d213fdf4-4455-4096-a317-7ab1b6de8080"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:49:12.011698 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:12.011595    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d213fdf4-4455-4096-a317-7ab1b6de8080-maas-api-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:49:12.753915 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:12.753883    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g"
Jul 27 23:49:12.754095 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:12.753919    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g" event={"ID":"d213fdf4-4455-4096-a317-7ab1b6de8080","Type":"ContainerDied","Data":"187ee37f6727f417c9bac63cfc50825e92dc68f949b05c6d6392f94e6686bc9d"}
Jul 27 23:49:12.754095 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:12.753970    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="187ee37f6727f417c9bac63cfc50825e92dc68f949b05c6d6392f94e6686bc9d"
Jul 27 23:49:24.412540 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:24.412483    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g"]
Jul 27 23:49:24.414473 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:24.414446    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-21351db5-bdw7g"]
Jul 27 23:49:25.562862 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:49:25.562828    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d213fdf4-4455-4096-a317-7ab1b6de8080" path="/var/lib/kubelet/pods/d213fdf4-4455-4096-a317-7ab1b6de8080/volumes"
Jul 27 23:51:17.629908 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:17.629876    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:51:17.631998 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:17.631974    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:51:25.438455 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:25.438415    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j"]
Jul 27 23:51:25.438874 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:25.438859    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d213fdf4-4455-4096-a317-7ab1b6de8080" containerName="revoke-keys"
Jul 27 23:51:25.438874 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:25.438876    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d213fdf4-4455-4096-a317-7ab1b6de8080" containerName="revoke-keys"
Jul 27 23:51:25.438956 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:25.438897    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" containerName="istio-proxy"
Jul 27 23:51:25.438956 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:25.438902    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d39e6c14-5c7f-47e4-b3fb-f5891c6fb9a2" containerName="istio-proxy"
Jul 27 23:51:25.439027 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:25.438976    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d213fdf4-4455-4096-a317-7ab1b6de8080" containerName="revoke-keys"
Jul 27 23:51:25.442080 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:25.442065    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j"
Jul 27 23:51:25.444095 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:25.444074    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 27 23:51:25.447948 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:25.447892    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j"]
Jul 27 23:51:25.563599 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:25.563561    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f865bf1d-8047-4b02-9ad3-de2321995b77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j\" (UID: \"f865bf1d-8047-4b02-9ad3-de2321995b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j"
Jul 27 23:51:25.664556 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:25.664511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f865bf1d-8047-4b02-9ad3-de2321995b77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j\" (UID: \"f865bf1d-8047-4b02-9ad3-de2321995b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j"
Jul 27 23:51:25.665149 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:25.665129    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f865bf1d-8047-4b02-9ad3-de2321995b77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j\" (UID: \"f865bf1d-8047-4b02-9ad3-de2321995b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j"
Jul 27 23:51:25.755873 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:25.755782    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j"
Jul 27 23:51:25.899407 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:25.899359    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j"]
Jul 27 23:51:25.901881 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:51:25.901850    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf865bf1d_8047_4b02_9ad3_de2321995b77.slice/crio-aa20e7073900420106f2dc8f89e9daf595f434d55f7bc6896194bcb8476aaafd WatchSource:0}: Error finding container aa20e7073900420106f2dc8f89e9daf595f434d55f7bc6896194bcb8476aaafd: Status 404 returned error can't find the container with id aa20e7073900420106f2dc8f89e9daf595f434d55f7bc6896194bcb8476aaafd
Jul 27 23:51:26.430167 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:26.430061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j" event={"ID":"f865bf1d-8047-4b02-9ad3-de2321995b77","Type":"ContainerStarted","Data":"44fa6c4fd5ef97fbae3687268997be8fc09f63880ca278582b5fd0497a066c3a"}
Jul 27 23:51:26.430167 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:26.430133    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j" event={"ID":"f865bf1d-8047-4b02-9ad3-de2321995b77","Type":"ContainerStarted","Data":"aa20e7073900420106f2dc8f89e9daf595f434d55f7bc6896194bcb8476aaafd"}
Jul 27 23:51:26.457744 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:26.457688    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j" podStartSLOduration=1.4576730549999999 podStartE2EDuration="1.457673055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:51:26.455983719 +0000 UTC m=+2109.400214466" watchObservedRunningTime="2026-07-27 23:51:26.457673055 +0000 UTC m=+2109.401903801"
Jul 27 23:51:27.436971 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:27.436934    2559 generic.go:356] "Generic (PLEG): container finished" podID="f865bf1d-8047-4b02-9ad3-de2321995b77" containerID="44fa6c4fd5ef97fbae3687268997be8fc09f63880ca278582b5fd0497a066c3a" exitCode=0
Jul 27 23:51:27.437171 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:27.437028    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j" event={"ID":"f865bf1d-8047-4b02-9ad3-de2321995b77","Type":"ContainerDied","Data":"44fa6c4fd5ef97fbae3687268997be8fc09f63880ca278582b5fd0497a066c3a"}
Jul 27 23:51:28.575455 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:28.575432    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j"
Jul 27 23:51:28.693293 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:28.693202    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f865bf1d-8047-4b02-9ad3-de2321995b77-maas-api-service-ca\") pod \"f865bf1d-8047-4b02-9ad3-de2321995b77\" (UID: \"f865bf1d-8047-4b02-9ad3-de2321995b77\") "
Jul 27 23:51:28.693604 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:28.693584    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f865bf1d-8047-4b02-9ad3-de2321995b77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f865bf1d-8047-4b02-9ad3-de2321995b77" (UID: "f865bf1d-8047-4b02-9ad3-de2321995b77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:51:28.794382 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:28.794347    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f865bf1d-8047-4b02-9ad3-de2321995b77-maas-api-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:51:29.449182 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:29.449152    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j"
Jul 27 23:51:29.449360 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:29.449181    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j" event={"ID":"f865bf1d-8047-4b02-9ad3-de2321995b77","Type":"ContainerDied","Data":"aa20e7073900420106f2dc8f89e9daf595f434d55f7bc6896194bcb8476aaafd"}
Jul 27 23:51:29.449360 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:29.449228    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa20e7073900420106f2dc8f89e9daf595f434d55f7bc6896194bcb8476aaafd"
Jul 27 23:51:40.513654 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:40.513623    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j"]
Jul 27 23:51:40.516483 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:40.516458    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0050c7fc-wjg7j"]
Jul 27 23:51:41.563309 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:51:41.563274    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f865bf1d-8047-4b02-9ad3-de2321995b77" path="/var/lib/kubelet/pods/f865bf1d-8047-4b02-9ad3-de2321995b77/volumes"
Jul 27 23:52:23.439216 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.439179    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"]
Jul 27 23:52:23.439645 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.439578    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f865bf1d-8047-4b02-9ad3-de2321995b77" containerName="revoke-keys"
Jul 27 23:52:23.439645 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.439589    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f865bf1d-8047-4b02-9ad3-de2321995b77" containerName="revoke-keys"
Jul 27 23:52:23.439718 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.439650    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f865bf1d-8047-4b02-9ad3-de2321995b77" containerName="revoke-keys"
Jul 27 23:52:23.442898 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.442877    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.444977 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.444952    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-27ab6b49\"/\"openshift-service-ca.crt\""
Jul 27 23:52:23.445564 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.445542    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-27ab6b49\"/\"e2e-lifecycle-model-27ab6b49-kserve-self-signed-certs\""
Jul 27 23:52:23.445753 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.445730    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-27ab6b49\"/\"default-dockercfg-x5fk8\""
Jul 27 23:52:23.445841 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.445599    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-27ab6b49\"/\"kube-root-ca.crt\""
Jul 27 23:52:23.453281 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.453258    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"]
Jul 27 23:52:23.609365 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.609329    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c64696f-d11a-42bb-9f19-553db9b62b56-tls-certs\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.609365 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.609394    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-model-cache\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.609632 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.609448    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-tmp-dir\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.609632 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.609535    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tvn6\" (UniqueName: \"kubernetes.io/projected/4c64696f-d11a-42bb-9f19-553db9b62b56-kube-api-access-7tvn6\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.609632 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.609555    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-dshm\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.609632 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.609581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-home\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.710394 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.710271    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c64696f-d11a-42bb-9f19-553db9b62b56-tls-certs\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.710394 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.710320    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-model-cache\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.710635 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.710448    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-tmp-dir\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.710635 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.710552    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tvn6\" (UniqueName: \"kubernetes.io/projected/4c64696f-d11a-42bb-9f19-553db9b62b56-kube-api-access-7tvn6\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.710635 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.710574    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-dshm\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.710635 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.710598    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-home\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.710861 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.710756    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-model-cache\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.710861 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.710802    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-tmp-dir\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.710940 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.710912    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-home\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.712802 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.712785    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-dshm\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.713199 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.713179    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c64696f-d11a-42bb-9f19-553db9b62b56-tls-certs\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.718464 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.718416    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tvn6\" (UniqueName: \"kubernetes.io/projected/4c64696f-d11a-42bb-9f19-553db9b62b56-kube-api-access-7tvn6\") pod \"e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") " pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.752567 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.752537    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:23.892325 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:23.892297    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"]
Jul 27 23:52:23.894530 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:52:23.894503    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c64696f_d11a_42bb_9f19_553db9b62b56.slice/crio-51d6a43ce8a81b5246acc6e2112639f0122390eb4c1772f3e5a82a258df6d8eb WatchSource:0}: Error finding container 51d6a43ce8a81b5246acc6e2112639f0122390eb4c1772f3e5a82a258df6d8eb: Status 404 returned error can't find the container with id 51d6a43ce8a81b5246acc6e2112639f0122390eb4c1772f3e5a82a258df6d8eb
Jul 27 23:52:24.729212 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:24.729168    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk" event={"ID":"4c64696f-d11a-42bb-9f19-553db9b62b56","Type":"ContainerStarted","Data":"a0953d9c91f42f18e682b51aaa38673ba06211a339db348c98d53b59569e94da"}
Jul 27 23:52:24.729212 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:24.729211    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk" event={"ID":"4c64696f-d11a-42bb-9f19-553db9b62b56","Type":"ContainerStarted","Data":"51d6a43ce8a81b5246acc6e2112639f0122390eb4c1772f3e5a82a258df6d8eb"}
Jul 27 23:52:24.743866 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:24.743811    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk" podStartSLOduration=1.74379277 podStartE2EDuration="1.74379277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:52:24.743435919 +0000 UTC m=+2167.687666667" watchObservedRunningTime="2026-07-27 23:52:24.74379277 +0000 UTC m=+2167.688023519"
Jul 27 23:52:33.753547 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:33.753497    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:33.754021 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:33.753557    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:33.766069 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:33.766040    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:33.796198 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:33.796171    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:52:53.648040 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:53.648006    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw"]
Jul 27 23:52:53.651516 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:53.651499    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw"
Jul 27 23:52:53.653433 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:53.653413    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 27 23:52:53.657740 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:53.657661    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw"]
Jul 27 23:52:53.689697 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:53.689657    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810955ab-9aad-440e-81a7-7ef1ae08c04d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw\" (UID: \"810955ab-9aad-440e-81a7-7ef1ae08c04d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw"
Jul 27 23:52:53.790572 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:53.790524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810955ab-9aad-440e-81a7-7ef1ae08c04d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw\" (UID: \"810955ab-9aad-440e-81a7-7ef1ae08c04d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw"
Jul 27 23:52:53.791220 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:53.791196    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810955ab-9aad-440e-81a7-7ef1ae08c04d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw\" (UID: \"810955ab-9aad-440e-81a7-7ef1ae08c04d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw"
Jul 27 23:52:53.967203 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:53.967112    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw"
Jul 27 23:52:54.104802 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:54.104727    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw"]
Jul 27 23:52:54.107406 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:52:54.107353    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810955ab_9aad_440e_81a7_7ef1ae08c04d.slice/crio-a704a9cff8e8b41279085ce45f03f3b2d2a089783d1e8b2ce0607d8b91dddfbc WatchSource:0}: Error finding container a704a9cff8e8b41279085ce45f03f3b2d2a089783d1e8b2ce0607d8b91dddfbc: Status 404 returned error can't find the container with id a704a9cff8e8b41279085ce45f03f3b2d2a089783d1e8b2ce0607d8b91dddfbc
Jul 27 23:52:54.888220 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:54.888179    2559 generic.go:356] "Generic (PLEG): container finished" podID="810955ab-9aad-440e-81a7-7ef1ae08c04d" containerID="9186e9faf24840ee8c4827aca1aba1faf0969d75d52edf622655509afd84192d" exitCode=0
Jul 27 23:52:54.888644 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:54.888264    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw" event={"ID":"810955ab-9aad-440e-81a7-7ef1ae08c04d","Type":"ContainerDied","Data":"9186e9faf24840ee8c4827aca1aba1faf0969d75d52edf622655509afd84192d"}
Jul 27 23:52:54.888644 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:54.888315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw" event={"ID":"810955ab-9aad-440e-81a7-7ef1ae08c04d","Type":"ContainerStarted","Data":"a704a9cff8e8b41279085ce45f03f3b2d2a089783d1e8b2ce0607d8b91dddfbc"}
Jul 27 23:52:56.034920 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:56.034889    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw"
Jul 27 23:52:56.112844 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:56.112813    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810955ab-9aad-440e-81a7-7ef1ae08c04d-maas-api-service-ca\") pod \"810955ab-9aad-440e-81a7-7ef1ae08c04d\" (UID: \"810955ab-9aad-440e-81a7-7ef1ae08c04d\") "
Jul 27 23:52:56.113162 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:56.113139    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/810955ab-9aad-440e-81a7-7ef1ae08c04d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "810955ab-9aad-440e-81a7-7ef1ae08c04d" (UID: "810955ab-9aad-440e-81a7-7ef1ae08c04d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:52:56.213975 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:56.213890    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810955ab-9aad-440e-81a7-7ef1ae08c04d-maas-api-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:52:56.902786 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:56.902749    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw"
Jul 27 23:52:56.902786 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:56.902757    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw" event={"ID":"810955ab-9aad-440e-81a7-7ef1ae08c04d","Type":"ContainerDied","Data":"a704a9cff8e8b41279085ce45f03f3b2d2a089783d1e8b2ce0607d8b91dddfbc"}
Jul 27 23:52:56.903008 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:52:56.902842    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a704a9cff8e8b41279085ce45f03f3b2d2a089783d1e8b2ce0607d8b91dddfbc"
Jul 27 23:53:13.715621 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:13.715582    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw"]
Jul 27 23:53:13.718020 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:13.717219    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27ab6b49-m8czw"]
Jul 27 23:53:15.562932 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:15.562899    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="810955ab-9aad-440e-81a7-7ef1ae08c04d" path="/var/lib/kubelet/pods/810955ab-9aad-440e-81a7-7ef1ae08c04d/volumes"
Jul 27 23:53:21.763970 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:21.763933    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"]
Jul 27 23:53:36.784215 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:36.784149    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk" podUID="4c64696f-d11a-42bb-9f19-553db9b62b56" containerName="main" containerID="cri-o://a0953d9c91f42f18e682b51aaa38673ba06211a339db348c98d53b59569e94da" gracePeriod=45
Jul 27 23:53:37.042517 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.042448    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:53:37.100166 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.100132    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-dshm\") pod \"4c64696f-d11a-42bb-9f19-553db9b62b56\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") "
Jul 27 23:53:37.100328 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.100219    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c64696f-d11a-42bb-9f19-553db9b62b56-tls-certs\") pod \"4c64696f-d11a-42bb-9f19-553db9b62b56\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") "
Jul 27 23:53:37.100328 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.100282    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-model-cache\") pod \"4c64696f-d11a-42bb-9f19-553db9b62b56\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") "
Jul 27 23:53:37.100328 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.100309    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-tmp-dir\") pod \"4c64696f-d11a-42bb-9f19-553db9b62b56\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") "
Jul 27 23:53:37.100512 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.100362    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tvn6\" (UniqueName: \"kubernetes.io/projected/4c64696f-d11a-42bb-9f19-553db9b62b56-kube-api-access-7tvn6\") pod \"4c64696f-d11a-42bb-9f19-553db9b62b56\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") "
Jul 27 23:53:37.100512 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.100418    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-home\") pod \"4c64696f-d11a-42bb-9f19-553db9b62b56\" (UID: \"4c64696f-d11a-42bb-9f19-553db9b62b56\") "
Jul 27 23:53:37.100605 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.100558    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c64696f-d11a-42bb-9f19-553db9b62b56" (UID: "4c64696f-d11a-42bb-9f19-553db9b62b56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:53:37.100660 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.100614    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c64696f-d11a-42bb-9f19-553db9b62b56" (UID: "4c64696f-d11a-42bb-9f19-553db9b62b56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:53:37.100772 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.100744    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-home" (OuterVolumeSpecName: "home") pod "4c64696f-d11a-42bb-9f19-553db9b62b56" (UID: "4c64696f-d11a-42bb-9f19-553db9b62b56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:53:37.100907 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.100833    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-model-cache\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:53:37.100907 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.100854    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-tmp-dir\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:53:37.102863 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.102833    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c64696f-d11a-42bb-9f19-553db9b62b56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c64696f-d11a-42bb-9f19-553db9b62b56" (UID: "4c64696f-d11a-42bb-9f19-553db9b62b56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:53:37.102969 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.102953    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c64696f-d11a-42bb-9f19-553db9b62b56-kube-api-access-7tvn6" (OuterVolumeSpecName: "kube-api-access-7tvn6") pod "4c64696f-d11a-42bb-9f19-553db9b62b56" (UID: "4c64696f-d11a-42bb-9f19-553db9b62b56"). InnerVolumeSpecName "kube-api-access-7tvn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:53:37.102969 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.102952    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-dshm" (OuterVolumeSpecName: "dshm") pod "4c64696f-d11a-42bb-9f19-553db9b62b56" (UID: "4c64696f-d11a-42bb-9f19-553db9b62b56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:53:37.106591 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.106566    2559 generic.go:356] "Generic (PLEG): container finished" podID="4c64696f-d11a-42bb-9f19-553db9b62b56" containerID="a0953d9c91f42f18e682b51aaa38673ba06211a339db348c98d53b59569e94da" exitCode=0
Jul 27 23:53:37.106684 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.106625    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"
Jul 27 23:53:37.106684 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.106654    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk" event={"ID":"4c64696f-d11a-42bb-9f19-553db9b62b56","Type":"ContainerDied","Data":"a0953d9c91f42f18e682b51aaa38673ba06211a339db348c98d53b59569e94da"}
Jul 27 23:53:37.106766 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.106698    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk" event={"ID":"4c64696f-d11a-42bb-9f19-553db9b62b56","Type":"ContainerDied","Data":"51d6a43ce8a81b5246acc6e2112639f0122390eb4c1772f3e5a82a258df6d8eb"}
Jul 27 23:53:37.106766 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.106720    2559 scope.go:117] "RemoveContainer" containerID="a0953d9c91f42f18e682b51aaa38673ba06211a339db348c98d53b59569e94da"
Jul 27 23:53:37.123973 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.123955    2559 scope.go:117] "RemoveContainer" containerID="a0953d9c91f42f18e682b51aaa38673ba06211a339db348c98d53b59569e94da"
Jul 27 23:53:37.124239 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:53:37.124218    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0953d9c91f42f18e682b51aaa38673ba06211a339db348c98d53b59569e94da\": container with ID starting with a0953d9c91f42f18e682b51aaa38673ba06211a339db348c98d53b59569e94da not found: ID does not exist" containerID="a0953d9c91f42f18e682b51aaa38673ba06211a339db348c98d53b59569e94da"
Jul 27 23:53:37.124304 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.124247    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0953d9c91f42f18e682b51aaa38673ba06211a339db348c98d53b59569e94da"} err="failed to get container status \"a0953d9c91f42f18e682b51aaa38673ba06211a339db348c98d53b59569e94da\": rpc error: code = NotFound desc = could not find container \"a0953d9c91f42f18e682b51aaa38673ba06211a339db348c98d53b59569e94da\": container with ID starting with a0953d9c91f42f18e682b51aaa38673ba06211a339db348c98d53b59569e94da not found: ID does not exist"
Jul 27 23:53:37.132990 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.132970    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"]
Jul 27 23:53:37.135836 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.135812    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-27ab6b49/e2e-lifecycle-model-27ab6b49-kserve-5877c6d595-8g8lk"]
Jul 27 23:53:37.201851 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.201825    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-dshm\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:53:37.201851 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.201847    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c64696f-d11a-42bb-9f19-553db9b62b56-tls-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:53:37.202024 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.201857    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tvn6\" (UniqueName: \"kubernetes.io/projected/4c64696f-d11a-42bb-9f19-553db9b62b56-kube-api-access-7tvn6\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:53:37.202024 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.201868    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c64696f-d11a-42bb-9f19-553db9b62b56-home\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:53:37.564092 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:53:37.564060    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c64696f-d11a-42bb-9f19-553db9b62b56" path="/var/lib/kubelet/pods/4c64696f-d11a-42bb-9f19-553db9b62b56/volumes"
Jul 27 23:54:17.716868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:54:17.716835    2559 scope.go:117] "RemoveContainer" containerID="de81841c8b372b789f155f4473c6076e65413a65cdbe2f5315b96120c81ff26f"
Jul 27 23:55:07.120336 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.120300    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"]
Jul 27 23:55:07.120940 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.120910    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c64696f-d11a-42bb-9f19-553db9b62b56" containerName="main"
Jul 27 23:55:07.120940 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.120930    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c64696f-d11a-42bb-9f19-553db9b62b56" containerName="main"
Jul 27 23:55:07.121069 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.120942    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="810955ab-9aad-440e-81a7-7ef1ae08c04d" containerName="revoke-keys"
Jul 27 23:55:07.121069 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.120954    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="810955ab-9aad-440e-81a7-7ef1ae08c04d" containerName="revoke-keys"
Jul 27 23:55:07.121069 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.121052    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="810955ab-9aad-440e-81a7-7ef1ae08c04d" containerName="revoke-keys"
Jul 27 23:55:07.121069 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.121066    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c64696f-d11a-42bb-9f19-553db9b62b56" containerName="main"
Jul 27 23:55:07.124510 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.124484    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.126632 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.126611    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3ed4f0-openshift-default-dockercfg-5w8dm\""
Jul 27 23:55:07.134462 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.134442    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"]
Jul 27 23:55:07.195410 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.195356    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56e4693-02ee-45dd-a723-4af714c56927-istiod-ca-cert\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.195592 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.195444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-istio-data\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.195592 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.195472    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-credential-socket\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.195592 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.195488    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dlcn\" (UniqueName: \"kubernetes.io/projected/b56e4693-02ee-45dd-a723-4af714c56927-kube-api-access-7dlcn\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.195592 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.195554    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-istio-envoy\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.195592 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.195569    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-workload-socket\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.195592 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.195585    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56e4693-02ee-45dd-a723-4af714c56927-istio-podinfo\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.195805 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.195633    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-workload-certs\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.195805 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.195648    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56e4693-02ee-45dd-a723-4af714c56927-istio-token\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.296746 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.296708    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-workload-certs\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.296746 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.296748    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56e4693-02ee-45dd-a723-4af714c56927-istio-token\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.296988 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.296814    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56e4693-02ee-45dd-a723-4af714c56927-istiod-ca-cert\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.296988 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.296863    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-istio-data\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.296988 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.296896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-credential-socket\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.296988 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.296923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dlcn\" (UniqueName: \"kubernetes.io/projected/b56e4693-02ee-45dd-a723-4af714c56927-kube-api-access-7dlcn\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.296988 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.296979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-istio-envoy\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.297261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.297005    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-workload-socket\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.297261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.297030    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56e4693-02ee-45dd-a723-4af714c56927-istio-podinfo\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.297261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.297166    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-workload-certs\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.297451 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.297292    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-istio-data\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.297517 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.297474    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-credential-socket\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.297517 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.297495    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-workload-socket\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.297675 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.297654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56e4693-02ee-45dd-a723-4af714c56927-istiod-ca-cert\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.300185 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.300155    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-istio-envoy\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.300326 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.300304    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56e4693-02ee-45dd-a723-4af714c56927-istio-podinfo\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.305037 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.305003    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56e4693-02ee-45dd-a723-4af714c56927-istio-token\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.305156 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.305137    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dlcn\" (UniqueName: \"kubernetes.io/projected/b56e4693-02ee-45dd-a723-4af714c56927-kube-api-access-7dlcn\") pod \"e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") " pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.436947 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.436866    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:07.578842 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.578817    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"]
Jul 27 23:55:07.581910 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:55:07.581872    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56e4693_02ee_45dd_a723_4af714c56927.slice/crio-4d1e3729f0293d398c64980bfd1e87c0a73cd7f729d4cabb9c08a7dab0560449 WatchSource:0}: Error finding container 4d1e3729f0293d398c64980bfd1e87c0a73cd7f729d4cabb9c08a7dab0560449: Status 404 returned error can't find the container with id 4d1e3729f0293d398c64980bfd1e87c0a73cd7f729d4cabb9c08a7dab0560449
Jul 27 23:55:07.583626 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.583595    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:55:07.583724 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.583655    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:55:07.583724 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:07.583686    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:55:08.580880 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:08.580846    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv" event={"ID":"b56e4693-02ee-45dd-a723-4af714c56927","Type":"ContainerStarted","Data":"231832a22b69bdd9d3f619ff0078d3ed1697c3d30eb67248a94f1d7c5d5a89a9"}
Jul 27 23:55:08.581391 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:08.580895    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv" event={"ID":"b56e4693-02ee-45dd-a723-4af714c56927","Type":"ContainerStarted","Data":"4d1e3729f0293d398c64980bfd1e87c0a73cd7f729d4cabb9c08a7dab0560449"}
Jul 27 23:55:08.599395 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:08.599323    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv" podStartSLOduration=1.599306629 podStartE2EDuration="1.599306629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:55:08.598281434 +0000 UTC m=+2331.542512179" watchObservedRunningTime="2026-07-27 23:55:08.599306629 +0000 UTC m=+2331.543537376"
Jul 27 23:55:09.438037 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:09.437996    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:09.442858 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:09.442834    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:09.585699 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:09.585673    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:09.586661 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:09.586634    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:17.741031 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:17.740988    2559 scope.go:117] "RemoveContainer" containerID="fd9354cfdcd5b67b9f3d7f2fb47981f8a59d9c28f999d3f8569441b087d1be37"
Jul 27 23:55:22.657535 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:22.657500    2559 generic.go:356] "Generic (PLEG): container finished" podID="b56e4693-02ee-45dd-a723-4af714c56927" containerID="231832a22b69bdd9d3f619ff0078d3ed1697c3d30eb67248a94f1d7c5d5a89a9" exitCode=0
Jul 27 23:55:22.657946 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:22.657576    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv" event={"ID":"b56e4693-02ee-45dd-a723-4af714c56927","Type":"ContainerDied","Data":"231832a22b69bdd9d3f619ff0078d3ed1697c3d30eb67248a94f1d7c5d5a89a9"}
Jul 27 23:55:22.658006 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:22.657989    2559 scope.go:117] "RemoveContainer" containerID="231832a22b69bdd9d3f619ff0078d3ed1697c3d30eb67248a94f1d7c5d5a89a9"
Jul 27 23:55:22.659134 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:22.659108    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:55:22.659254 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:22.659187    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:55:22.659254 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:22.659231    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:55:23.664361 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:23.664322    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv" event={"ID":"b56e4693-02ee-45dd-a723-4af714c56927","Type":"ContainerStarted","Data":"12c75307c188736345dc631050481904d35e52f034f8cf2f3d295040544217e2"}
Jul 27 23:55:24.438019 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:24.437987    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:24.438824 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:24.438795    2559 patch_prober.go:28] interesting pod/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Jul 27 23:55:24.438936 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:24.438848    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv" podUID="b56e4693-02ee-45dd-a723-4af714c56927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Jul 27 23:55:24.585722 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:24.585692    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:25.437821 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:25.437784    2559 patch_prober.go:28] interesting pod/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Jul 27 23:55:25.438210 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:25.437839    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv" podUID="b56e4693-02ee-45dd-a723-4af714c56927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Jul 27 23:55:26.437981 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:26.437940    2559 patch_prober.go:28] interesting pod/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Jul 27 23:55:26.438405 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:26.438008    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv" podUID="b56e4693-02ee-45dd-a723-4af714c56927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Jul 27 23:55:27.437734 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:27.437695    2559 patch_prober.go:28] interesting pod/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Jul 27 23:55:27.437920 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:27.437750    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv" podUID="b56e4693-02ee-45dd-a723-4af714c56927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Jul 27 23:55:28.441520 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:28.441486    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:28.442363 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:28.442346    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:55:35.440741 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.440705    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"]
Jul 27 23:55:35.444743 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.444723    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.446835 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.446809    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-05a1fc\"/\"openshift-service-ca.crt\""
Jul 27 23:55:35.446930 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.446879    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-05a1fc\"/\"kube-root-ca.crt\""
Jul 27 23:55:35.447604 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.447585    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-05a1fc\"/\"default-dockercfg-8psk6\""
Jul 27 23:55:35.447994 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.447624    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-05a1fc\"/\"test-model-05a1fc-kserve-self-signed-certs\""
Jul 27 23:55:35.450183 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.450160    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"]
Jul 27 23:55:35.566253 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.566221    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmdwd\" (UniqueName: \"kubernetes.io/projected/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-kube-api-access-lmdwd\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.566434 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.566259    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-tmp-dir\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.566434 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.566325    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-dshm\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.566434 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.566357    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-model-cache\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.566548 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.566440    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-tls-certs\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.566548 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.566488    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-home\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.667304 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.667264    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-home\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.667525 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.667328    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdwd\" (UniqueName: \"kubernetes.io/projected/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-kube-api-access-lmdwd\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.667525 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.667361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-tmp-dir\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.667525 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.667470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-dshm\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.667706 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.667617    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-model-cache\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.667772 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.667718    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-home\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.667772 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.667724    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-tls-certs\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.667868 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.667851    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-tmp-dir\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.668063 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.668041    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-model-cache\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.669786 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.669762    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-dshm\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.670238 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.670220    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-tls-certs\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.674469 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.674444    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdwd\" (UniqueName: \"kubernetes.io/projected/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-kube-api-access-lmdwd\") pod \"test-model-05a1fc-kserve-5554d868f7-558rc\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") " pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.755469 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.755363    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:35.892635 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:35.892606    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"]
Jul 27 23:55:35.895273 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:55:35.895244    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cb45fc0_7561_40a3_884b_8e9d8829c0c8.slice/crio-11d89314a516c74fe2f1e0671d085dbf61f651cd17c6029836c3d759cd8837a0 WatchSource:0}: Error finding container 11d89314a516c74fe2f1e0671d085dbf61f651cd17c6029836c3d759cd8837a0: Status 404 returned error can't find the container with id 11d89314a516c74fe2f1e0671d085dbf61f651cd17c6029836c3d759cd8837a0
Jul 27 23:55:36.732597 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:36.732561    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc" event={"ID":"5cb45fc0-7561-40a3-884b-8e9d8829c0c8","Type":"ContainerStarted","Data":"9be0361c783a772169063b2e3a71cc13ad38a2fdaf12c896014cc73e1291ce4f"}
Jul 27 23:55:36.733019 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:36.732614    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc" event={"ID":"5cb45fc0-7561-40a3-884b-8e9d8829c0c8","Type":"ContainerStarted","Data":"11d89314a516c74fe2f1e0671d085dbf61f651cd17c6029836c3d759cd8837a0"}
Jul 27 23:55:36.752204 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:36.752148    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc" podStartSLOduration=1.752129387 podStartE2EDuration="1.752129387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:55:36.75165916 +0000 UTC m=+2359.695889919" watchObservedRunningTime="2026-07-27 23:55:36.752129387 +0000 UTC m=+2359.696360135"
Jul 27 23:55:45.755916 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:45.755877    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:45.756343 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:45.755934    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:45.768526 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:45.768501    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:55:45.788827 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:55:45.788800    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:56:17.661931 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:17.661903    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:56:17.664052 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:17.664027    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 27 23:56:50.425023 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:50.424933    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"]
Jul 27 23:56:51.739362 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:51.739319    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p"]
Jul 27 23:56:51.743414 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:51.743391    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p"
Jul 27 23:56:51.745536 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:51.745515    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 27 23:56:51.751126 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:51.751096    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p"]
Jul 27 23:56:51.881486 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:51.881447    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6742ec-d08a-4a01-b9a8-f61e4a314a89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p\" (UID: \"da6742ec-d08a-4a01-b9a8-f61e4a314a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p"
Jul 27 23:56:51.982345 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:51.982302    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6742ec-d08a-4a01-b9a8-f61e4a314a89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p\" (UID: \"da6742ec-d08a-4a01-b9a8-f61e4a314a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p"
Jul 27 23:56:51.982938 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:51.982917    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6742ec-d08a-4a01-b9a8-f61e4a314a89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p\" (UID: \"da6742ec-d08a-4a01-b9a8-f61e4a314a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p"
Jul 27 23:56:52.055437 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:52.055403    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p"
Jul 27 23:56:52.195704 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:52.195670    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p"]
Jul 27 23:56:52.197961 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:56:52.197932    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6742ec_d08a_4a01_b9a8_f61e4a314a89.slice/crio-c933ad0ccbeeb0220957849a27f9f3f44c8412d97c7f4b6004129ec6d3983bde WatchSource:0}: Error finding container c933ad0ccbeeb0220957849a27f9f3f44c8412d97c7f4b6004129ec6d3983bde: Status 404 returned error can't find the container with id c933ad0ccbeeb0220957849a27f9f3f44c8412d97c7f4b6004129ec6d3983bde
Jul 27 23:56:53.125725 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:53.125690    2559 generic.go:356] "Generic (PLEG): container finished" podID="da6742ec-d08a-4a01-b9a8-f61e4a314a89" containerID="7f1754ffa44905115da8a0c2e7cb1de3982b1ddd61c7565c905b868984e8e650" exitCode=0
Jul 27 23:56:53.126148 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:53.125754    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p" event={"ID":"da6742ec-d08a-4a01-b9a8-f61e4a314a89","Type":"ContainerDied","Data":"7f1754ffa44905115da8a0c2e7cb1de3982b1ddd61c7565c905b868984e8e650"}
Jul 27 23:56:53.126148 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:53.125796    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p" event={"ID":"da6742ec-d08a-4a01-b9a8-f61e4a314a89","Type":"ContainerStarted","Data":"c933ad0ccbeeb0220957849a27f9f3f44c8412d97c7f4b6004129ec6d3983bde"}
Jul 27 23:56:54.269624 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:54.269597    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p"
Jul 27 23:56:54.405179 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:54.405094    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6742ec-d08a-4a01-b9a8-f61e4a314a89-maas-api-service-ca\") pod \"da6742ec-d08a-4a01-b9a8-f61e4a314a89\" (UID: \"da6742ec-d08a-4a01-b9a8-f61e4a314a89\") "
Jul 27 23:56:54.405465 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:54.405441    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6742ec-d08a-4a01-b9a8-f61e4a314a89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da6742ec-d08a-4a01-b9a8-f61e4a314a89" (UID: "da6742ec-d08a-4a01-b9a8-f61e4a314a89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:56:54.506081 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:54.506051    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6742ec-d08a-4a01-b9a8-f61e4a314a89-maas-api-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:56:55.136417 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:55.136361    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p" event={"ID":"da6742ec-d08a-4a01-b9a8-f61e4a314a89","Type":"ContainerDied","Data":"c933ad0ccbeeb0220957849a27f9f3f44c8412d97c7f4b6004129ec6d3983bde"}
Jul 27 23:56:55.136589 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:55.136441    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c933ad0ccbeeb0220957849a27f9f3f44c8412d97c7f4b6004129ec6d3983bde"
Jul 27 23:56:55.136589 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:56:55.136397    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p"
Jul 27 23:57:05.445359 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.445314    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc" podUID="5cb45fc0-7561-40a3-884b-8e9d8829c0c8" containerName="main" containerID="cri-o://9be0361c783a772169063b2e3a71cc13ad38a2fdaf12c896014cc73e1291ce4f" gracePeriod=45
Jul 27 23:57:05.730061 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.730018    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:57:05.916065 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.916027    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-tls-certs\") pod \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") "
Jul 27 23:57:05.916265 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.916090    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmdwd\" (UniqueName: \"kubernetes.io/projected/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-kube-api-access-lmdwd\") pod \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") "
Jul 27 23:57:05.916265 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.916119    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-tmp-dir\") pod \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") "
Jul 27 23:57:05.916265 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.916135    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-dshm\") pod \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") "
Jul 27 23:57:05.916265 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.916256    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-model-cache\") pod \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") "
Jul 27 23:57:05.916537 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.916334    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-home\") pod \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\" (UID: \"5cb45fc0-7561-40a3-884b-8e9d8829c0c8\") "
Jul 27 23:57:05.916537 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.916491    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5cb45fc0-7561-40a3-884b-8e9d8829c0c8" (UID: "5cb45fc0-7561-40a3-884b-8e9d8829c0c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:57:05.916665 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.916584    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "5cb45fc0-7561-40a3-884b-8e9d8829c0c8" (UID: "5cb45fc0-7561-40a3-884b-8e9d8829c0c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:57:05.916665 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.916603    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-home" (OuterVolumeSpecName: "home") pod "5cb45fc0-7561-40a3-884b-8e9d8829c0c8" (UID: "5cb45fc0-7561-40a3-884b-8e9d8829c0c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:57:05.916773 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.916720    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-home\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:05.916773 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.916740    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-tmp-dir\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:05.916773 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.916753    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-model-cache\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:05.918545 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.918517    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-kube-api-access-lmdwd" (OuterVolumeSpecName: "kube-api-access-lmdwd") pod "5cb45fc0-7561-40a3-884b-8e9d8829c0c8" (UID: "5cb45fc0-7561-40a3-884b-8e9d8829c0c8"). InnerVolumeSpecName "kube-api-access-lmdwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:57:05.918661 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.918519    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5cb45fc0-7561-40a3-884b-8e9d8829c0c8" (UID: "5cb45fc0-7561-40a3-884b-8e9d8829c0c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:57:05.918661 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:05.918554    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-dshm" (OuterVolumeSpecName: "dshm") pod "5cb45fc0-7561-40a3-884b-8e9d8829c0c8" (UID: "5cb45fc0-7561-40a3-884b-8e9d8829c0c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:57:06.017237 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:06.017165    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-tls-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:06.017237 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:06.017190    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmdwd\" (UniqueName: \"kubernetes.io/projected/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-kube-api-access-lmdwd\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:06.017237 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:06.017200    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cb45fc0-7561-40a3-884b-8e9d8829c0c8-dshm\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:06.194006 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:06.193968    2559 generic.go:356] "Generic (PLEG): container finished" podID="5cb45fc0-7561-40a3-884b-8e9d8829c0c8" containerID="9be0361c783a772169063b2e3a71cc13ad38a2fdaf12c896014cc73e1291ce4f" exitCode=0
Jul 27 23:57:06.194203 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:06.194014    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"
Jul 27 23:57:06.194203 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:06.194038    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc" event={"ID":"5cb45fc0-7561-40a3-884b-8e9d8829c0c8","Type":"ContainerDied","Data":"9be0361c783a772169063b2e3a71cc13ad38a2fdaf12c896014cc73e1291ce4f"}
Jul 27 23:57:06.194203 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:06.194076    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc" event={"ID":"5cb45fc0-7561-40a3-884b-8e9d8829c0c8","Type":"ContainerDied","Data":"11d89314a516c74fe2f1e0671d085dbf61f651cd17c6029836c3d759cd8837a0"}
Jul 27 23:57:06.194203 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:06.194099    2559 scope.go:117] "RemoveContainer" containerID="9be0361c783a772169063b2e3a71cc13ad38a2fdaf12c896014cc73e1291ce4f"
Jul 27 23:57:06.203889 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:06.203868    2559 scope.go:117] "RemoveContainer" containerID="9be0361c783a772169063b2e3a71cc13ad38a2fdaf12c896014cc73e1291ce4f"
Jul 27 23:57:06.204153 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:57:06.204134    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be0361c783a772169063b2e3a71cc13ad38a2fdaf12c896014cc73e1291ce4f\": container with ID starting with 9be0361c783a772169063b2e3a71cc13ad38a2fdaf12c896014cc73e1291ce4f not found: ID does not exist" containerID="9be0361c783a772169063b2e3a71cc13ad38a2fdaf12c896014cc73e1291ce4f"
Jul 27 23:57:06.204200 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:06.204164    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be0361c783a772169063b2e3a71cc13ad38a2fdaf12c896014cc73e1291ce4f"} err="failed to get container status \"9be0361c783a772169063b2e3a71cc13ad38a2fdaf12c896014cc73e1291ce4f\": rpc error: code = NotFound desc = could not find container \"9be0361c783a772169063b2e3a71cc13ad38a2fdaf12c896014cc73e1291ce4f\": container with ID starting with 9be0361c783a772169063b2e3a71cc13ad38a2fdaf12c896014cc73e1291ce4f not found: ID does not exist"
Jul 27 23:57:06.214508 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:06.214470    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"]
Jul 27 23:57:06.218831 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:06.218811    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-05a1fc/test-model-05a1fc-kserve-5554d868f7-558rc"]
Jul 27 23:57:06.816897 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:06.816861    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p"]
Jul 27 23:57:06.819170 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:06.819149    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-05a1fc-8kz5p"]
Jul 27 23:57:07.563478 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:07.563447    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cb45fc0-7561-40a3-884b-8e9d8829c0c8" path="/var/lib/kubelet/pods/5cb45fc0-7561-40a3-884b-8e9d8829c0c8/volumes"
Jul 27 23:57:07.563785 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:07.563773    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6742ec-d08a-4a01-b9a8-f61e4a314a89" path="/var/lib/kubelet/pods/da6742ec-d08a-4a01-b9a8-f61e4a314a89/volumes"
Jul 27 23:57:13.495335 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.495294    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g"]
Jul 27 23:57:13.495821 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.495742    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6742ec-d08a-4a01-b9a8-f61e4a314a89" containerName="revoke-keys"
Jul 27 23:57:13.495821 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.495754    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6742ec-d08a-4a01-b9a8-f61e4a314a89" containerName="revoke-keys"
Jul 27 23:57:13.495821 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.495770    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cb45fc0-7561-40a3-884b-8e9d8829c0c8" containerName="main"
Jul 27 23:57:13.495821 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.495779    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cb45fc0-7561-40a3-884b-8e9d8829c0c8" containerName="main"
Jul 27 23:57:13.495960 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.495845    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cb45fc0-7561-40a3-884b-8e9d8829c0c8" containerName="main"
Jul 27 23:57:13.495960 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.495857    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6742ec-d08a-4a01-b9a8-f61e4a314a89" containerName="revoke-keys"
Jul 27 23:57:13.500482 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.500459    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g"
Jul 27 23:57:13.502575 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.502555    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 27 23:57:13.505339 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.505315    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g"]
Jul 27 23:57:13.582095 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.582064    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g\" (UID: \"2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g"
Jul 27 23:57:13.682826 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.682779    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g\" (UID: \"2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g"
Jul 27 23:57:13.683434 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.683414    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g\" (UID: \"2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g"
Jul 27 23:57:13.814142 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.814101    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g"
Jul 27 23:57:13.952090 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:13.952062    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g"]
Jul 27 23:57:13.954535 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:57:13.954505    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9ea73b_7463_4ec4_b6e5_68aedcfe8bd6.slice/crio-922c842ef6a743ce17eab2cbb3192a5f3b012433844edcdb88bd5b32a7611e6a WatchSource:0}: Error finding container 922c842ef6a743ce17eab2cbb3192a5f3b012433844edcdb88bd5b32a7611e6a: Status 404 returned error can't find the container with id 922c842ef6a743ce17eab2cbb3192a5f3b012433844edcdb88bd5b32a7611e6a
Jul 27 23:57:14.237242 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:14.237211    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g" event={"ID":"2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6","Type":"ContainerStarted","Data":"922c842ef6a743ce17eab2cbb3192a5f3b012433844edcdb88bd5b32a7611e6a"}
Jul 27 23:57:15.243717 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:15.243630    2559 generic.go:356] "Generic (PLEG): container finished" podID="2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6" containerID="db4d02e0eecafee83f3da7ee89c2af93501b688047ce8afd8190f13ccf5f85a0" exitCode=0
Jul 27 23:57:15.243717 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:15.243705    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g" event={"ID":"2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6","Type":"ContainerDied","Data":"db4d02e0eecafee83f3da7ee89c2af93501b688047ce8afd8190f13ccf5f85a0"}
Jul 27 23:57:16.383598 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:16.383575    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g"
Jul 27 23:57:16.409650 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:16.409616    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6-maas-api-service-ca\") pod \"2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6\" (UID: \"2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6\") "
Jul 27 23:57:16.409960 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:16.409933    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6" (UID: "2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:57:16.510717 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:16.510624    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6-maas-api-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:17.254293 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:17.254260    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g" event={"ID":"2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6","Type":"ContainerDied","Data":"922c842ef6a743ce17eab2cbb3192a5f3b012433844edcdb88bd5b32a7611e6a"}
Jul 27 23:57:17.254494 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:17.254309    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="922c842ef6a743ce17eab2cbb3192a5f3b012433844edcdb88bd5b32a7611e6a"
Jul 27 23:57:17.254494 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:17.254267    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g"
Jul 27 23:57:28.561216 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:28.561173    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g"]
Jul 27 23:57:28.563554 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:28.563529    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3ed4f0-dvh7g"]
Jul 27 23:57:29.562728 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:29.562696    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6" path="/var/lib/kubelet/pods/2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6/volumes"
Jul 27 23:57:34.927357 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:34.927323    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"]
Jul 27 23:57:34.927850 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:34.927606    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv" podUID="b56e4693-02ee-45dd-a723-4af714c56927" containerName="istio-proxy" containerID="cri-o://12c75307c188736345dc631050481904d35e52f034f8cf2f3d295040544217e2" gracePeriod=30
Jul 27 23:57:40.194540 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.194482    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:57:40.331261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.331222    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-istio-data\") pod \"b56e4693-02ee-45dd-a723-4af714c56927\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") "
Jul 27 23:57:40.331261 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.331271    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-workload-certs\") pod \"b56e4693-02ee-45dd-a723-4af714c56927\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") "
Jul 27 23:57:40.331520 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.331344    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-credential-socket\") pod \"b56e4693-02ee-45dd-a723-4af714c56927\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") "
Jul 27 23:57:40.331520 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.331363    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56e4693-02ee-45dd-a723-4af714c56927-istio-podinfo\") pod \"b56e4693-02ee-45dd-a723-4af714c56927\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") "
Jul 27 23:57:40.331520 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.331416    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-workload-socket\") pod \"b56e4693-02ee-45dd-a723-4af714c56927\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") "
Jul 27 23:57:40.331520 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.331440    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56e4693-02ee-45dd-a723-4af714c56927-istio-token\") pod \"b56e4693-02ee-45dd-a723-4af714c56927\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") "
Jul 27 23:57:40.331520 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.331470    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dlcn\" (UniqueName: \"kubernetes.io/projected/b56e4693-02ee-45dd-a723-4af714c56927-kube-api-access-7dlcn\") pod \"b56e4693-02ee-45dd-a723-4af714c56927\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") "
Jul 27 23:57:40.331520 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.331501    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-istio-envoy\") pod \"b56e4693-02ee-45dd-a723-4af714c56927\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") "
Jul 27 23:57:40.331819 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.331547    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56e4693-02ee-45dd-a723-4af714c56927-istiod-ca-cert\") pod \"b56e4693-02ee-45dd-a723-4af714c56927\" (UID: \"b56e4693-02ee-45dd-a723-4af714c56927\") "
Jul 27 23:57:40.331819 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.331695    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b56e4693-02ee-45dd-a723-4af714c56927" (UID: "b56e4693-02ee-45dd-a723-4af714c56927"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:57:40.331819 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.331754    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b56e4693-02ee-45dd-a723-4af714c56927" (UID: "b56e4693-02ee-45dd-a723-4af714c56927"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:57:40.331984 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.331873    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-credential-socket\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:40.331984 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.331895    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-workload-socket\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:40.331984 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.331918    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b56e4693-02ee-45dd-a723-4af714c56927" (UID: "b56e4693-02ee-45dd-a723-4af714c56927"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:57:40.332432 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.332386    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-istio-data" (OuterVolumeSpecName: "istio-data") pod "b56e4693-02ee-45dd-a723-4af714c56927" (UID: "b56e4693-02ee-45dd-a723-4af714c56927"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:57:40.332432 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.332400    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56e4693-02ee-45dd-a723-4af714c56927-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b56e4693-02ee-45dd-a723-4af714c56927" (UID: "b56e4693-02ee-45dd-a723-4af714c56927"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:57:40.333904 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.333871    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56e4693-02ee-45dd-a723-4af714c56927-kube-api-access-7dlcn" (OuterVolumeSpecName: "kube-api-access-7dlcn") pod "b56e4693-02ee-45dd-a723-4af714c56927" (UID: "b56e4693-02ee-45dd-a723-4af714c56927"). InnerVolumeSpecName "kube-api-access-7dlcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:57:40.333986 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.333896    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56e4693-02ee-45dd-a723-4af714c56927-istio-token" (OuterVolumeSpecName: "istio-token") pod "b56e4693-02ee-45dd-a723-4af714c56927" (UID: "b56e4693-02ee-45dd-a723-4af714c56927"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:57:40.333986 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.333893    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b56e4693-02ee-45dd-a723-4af714c56927-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b56e4693-02ee-45dd-a723-4af714c56927" (UID: "b56e4693-02ee-45dd-a723-4af714c56927"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 23:57:40.333986 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.333941    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b56e4693-02ee-45dd-a723-4af714c56927" (UID: "b56e4693-02ee-45dd-a723-4af714c56927"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:57:40.383290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.383262    2559 generic.go:356] "Generic (PLEG): container finished" podID="b56e4693-02ee-45dd-a723-4af714c56927" containerID="12c75307c188736345dc631050481904d35e52f034f8cf2f3d295040544217e2" exitCode=0
Jul 27 23:57:40.383403 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.383310    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"
Jul 27 23:57:40.383403 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.383344    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv" event={"ID":"b56e4693-02ee-45dd-a723-4af714c56927","Type":"ContainerDied","Data":"12c75307c188736345dc631050481904d35e52f034f8cf2f3d295040544217e2"}
Jul 27 23:57:40.383487 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.383431    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv" event={"ID":"b56e4693-02ee-45dd-a723-4af714c56927","Type":"ContainerDied","Data":"4d1e3729f0293d398c64980bfd1e87c0a73cd7f729d4cabb9c08a7dab0560449"}
Jul 27 23:57:40.383487 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.383463    2559 scope.go:117] "RemoveContainer" containerID="12c75307c188736345dc631050481904d35e52f034f8cf2f3d295040544217e2"
Jul 27 23:57:40.393290 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.393274    2559 scope.go:117] "RemoveContainer" containerID="231832a22b69bdd9d3f619ff0078d3ed1697c3d30eb67248a94f1d7c5d5a89a9"
Jul 27 23:57:40.401271 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.401239    2559 scope.go:117] "RemoveContainer" containerID="12c75307c188736345dc631050481904d35e52f034f8cf2f3d295040544217e2"
Jul 27 23:57:40.401961 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:57:40.401825    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c75307c188736345dc631050481904d35e52f034f8cf2f3d295040544217e2\": container with ID starting with 12c75307c188736345dc631050481904d35e52f034f8cf2f3d295040544217e2 not found: ID does not exist" containerID="12c75307c188736345dc631050481904d35e52f034f8cf2f3d295040544217e2"
Jul 27 23:57:40.401961 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.401908    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c75307c188736345dc631050481904d35e52f034f8cf2f3d295040544217e2"} err="failed to get container status \"12c75307c188736345dc631050481904d35e52f034f8cf2f3d295040544217e2\": rpc error: code = NotFound desc = could not find container \"12c75307c188736345dc631050481904d35e52f034f8cf2f3d295040544217e2\": container with ID starting with 12c75307c188736345dc631050481904d35e52f034f8cf2f3d295040544217e2 not found: ID does not exist"
Jul 27 23:57:40.401961 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.401932    2559 scope.go:117] "RemoveContainer" containerID="231832a22b69bdd9d3f619ff0078d3ed1697c3d30eb67248a94f1d7c5d5a89a9"
Jul 27 23:57:40.402647 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:57:40.402627    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231832a22b69bdd9d3f619ff0078d3ed1697c3d30eb67248a94f1d7c5d5a89a9\": container with ID starting with 231832a22b69bdd9d3f619ff0078d3ed1697c3d30eb67248a94f1d7c5d5a89a9 not found: ID does not exist" containerID="231832a22b69bdd9d3f619ff0078d3ed1697c3d30eb67248a94f1d7c5d5a89a9"
Jul 27 23:57:40.402731 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.402655    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231832a22b69bdd9d3f619ff0078d3ed1697c3d30eb67248a94f1d7c5d5a89a9"} err="failed to get container status \"231832a22b69bdd9d3f619ff0078d3ed1697c3d30eb67248a94f1d7c5d5a89a9\": rpc error: code = NotFound desc = could not find container \"231832a22b69bdd9d3f619ff0078d3ed1697c3d30eb67248a94f1d7c5d5a89a9\": container with ID starting with 231832a22b69bdd9d3f619ff0078d3ed1697c3d30eb67248a94f1d7c5d5a89a9 not found: ID does not exist"
Jul 27 23:57:40.403972 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.403953    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"]
Jul 27 23:57:40.407894 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.407874    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3ed4f0-openshift-default-678bcf7c68-knbtv"]
Jul 27 23:57:40.433349 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.433332    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56e4693-02ee-45dd-a723-4af714c56927-istio-podinfo\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:40.433441 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.433351    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56e4693-02ee-45dd-a723-4af714c56927-istio-token\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:40.433441 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.433360    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dlcn\" (UniqueName: \"kubernetes.io/projected/b56e4693-02ee-45dd-a723-4af714c56927-kube-api-access-7dlcn\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:40.433441 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.433392    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-istio-envoy\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:40.433441 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.433402    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56e4693-02ee-45dd-a723-4af714c56927-istiod-ca-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:40.433441 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.433409    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-istio-data\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:40.433441 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:40.433418    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56e4693-02ee-45dd-a723-4af714c56927-workload-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:57:41.562687 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:41.562655    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b56e4693-02ee-45dd-a723-4af714c56927" path="/var/lib/kubelet/pods/b56e4693-02ee-45dd-a723-4af714c56927/volumes"
Jul 27 23:57:44.039513 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.039478    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"]
Jul 27 23:57:44.039957 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.039882    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6" containerName="revoke-keys"
Jul 27 23:57:44.039957 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.039898    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6" containerName="revoke-keys"
Jul 27 23:57:44.039957 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.039920    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56e4693-02ee-45dd-a723-4af714c56927" containerName="istio-proxy"
Jul 27 23:57:44.039957 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.039929    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56e4693-02ee-45dd-a723-4af714c56927" containerName="istio-proxy"
Jul 27 23:57:44.039957 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.039939    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56e4693-02ee-45dd-a723-4af714c56927" containerName="istio-proxy"
Jul 27 23:57:44.039957 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.039947    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56e4693-02ee-45dd-a723-4af714c56927" containerName="istio-proxy"
Jul 27 23:57:44.040193 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.040056    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e9ea73b-7463-4ec4-b6e5-68aedcfe8bd6" containerName="revoke-keys"
Jul 27 23:57:44.040193 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.040067    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56e4693-02ee-45dd-a723-4af714c56927" containerName="istio-proxy"
Jul 27 23:57:44.044666 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.044639    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.046891 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.046869    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5d597b-openshift-default-dockercfg-t4vmd\""
Jul 27 23:57:44.053532 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.053508    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"]
Jul 27 23:57:44.168677 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.168646    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g569c\" (UniqueName: \"kubernetes.io/projected/9c46785a-f040-4dc7-8879-c9e14ddc4a81-kube-api-access-g569c\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.168677 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.168692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-data\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.168953 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.168752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-token\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.168953 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.168833    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-workload-socket\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.168953 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.168866    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istiod-ca-cert\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.168953 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.168921    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-credential-socket\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.169153 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.168955    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-envoy\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.169153 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.168986    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-workload-certs\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.169153 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.169029    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-podinfo\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.269773 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.269737    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istiod-ca-cert\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.269773 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.269778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-credential-socket\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.270107 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.269798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-envoy\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.270107 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.269816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-workload-certs\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.270107 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.269838    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-podinfo\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.270107 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.269867    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g569c\" (UniqueName: \"kubernetes.io/projected/9c46785a-f040-4dc7-8879-c9e14ddc4a81-kube-api-access-g569c\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.270107 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.269902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-data\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.270107 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.269935    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-token\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.270107 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.269989    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-workload-socket\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.270699 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.270186    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-credential-socket\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.270699 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.270307    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-workload-certs\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.270699 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.270470    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-workload-socket\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.270699 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.270580    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istiod-ca-cert\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.270699 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.270581    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-data\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.272266 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.272242    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-envoy\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.272418 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.272318    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-podinfo\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.276929 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.276904    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-token\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.277874 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.277854    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g569c\" (UniqueName: \"kubernetes.io/projected/9c46785a-f040-4dc7-8879-c9e14ddc4a81-kube-api-access-g569c\") pod \"e2e-api-b-5d597b-openshift-default-758b857746-7nh4w\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") " pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.356129 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.356041    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:44.494353 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.494327    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"]
Jul 27 23:57:44.497328 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:57:44.497299    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c46785a_f040_4dc7_8879_c9e14ddc4a81.slice/crio-dc35137f76514baf9e77e99b1854d1cda2915102d31f1db39ca8e4f0f04beb8a WatchSource:0}: Error finding container dc35137f76514baf9e77e99b1854d1cda2915102d31f1db39ca8e4f0f04beb8a: Status 404 returned error can't find the container with id dc35137f76514baf9e77e99b1854d1cda2915102d31f1db39ca8e4f0f04beb8a
Jul 27 23:57:44.499899 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.499868    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:57:44.499999 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.499936    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:57:44.499999 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:44.499964    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:57:45.418552 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:45.418514    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w" event={"ID":"9c46785a-f040-4dc7-8879-c9e14ddc4a81","Type":"ContainerStarted","Data":"c9bf7a0e50d8a1380361e7b73344dca55b2a7d71987dd5ac07e76e36e922930e"}
Jul 27 23:57:45.419035 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:45.418568    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w" event={"ID":"9c46785a-f040-4dc7-8879-c9e14ddc4a81","Type":"ContainerStarted","Data":"dc35137f76514baf9e77e99b1854d1cda2915102d31f1db39ca8e4f0f04beb8a"}
Jul 27 23:57:45.439545 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:45.439501    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w" podStartSLOduration=1.439486263 podStartE2EDuration="1.439486263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:57:45.436991964 +0000 UTC m=+2488.381222711" watchObservedRunningTime="2026-07-27 23:57:45.439486263 +0000 UTC m=+2488.383717009"
Jul 27 23:57:46.356784 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:46.356750    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:46.361680 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:46.361656    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:46.422841 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:46.422816    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:46.423677 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:46.423655    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:57:59.702873 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:59.702836    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2"]
Jul 27 23:57:59.703532 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:59.703512    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56e4693-02ee-45dd-a723-4af714c56927" containerName="istio-proxy"
Jul 27 23:57:59.706640 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:59.706623    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2"
Jul 27 23:57:59.708774 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:59.708754    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 27 23:57:59.712545 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:59.712523    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2"]
Jul 27 23:57:59.810969 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:59.810933    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54580dc9-ff8f-49e1-a1d8-67d2d06ae636-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2\" (UID: \"54580dc9-ff8f-49e1-a1d8-67d2d06ae636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2"
Jul 27 23:57:59.911412 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:59.911367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54580dc9-ff8f-49e1-a1d8-67d2d06ae636-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2\" (UID: \"54580dc9-ff8f-49e1-a1d8-67d2d06ae636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2"
Jul 27 23:57:59.911998 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:57:59.911979    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54580dc9-ff8f-49e1-a1d8-67d2d06ae636-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2\" (UID: \"54580dc9-ff8f-49e1-a1d8-67d2d06ae636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2"
Jul 27 23:58:00.022871 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:00.022780    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2"
Jul 27 23:58:00.179581 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:00.179555    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2"]
Jul 27 23:58:00.182041 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:58:00.182013    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54580dc9_ff8f_49e1_a1d8_67d2d06ae636.slice/crio-4cccb4f4d2ebdff77521872729b90daa965aa80e97317977861532928f13f7cd WatchSource:0}: Error finding container 4cccb4f4d2ebdff77521872729b90daa965aa80e97317977861532928f13f7cd: Status 404 returned error can't find the container with id 4cccb4f4d2ebdff77521872729b90daa965aa80e97317977861532928f13f7cd
Jul 27 23:58:00.496171 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:00.496137    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2" event={"ID":"54580dc9-ff8f-49e1-a1d8-67d2d06ae636","Type":"ContainerStarted","Data":"4cccb4f4d2ebdff77521872729b90daa965aa80e97317977861532928f13f7cd"}
Jul 27 23:58:01.504092 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:01.504054    2559 generic.go:356] "Generic (PLEG): container finished" podID="54580dc9-ff8f-49e1-a1d8-67d2d06ae636" containerID="a45e799c13fb1057bad60a77a9c7a721af431a8863e5e0d43913cfdb92b250b4" exitCode=0
Jul 27 23:58:01.504628 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:01.504101    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2" event={"ID":"54580dc9-ff8f-49e1-a1d8-67d2d06ae636","Type":"ContainerDied","Data":"a45e799c13fb1057bad60a77a9c7a721af431a8863e5e0d43913cfdb92b250b4"}
Jul 27 23:58:02.465546 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:02.465492    2559 patch_prober.go:28] interesting pod/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 23:58:02.465760 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:02.465562    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w" podUID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 23:58:02.659775 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:02.659751    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2"
Jul 27 23:58:02.743384 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:02.743310    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54580dc9-ff8f-49e1-a1d8-67d2d06ae636-maas-api-service-ca\") pod \"54580dc9-ff8f-49e1-a1d8-67d2d06ae636\" (UID: \"54580dc9-ff8f-49e1-a1d8-67d2d06ae636\") "
Jul 27 23:58:02.743636 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:02.743616    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54580dc9-ff8f-49e1-a1d8-67d2d06ae636-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54580dc9-ff8f-49e1-a1d8-67d2d06ae636" (UID: "54580dc9-ff8f-49e1-a1d8-67d2d06ae636"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:58:02.844932 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:02.844872    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54580dc9-ff8f-49e1-a1d8-67d2d06ae636-maas-api-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:58:03.516515 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:03.516482    2559 generic.go:356] "Generic (PLEG): container finished" podID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerID="c9bf7a0e50d8a1380361e7b73344dca55b2a7d71987dd5ac07e76e36e922930e" exitCode=0
Jul 27 23:58:03.516684 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:03.516553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w" event={"ID":"9c46785a-f040-4dc7-8879-c9e14ddc4a81","Type":"ContainerDied","Data":"c9bf7a0e50d8a1380361e7b73344dca55b2a7d71987dd5ac07e76e36e922930e"}
Jul 27 23:58:03.517042 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:03.517020    2559 scope.go:117] "RemoveContainer" containerID="c9bf7a0e50d8a1380361e7b73344dca55b2a7d71987dd5ac07e76e36e922930e"
Jul 27 23:58:03.517889 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:03.517867    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2" event={"ID":"54580dc9-ff8f-49e1-a1d8-67d2d06ae636","Type":"ContainerDied","Data":"4cccb4f4d2ebdff77521872729b90daa965aa80e97317977861532928f13f7cd"}
Jul 27 23:58:03.517952 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:03.517900    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2"
Jul 27 23:58:03.517952 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:03.517908    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cccb4f4d2ebdff77521872729b90daa965aa80e97317977861532928f13f7cd"
Jul 27 23:58:03.518198 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:03.518165    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:58:03.518313 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:03.518240    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:58:03.518313 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:03.518281    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:58:04.526018 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:04.525982    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w" event={"ID":"9c46785a-f040-4dc7-8879-c9e14ddc4a81","Type":"ContainerStarted","Data":"8281ebe6a06898790ba9a0f89cbab51d6a73e7fbf1cae7283f4ea7a457b6eb76"}
Jul 27 23:58:05.357282 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:05.357252    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:58:05.357482 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:05.357448    2559 patch_prober.go:28] interesting pod/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 27 23:58:05.357537 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:05.357491    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w" podUID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 27 23:58:06.356825 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:06.356790    2559 patch_prober.go:28] interesting pod/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 27 23:58:06.357216 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:06.356842    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w" podUID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 27 23:58:07.356653 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:07.356613    2559 patch_prober.go:28] interesting pod/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 27 23:58:07.356897 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:07.356671    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w" podUID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 27 23:58:08.357106 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:08.357068    2559 patch_prober.go:28] interesting pod/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 27 23:58:08.357523 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:08.357123    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w" podUID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 27 23:58:09.360453 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:09.360423    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:58:09.360837 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:09.360690    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:58:09.361325 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:09.361309    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:58:17.780509 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:17.780476    2559 scope.go:117] "RemoveContainer" containerID="44fa6c4fd5ef97fbae3687268997be8fc09f63880ca278582b5fd0497a066c3a"
Jul 27 23:58:19.773401 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:19.773348    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2"]
Jul 27 23:58:19.776460 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:19.776435    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f3414a-4lrx2"]
Jul 27 23:58:21.563215 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:21.563177    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54580dc9-ff8f-49e1-a1d8-67d2d06ae636" path="/var/lib/kubelet/pods/54580dc9-ff8f-49e1-a1d8-67d2d06ae636/volumes"
Jul 27 23:58:26.280618 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:26.280587    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh"]
Jul 27 23:58:26.281039 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:26.281024    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54580dc9-ff8f-49e1-a1d8-67d2d06ae636" containerName="revoke-keys"
Jul 27 23:58:26.281095 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:26.281042    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="54580dc9-ff8f-49e1-a1d8-67d2d06ae636" containerName="revoke-keys"
Jul 27 23:58:26.281143 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:26.281127    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="54580dc9-ff8f-49e1-a1d8-67d2d06ae636" containerName="revoke-keys"
Jul 27 23:58:26.284213 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:26.284198    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh"
Jul 27 23:58:26.286135 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:26.286112    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 27 23:58:26.290362 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:26.290338    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh"]
Jul 27 23:58:26.364798 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:26.364767    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b94bc34-11c6-405a-bcd2-c45a42faff20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5d597b-nppzh\" (UID: \"5b94bc34-11c6-405a-bcd2-c45a42faff20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh"
Jul 27 23:58:26.465678 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:26.465649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b94bc34-11c6-405a-bcd2-c45a42faff20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5d597b-nppzh\" (UID: \"5b94bc34-11c6-405a-bcd2-c45a42faff20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh"
Jul 27 23:58:26.466236 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:26.466217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b94bc34-11c6-405a-bcd2-c45a42faff20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5d597b-nppzh\" (UID: \"5b94bc34-11c6-405a-bcd2-c45a42faff20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh"
Jul 27 23:58:26.598393 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:26.598289    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh"
Jul 27 23:58:26.735615 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:26.735587    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh"]
Jul 27 23:58:26.737998 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:58:26.737965    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b94bc34_11c6_405a_bcd2_c45a42faff20.slice/crio-1aef3a9aed5c8eb2b25d2ee866abd6f5aebf1e77c6a2137b3617aa9f2f65ca55 WatchSource:0}: Error finding container 1aef3a9aed5c8eb2b25d2ee866abd6f5aebf1e77c6a2137b3617aa9f2f65ca55: Status 404 returned error can't find the container with id 1aef3a9aed5c8eb2b25d2ee866abd6f5aebf1e77c6a2137b3617aa9f2f65ca55
Jul 27 23:58:27.648581 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:27.648542    2559 generic.go:356] "Generic (PLEG): container finished" podID="5b94bc34-11c6-405a-bcd2-c45a42faff20" containerID="e4023252dc4e95c51cced8fe4c608e6c26d2ff0d0ad636f5fce423ed372e0f96" exitCode=0
Jul 27 23:58:27.649004 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:27.648596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh" event={"ID":"5b94bc34-11c6-405a-bcd2-c45a42faff20","Type":"ContainerDied","Data":"e4023252dc4e95c51cced8fe4c608e6c26d2ff0d0ad636f5fce423ed372e0f96"}
Jul 27 23:58:27.649004 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:27.648651    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh" event={"ID":"5b94bc34-11c6-405a-bcd2-c45a42faff20","Type":"ContainerStarted","Data":"1aef3a9aed5c8eb2b25d2ee866abd6f5aebf1e77c6a2137b3617aa9f2f65ca55"}
Jul 27 23:58:28.790274 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:28.790248    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh"
Jul 27 23:58:28.885906 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:28.885873    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b94bc34-11c6-405a-bcd2-c45a42faff20-maas-api-service-ca\") pod \"5b94bc34-11c6-405a-bcd2-c45a42faff20\" (UID: \"5b94bc34-11c6-405a-bcd2-c45a42faff20\") "
Jul 27 23:58:28.886239 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:28.886215    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b94bc34-11c6-405a-bcd2-c45a42faff20-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b94bc34-11c6-405a-bcd2-c45a42faff20" (UID: "5b94bc34-11c6-405a-bcd2-c45a42faff20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:58:28.987614 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:28.987532    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b94bc34-11c6-405a-bcd2-c45a42faff20-maas-api-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:58:29.659590 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:29.659554    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh" event={"ID":"5b94bc34-11c6-405a-bcd2-c45a42faff20","Type":"ContainerDied","Data":"1aef3a9aed5c8eb2b25d2ee866abd6f5aebf1e77c6a2137b3617aa9f2f65ca55"}
Jul 27 23:58:29.659771 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:29.659609    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aef3a9aed5c8eb2b25d2ee866abd6f5aebf1e77c6a2137b3617aa9f2f65ca55"
Jul 27 23:58:29.659771 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:29.659563    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh"
Jul 27 23:58:46.345679 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:46.345635    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh"]
Jul 27 23:58:46.347807 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:46.347782    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5d597b-nppzh"]
Jul 27 23:58:47.563508 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:47.563476    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b94bc34-11c6-405a-bcd2-c45a42faff20" path="/var/lib/kubelet/pods/5b94bc34-11c6-405a-bcd2-c45a42faff20/volumes"
Jul 27 23:58:52.794077 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:52.794039    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"]
Jul 27 23:58:52.794710 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:52.794289    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w" podUID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerName="istio-proxy" containerID="cri-o://8281ebe6a06898790ba9a0f89cbab51d6a73e7fbf1cae7283f4ea7a457b6eb76" gracePeriod=30
Jul 27 23:58:54.362983 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:54.362948    2559 patch_prober.go:28] interesting pod/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 23:58:54.363351 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:54.363007    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w" podUID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 23:58:58.048046 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.048022    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:58:58.144865 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.144776    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-credential-socket\") pod \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") "
Jul 27 23:58:58.144865 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.144819    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g569c\" (UniqueName: \"kubernetes.io/projected/9c46785a-f040-4dc7-8879-c9e14ddc4a81-kube-api-access-g569c\") pod \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") "
Jul 27 23:58:58.144865 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.144835    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-envoy\") pod \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") "
Jul 27 23:58:58.145160 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.144870    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-workload-certs\") pod \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") "
Jul 27 23:58:58.145160 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.144910    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istiod-ca-cert\") pod \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") "
Jul 27 23:58:58.145160 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.144973    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-podinfo\") pod \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") "
Jul 27 23:58:58.145160 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.145000    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-workload-socket\") pod \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") "
Jul 27 23:58:58.145160 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.145023    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-token\") pod \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") "
Jul 27 23:58:58.145160 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.145054    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-data\") pod \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\" (UID: \"9c46785a-f040-4dc7-8879-c9e14ddc4a81\") "
Jul 27 23:58:58.145160 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.145105    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c46785a-f040-4dc7-8879-c9e14ddc4a81" (UID: "9c46785a-f040-4dc7-8879-c9e14ddc4a81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:58:58.145581 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.145298    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c46785a-f040-4dc7-8879-c9e14ddc4a81" (UID: "9c46785a-f040-4dc7-8879-c9e14ddc4a81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:58:58.145581 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.145352    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c46785a-f040-4dc7-8879-c9e14ddc4a81" (UID: "9c46785a-f040-4dc7-8879-c9e14ddc4a81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:58:58.145581 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.145484    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c46785a-f040-4dc7-8879-c9e14ddc4a81" (UID: "9c46785a-f040-4dc7-8879-c9e14ddc4a81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:58:58.145750 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.145637    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-credential-socket\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:58:58.145750 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.145662    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-workload-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:58:58.145750 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.145677    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istiod-ca-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:58:58.145750 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.145720    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-workload-socket\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:58:58.146153 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.146127    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c46785a-f040-4dc7-8879-c9e14ddc4a81" (UID: "9c46785a-f040-4dc7-8879-c9e14ddc4a81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:58:58.147351 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.147319    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c46785a-f040-4dc7-8879-c9e14ddc4a81" (UID: "9c46785a-f040-4dc7-8879-c9e14ddc4a81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:58:58.147351 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.147341    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c46785a-f040-4dc7-8879-c9e14ddc4a81" (UID: "9c46785a-f040-4dc7-8879-c9e14ddc4a81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:58:58.147569 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.147413    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c46785a-f040-4dc7-8879-c9e14ddc4a81" (UID: "9c46785a-f040-4dc7-8879-c9e14ddc4a81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 23:58:58.147569 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.147526    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c46785a-f040-4dc7-8879-c9e14ddc4a81-kube-api-access-g569c" (OuterVolumeSpecName: "kube-api-access-g569c") pod "9c46785a-f040-4dc7-8879-c9e14ddc4a81" (UID: "9c46785a-f040-4dc7-8879-c9e14ddc4a81"). InnerVolumeSpecName "kube-api-access-g569c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:58:58.247033 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.247006    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-podinfo\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:58:58.247033 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.247032    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-token\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:58:58.247033 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.247040    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-data\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:58:58.247275 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.247049    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g569c\" (UniqueName: \"kubernetes.io/projected/9c46785a-f040-4dc7-8879-c9e14ddc4a81-kube-api-access-g569c\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:58:58.247275 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.247058    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c46785a-f040-4dc7-8879-c9e14ddc4a81-istio-envoy\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 27 23:58:58.810846 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.810814    2559 generic.go:356] "Generic (PLEG): container finished" podID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerID="8281ebe6a06898790ba9a0f89cbab51d6a73e7fbf1cae7283f4ea7a457b6eb76" exitCode=0
Jul 27 23:58:58.811028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.810890    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"
Jul 27 23:58:58.811028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.810901    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w" event={"ID":"9c46785a-f040-4dc7-8879-c9e14ddc4a81","Type":"ContainerDied","Data":"8281ebe6a06898790ba9a0f89cbab51d6a73e7fbf1cae7283f4ea7a457b6eb76"}
Jul 27 23:58:58.811028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.810967    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w" event={"ID":"9c46785a-f040-4dc7-8879-c9e14ddc4a81","Type":"ContainerDied","Data":"dc35137f76514baf9e77e99b1854d1cda2915102d31f1db39ca8e4f0f04beb8a"}
Jul 27 23:58:58.811028 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.810999    2559 scope.go:117] "RemoveContainer" containerID="8281ebe6a06898790ba9a0f89cbab51d6a73e7fbf1cae7283f4ea7a457b6eb76"
Jul 27 23:58:58.820751 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.820734    2559 scope.go:117] "RemoveContainer" containerID="c9bf7a0e50d8a1380361e7b73344dca55b2a7d71987dd5ac07e76e36e922930e"
Jul 27 23:58:58.829624 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.829593    2559 scope.go:117] "RemoveContainer" containerID="8281ebe6a06898790ba9a0f89cbab51d6a73e7fbf1cae7283f4ea7a457b6eb76"
Jul 27 23:58:58.829926 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:58:58.829903    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8281ebe6a06898790ba9a0f89cbab51d6a73e7fbf1cae7283f4ea7a457b6eb76\": container with ID starting with 8281ebe6a06898790ba9a0f89cbab51d6a73e7fbf1cae7283f4ea7a457b6eb76 not found: ID does not exist" containerID="8281ebe6a06898790ba9a0f89cbab51d6a73e7fbf1cae7283f4ea7a457b6eb76"
Jul 27 23:58:58.830011 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.829937    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8281ebe6a06898790ba9a0f89cbab51d6a73e7fbf1cae7283f4ea7a457b6eb76"} err="failed to get container status \"8281ebe6a06898790ba9a0f89cbab51d6a73e7fbf1cae7283f4ea7a457b6eb76\": rpc error: code = NotFound desc = could not find container \"8281ebe6a06898790ba9a0f89cbab51d6a73e7fbf1cae7283f4ea7a457b6eb76\": container with ID starting with 8281ebe6a06898790ba9a0f89cbab51d6a73e7fbf1cae7283f4ea7a457b6eb76 not found: ID does not exist"
Jul 27 23:58:58.830011 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.829961    2559 scope.go:117] "RemoveContainer" containerID="c9bf7a0e50d8a1380361e7b73344dca55b2a7d71987dd5ac07e76e36e922930e"
Jul 27 23:58:58.830228 ip-10-0-142-53 kubenswrapper[2559]: E0727 23:58:58.830207    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9bf7a0e50d8a1380361e7b73344dca55b2a7d71987dd5ac07e76e36e922930e\": container with ID starting with c9bf7a0e50d8a1380361e7b73344dca55b2a7d71987dd5ac07e76e36e922930e not found: ID does not exist" containerID="c9bf7a0e50d8a1380361e7b73344dca55b2a7d71987dd5ac07e76e36e922930e"
Jul 27 23:58:58.830275 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.830235    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9bf7a0e50d8a1380361e7b73344dca55b2a7d71987dd5ac07e76e36e922930e"} err="failed to get container status \"c9bf7a0e50d8a1380361e7b73344dca55b2a7d71987dd5ac07e76e36e922930e\": rpc error: code = NotFound desc = could not find container \"c9bf7a0e50d8a1380361e7b73344dca55b2a7d71987dd5ac07e76e36e922930e\": container with ID starting with c9bf7a0e50d8a1380361e7b73344dca55b2a7d71987dd5ac07e76e36e922930e not found: ID does not exist"
Jul 27 23:58:58.831580 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.831561    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"]
Jul 27 23:58:58.834561 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:58.834542    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5d597b-openshift-default-758b857746-7nh4w"]
Jul 27 23:58:59.564336 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:58:59.564302    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" path="/var/lib/kubelet/pods/9c46785a-f040-4dc7-8879-c9e14ddc4a81/volumes"
Jul 27 23:59:01.400568 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.400534    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"]
Jul 27 23:59:01.400978 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.400964    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerName="istio-proxy"
Jul 27 23:59:01.400978 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.400980    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerName="istio-proxy"
Jul 27 23:59:01.401061 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.400999    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b94bc34-11c6-405a-bcd2-c45a42faff20" containerName="revoke-keys"
Jul 27 23:59:01.401061 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.401005    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b94bc34-11c6-405a-bcd2-c45a42faff20" containerName="revoke-keys"
Jul 27 23:59:01.401061 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.401060    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b94bc34-11c6-405a-bcd2-c45a42faff20" containerName="revoke-keys"
Jul 27 23:59:01.401164 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.401068    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerName="istio-proxy"
Jul 27 23:59:01.401164 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.401078    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerName="istio-proxy"
Jul 27 23:59:01.406044 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.406022    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.408477 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.408455    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-4a8122-openshift-default-dockercfg-dvjzf\""
Jul 27 23:59:01.415715 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.415695    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"]
Jul 27 23:59:01.479256 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.479228    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-credential-socket\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.479256 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.479258    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-data\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.479525 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.479277    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istiod-ca-cert\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.479525 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.479439    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-token\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.479525 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.479489    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfbsp\" (UniqueName: \"kubernetes.io/projected/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-kube-api-access-nfbsp\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.479698 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.479524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-workload-certs\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.479698 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.479562    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-workload-socket\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.479698 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.479593    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-envoy\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.479698 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.479640    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-podinfo\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.580507 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.580477    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-token\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.580701 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.580514    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfbsp\" (UniqueName: \"kubernetes.io/projected/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-kube-api-access-nfbsp\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.580701 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.580535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-workload-certs\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.580701 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.580664    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-workload-socket\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.580701 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.580701    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-envoy\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.580934 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.580751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-podinfo\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.580934 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.580823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-credential-socket\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.580934 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.580849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-data\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.580934 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.580873    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istiod-ca-cert\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.581142 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.580940    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-workload-certs\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.581142 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.581005    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-workload-socket\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.581260 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.581145    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-data\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.581260 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.581145    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-credential-socket\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.581652 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.581629    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istiod-ca-cert\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.583045 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.583021    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-envoy\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.583480 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.583458    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-podinfo\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.587727 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.587696    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-token\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.587923 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.587906    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfbsp\" (UniqueName: \"kubernetes.io/projected/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-kube-api-access-nfbsp\") pod \"e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") " pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.718009 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.717932    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:01.868504 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.868314    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"]
Jul 27 23:59:01.870866 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:59:01.870834    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa6fdee_2279_43ac_94fb_1a8d3b657a24.slice/crio-2f33ef24f7b297d0d9c07d4993150a3c4ec13f1f9fca501a433b68df2b589a94 WatchSource:0}: Error finding container 2f33ef24f7b297d0d9c07d4993150a3c4ec13f1f9fca501a433b68df2b589a94: Status 404 returned error can't find the container with id 2f33ef24f7b297d0d9c07d4993150a3c4ec13f1f9fca501a433b68df2b589a94
Jul 27 23:59:01.873029 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.872990    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:59:01.873134 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.873072    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:59:01.873134 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:01.873116    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:59:02.835350 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:02.835301    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn" event={"ID":"3aa6fdee-2279-43ac-94fb-1a8d3b657a24","Type":"ContainerStarted","Data":"7935c265bc6e52545fd68562743a09c16104c2c66c552e986fc40e4440f1ddea"}
Jul 27 23:59:02.835765 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:02.835389    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn" event={"ID":"3aa6fdee-2279-43ac-94fb-1a8d3b657a24","Type":"ContainerStarted","Data":"2f33ef24f7b297d0d9c07d4993150a3c4ec13f1f9fca501a433b68df2b589a94"}
Jul 27 23:59:02.862824 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:02.862772    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn" podStartSLOduration=1.8627580940000001 podStartE2EDuration="1.862758094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:59:02.860833453 +0000 UTC m=+2565.805064200" watchObservedRunningTime="2026-07-27 23:59:02.862758094 +0000 UTC m=+2565.806988899"
Jul 27 23:59:03.718863 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:03.718826    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:03.723865 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:03.723839    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:03.841035 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:03.841004    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:03.841916 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:03.841895    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:16.956142 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:16.956110    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"]
Jul 27 23:59:16.956687 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:16.956537    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerName="istio-proxy"
Jul 27 23:59:16.956687 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:16.956555    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c46785a-f040-4dc7-8879-c9e14ddc4a81" containerName="istio-proxy"
Jul 27 23:59:16.960024 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:16.960005    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:16.962296 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:16.962276    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-42d790\"/\"openshift-service-ca.crt\""
Jul 27 23:59:16.963030 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:16.963012    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-42d790\"/\"kube-root-ca.crt\""
Jul 27 23:59:16.963089 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:16.963039    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-42d790\"/\"auth-test-model-42d790-kserve-self-signed-certs\""
Jul 27 23:59:16.963428 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:16.963365    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-42d790\"/\"default-dockercfg-6nkz8\""
Jul 27 23:59:16.969746 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:16.969725    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"]
Jul 27 23:59:17.034030 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.033992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-dshm\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.034195 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.034042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32da263-a70a-40f3-b5f8-974eb417f67b-tls-certs\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.034195 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.034091    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-home\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.034195 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.034108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6w8z\" (UniqueName: \"kubernetes.io/projected/e32da263-a70a-40f3-b5f8-974eb417f67b-kube-api-access-z6w8z\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.034195 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.034136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-model-cache\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.034195 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.034189    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-tmp-dir\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.134763 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.134730    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-home\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.134763 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.134768    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6w8z\" (UniqueName: \"kubernetes.io/projected/e32da263-a70a-40f3-b5f8-974eb417f67b-kube-api-access-z6w8z\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.134980 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.134809    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-model-cache\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.134980 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.134853    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-tmp-dir\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.134980 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.134896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-dshm\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.134980 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.134951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32da263-a70a-40f3-b5f8-974eb417f67b-tls-certs\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.135247 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.135227    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-home\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.135319 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.135264    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-model-cache\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.135412 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.135338    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-tmp-dir\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.137834 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.137670    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-dshm\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.137834 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.137803    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32da263-a70a-40f3-b5f8-974eb417f67b-tls-certs\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.142264 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.142240    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6w8z\" (UniqueName: \"kubernetes.io/projected/e32da263-a70a-40f3-b5f8-974eb417f67b-kube-api-access-z6w8z\") pod \"auth-test-model-42d790-kserve-798d49d5c9-9sjh8\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") " pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.271345 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.271258    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:17.460536 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.460509    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"]
Jul 27 23:59:17.463101 ip-10-0-142-53 kubenswrapper[2559]: W0727 23:59:17.463070    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode32da263_a70a_40f3_b5f8_974eb417f67b.slice/crio-0b937f8a3639f96769b65e1428fd64a9b1b92f9fb4542bd69900361513bd9810 WatchSource:0}: Error finding container 0b937f8a3639f96769b65e1428fd64a9b1b92f9fb4542bd69900361513bd9810: Status 404 returned error can't find the container with id 0b937f8a3639f96769b65e1428fd64a9b1b92f9fb4542bd69900361513bd9810
Jul 27 23:59:17.805277 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.805241    2559 scope.go:117] "RemoveContainer" containerID="9186e9faf24840ee8c4827aca1aba1faf0969d75d52edf622655509afd84192d"
Jul 27 23:59:17.925829 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.925795    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8" event={"ID":"e32da263-a70a-40f3-b5f8-974eb417f67b","Type":"ContainerStarted","Data":"07effb0fcd91b9d6d978a919ad611bb2f6ce019cea6ccf779f9fa1dc54a3789c"}
Jul 27 23:59:17.925995 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.925839    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8" event={"ID":"e32da263-a70a-40f3-b5f8-974eb417f67b","Type":"ContainerStarted","Data":"0b937f8a3639f96769b65e1428fd64a9b1b92f9fb4542bd69900361513bd9810"}
Jul 27 23:59:17.940904 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:17.940857    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8" podStartSLOduration=1.9408430399999999 podStartE2EDuration="1.94084304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:59:17.939611097 +0000 UTC m=+2580.883841847" watchObservedRunningTime="2026-07-27 23:59:17.94084304 +0000 UTC m=+2580.885073786"
Jul 27 23:59:19.882528 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:19.882486    2559 patch_prober.go:28] interesting pod/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 23:59:19.882997 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:19.882553    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn" podUID="3aa6fdee-2279-43ac-94fb-1a8d3b657a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 23:59:20.945118 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:20.945078    2559 generic.go:356] "Generic (PLEG): container finished" podID="3aa6fdee-2279-43ac-94fb-1a8d3b657a24" containerID="7935c265bc6e52545fd68562743a09c16104c2c66c552e986fc40e4440f1ddea" exitCode=0
Jul 27 23:59:20.945611 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:20.945152    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn" event={"ID":"3aa6fdee-2279-43ac-94fb-1a8d3b657a24","Type":"ContainerDied","Data":"7935c265bc6e52545fd68562743a09c16104c2c66c552e986fc40e4440f1ddea"}
Jul 27 23:59:20.945611 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:20.945600    2559 scope.go:117] "RemoveContainer" containerID="7935c265bc6e52545fd68562743a09c16104c2c66c552e986fc40e4440f1ddea"
Jul 27 23:59:20.946740 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:20.946717    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:59:20.946812 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:20.946803    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:59:20.946855 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:20.946838    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 23:59:21.952087 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:21.952050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn" event={"ID":"3aa6fdee-2279-43ac-94fb-1a8d3b657a24","Type":"ContainerStarted","Data":"767fba1f53708c8fe519423b962447e9c9c142c55e9838f74df814742f1929b6"}
Jul 27 23:59:22.718301 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:22.718263    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:22.718525 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:22.718503    2559 patch_prober.go:28] interesting pod/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 27 23:59:22.718577 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:22.718554    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn" podUID="3aa6fdee-2279-43ac-94fb-1a8d3b657a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 27 23:59:23.719000 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:23.718956    2559 patch_prober.go:28] interesting pod/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 27 23:59:23.719402 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:23.719018    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn" podUID="3aa6fdee-2279-43ac-94fb-1a8d3b657a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 27 23:59:24.718784 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:24.718745    2559 patch_prober.go:28] interesting pod/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 27 23:59:24.718982 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:24.718810    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn" podUID="3aa6fdee-2279-43ac-94fb-1a8d3b657a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 27 23:59:25.718921 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:25.718885    2559 patch_prober.go:28] interesting pod/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 27 23:59:25.719314 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:25.718943    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn" podUID="3aa6fdee-2279-43ac-94fb-1a8d3b657a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 27 23:59:26.721807 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:26.721779    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:26.722186 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:26.722091    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:26.722779 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:26.722756    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 27 23:59:27.271742 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:27.271711    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:27.271906 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:27.271779    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:27.284009 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:27.283987    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 27 23:59:27.995800 ip-10-0-142-53 kubenswrapper[2559]: I0727 23:59:27.995767    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 28 00:00:00.134941 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.134896    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a8122-29753280-gdk9j"]
Jul 28 00:00:00.138626 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.138607    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753280-698wv"]
Jul 28 00:00:00.138733 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.138717    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a8122-29753280-gdk9j"
Jul 28 00:00:00.140665 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.140641    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 28 00:00:00.142095 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.141908    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-42d790-29753280-c6h7b"]
Jul 28 00:00:00.142597 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.142425    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753280-698wv"
Jul 28 00:00:00.146428 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.146191    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-42d790-29753280-c6h7b"]
Jul 28 00:00:00.146428 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.146284    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-42d790-29753280-c6h7b"
Jul 28 00:00:00.147795 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.147714    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753280-698wv"]
Jul 28 00:00:00.150676 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.149991    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a8122-29753280-gdk9j"]
Jul 28 00:00:00.171817 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.171791    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a8122-29753280-gdk9j"
Jul 28 00:00:00.182694 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.182673    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753280-698wv"
Jul 28 00:00:00.216889 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.216857    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-42d790-29753280-c6h7b"
Jul 28 00:00:00.362636 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.362562    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a8122-29753280-gdk9j"]
Jul 28 00:00:00.368848 ip-10-0-142-53 kubenswrapper[2559]: W0728 00:00:00.368787    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009101ee_24c6_4e5b_bf21_4315e55a655f.slice/crio-0326f685f1f3d58472cbc5f514d87e82b236832749bdd52e214a95f92b482403 WatchSource:0}: Error finding container 0326f685f1f3d58472cbc5f514d87e82b236832749bdd52e214a95f92b482403: Status 404 returned error can't find the container with id 0326f685f1f3d58472cbc5f514d87e82b236832749bdd52e214a95f92b482403
Jul 28 00:00:00.399175 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.399097    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753280-698wv"]
Jul 28 00:00:00.401774 ip-10-0-142-53 kubenswrapper[2559]: W0728 00:00:00.401736    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1386c35_ad17_494a_9b57_aa9c3729968d.slice/crio-233053ceab346f5ade1e873574c250f3c67cdaf0f50c99ffa60b21492926c398 WatchSource:0}: Error finding container 233053ceab346f5ade1e873574c250f3c67cdaf0f50c99ffa60b21492926c398: Status 404 returned error can't find the container with id 233053ceab346f5ade1e873574c250f3c67cdaf0f50c99ffa60b21492926c398
Jul 28 00:00:00.419919 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:00.419899    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-42d790-29753280-c6h7b"]
Jul 28 00:00:00.422988 ip-10-0-142-53 kubenswrapper[2559]: W0728 00:00:00.422956    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254effe6_6f66_41ac_b99a_59a443838587.slice/crio-d2e83017f262e491110040de030a2ad07aaa2299c521501c526323bd04cf0794 WatchSource:0}: Error finding container d2e83017f262e491110040de030a2ad07aaa2299c521501c526323bd04cf0794: Status 404 returned error can't find the container with id d2e83017f262e491110040de030a2ad07aaa2299c521501c526323bd04cf0794
Jul 28 00:00:01.157585 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:01.157542    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a8122-29753280-gdk9j" event={"ID":"009101ee-24c6-4e5b-bf21-4315e55a655f","Type":"ContainerStarted","Data":"d223cc517943ca01108dd669165d3134f389669f68bf417d34ecf69fc899009b"}
Jul 28 00:00:01.158024 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:01.157606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a8122-29753280-gdk9j" event={"ID":"009101ee-24c6-4e5b-bf21-4315e55a655f","Type":"ContainerStarted","Data":"0326f685f1f3d58472cbc5f514d87e82b236832749bdd52e214a95f92b482403"}
Jul 28 00:00:01.159198 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:01.159172    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753280-698wv" event={"ID":"e1386c35-ad17-494a-9b57-aa9c3729968d","Type":"ContainerStarted","Data":"004234fae92ab352cbb2b55e4c4730c95467ef944e5ed73aa2b4a5a17f40ce80"}
Jul 28 00:00:01.159316 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:01.159218    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753280-698wv" event={"ID":"e1386c35-ad17-494a-9b57-aa9c3729968d","Type":"ContainerStarted","Data":"233053ceab346f5ade1e873574c250f3c67cdaf0f50c99ffa60b21492926c398"}
Jul 28 00:00:01.160660 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:01.160623    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-42d790-29753280-c6h7b" event={"ID":"254effe6-6f66-41ac-b99a-59a443838587","Type":"ContainerStarted","Data":"e2f1a51e3d2653fe6b3a826564c6726c5d30992fbad2aee3dcab876d06de8d64"}
Jul 28 00:00:01.160771 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:01.160668    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-42d790-29753280-c6h7b" event={"ID":"254effe6-6f66-41ac-b99a-59a443838587","Type":"ContainerStarted","Data":"d2e83017f262e491110040de030a2ad07aaa2299c521501c526323bd04cf0794"}
Jul 28 00:00:01.172992 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:01.172947    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a8122-29753280-gdk9j" podStartSLOduration=1.17293495 podStartE2EDuration="1.17293495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:00:01.172055459 +0000 UTC m=+2624.116286207" watchObservedRunningTime="2026-07-28 00:00:01.17293495 +0000 UTC m=+2624.117165696"
Jul 28 00:00:01.187727 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:01.186928    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753280-698wv" podStartSLOduration=1.186910983 podStartE2EDuration="1.186910983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:00:01.184351694 +0000 UTC m=+2624.128582441" watchObservedRunningTime="2026-07-28 00:00:01.186910983 +0000 UTC m=+2624.131141734"
Jul 28 00:00:01.197430 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:01.197388    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-42d790-29753280-c6h7b" podStartSLOduration=1.1973606430000001 podStartE2EDuration="1.197360643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:00:01.196514975 +0000 UTC m=+2624.140745735" watchObservedRunningTime="2026-07-28 00:00:01.197360643 +0000 UTC m=+2624.141591389"
Jul 28 00:00:02.167200 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:02.167164    2559 generic.go:356] "Generic (PLEG): container finished" podID="009101ee-24c6-4e5b-bf21-4315e55a655f" containerID="d223cc517943ca01108dd669165d3134f389669f68bf417d34ecf69fc899009b" exitCode=0
Jul 28 00:00:02.167635 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:02.167251    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a8122-29753280-gdk9j" event={"ID":"009101ee-24c6-4e5b-bf21-4315e55a655f","Type":"ContainerDied","Data":"d223cc517943ca01108dd669165d3134f389669f68bf417d34ecf69fc899009b"}
Jul 28 00:00:02.171581 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:02.171556    2559 generic.go:356] "Generic (PLEG): container finished" podID="e1386c35-ad17-494a-9b57-aa9c3729968d" containerID="004234fae92ab352cbb2b55e4c4730c95467ef944e5ed73aa2b4a5a17f40ce80" exitCode=0
Jul 28 00:00:02.171705 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:02.171634    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753280-698wv" event={"ID":"e1386c35-ad17-494a-9b57-aa9c3729968d","Type":"ContainerDied","Data":"004234fae92ab352cbb2b55e4c4730c95467ef944e5ed73aa2b4a5a17f40ce80"}
Jul 28 00:00:02.173418 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:02.173400    2559 generic.go:356] "Generic (PLEG): container finished" podID="254effe6-6f66-41ac-b99a-59a443838587" containerID="e2f1a51e3d2653fe6b3a826564c6726c5d30992fbad2aee3dcab876d06de8d64" exitCode=0
Jul 28 00:00:02.173526 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:02.173468    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-42d790-29753280-c6h7b" event={"ID":"254effe6-6f66-41ac-b99a-59a443838587","Type":"ContainerDied","Data":"e2f1a51e3d2653fe6b3a826564c6726c5d30992fbad2aee3dcab876d06de8d64"}
Jul 28 00:00:03.336358 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:03.336330    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a8122-29753280-gdk9j"
Jul 28 00:00:03.383204 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:03.383177    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-42d790-29753280-c6h7b"
Jul 28 00:00:03.386867 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:03.386847    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753280-698wv"
Jul 28 00:00:04.185492 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:04.185458    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a8122-29753280-gdk9j"
Jul 28 00:00:04.185492 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:04.185467    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a8122-29753280-gdk9j" event={"ID":"009101ee-24c6-4e5b-bf21-4315e55a655f","Type":"ContainerDied","Data":"0326f685f1f3d58472cbc5f514d87e82b236832749bdd52e214a95f92b482403"}
Jul 28 00:00:04.185717 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:04.185521    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0326f685f1f3d58472cbc5f514d87e82b236832749bdd52e214a95f92b482403"
Jul 28 00:00:04.186877 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:04.186848    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753280-698wv"
Jul 28 00:00:04.187014 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:04.186885    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753280-698wv" event={"ID":"e1386c35-ad17-494a-9b57-aa9c3729968d","Type":"ContainerDied","Data":"233053ceab346f5ade1e873574c250f3c67cdaf0f50c99ffa60b21492926c398"}
Jul 28 00:00:04.187014 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:04.186931    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="233053ceab346f5ade1e873574c250f3c67cdaf0f50c99ffa60b21492926c398"
Jul 28 00:00:04.188530 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:04.188507    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-42d790-29753280-c6h7b" event={"ID":"254effe6-6f66-41ac-b99a-59a443838587","Type":"ContainerDied","Data":"d2e83017f262e491110040de030a2ad07aaa2299c521501c526323bd04cf0794"}
Jul 28 00:00:04.188634 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:04.188554    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2e83017f262e491110040de030a2ad07aaa2299c521501c526323bd04cf0794"
Jul 28 00:00:04.188634 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:00:04.188516    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-42d790-29753280-c6h7b"
Jul 28 00:01:17.697202 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:01:17.697122    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 28 00:01:17.701416 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:01:17.701394    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 28 00:03:17.859279 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:03:17.859251    2559 scope.go:117] "RemoveContainer" containerID="db4d02e0eecafee83f3da7ee89c2af93501b688047ce8afd8190f13ccf5f85a0"
Jul 28 00:03:17.867652 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:03:17.867630    2559 scope.go:117] "RemoveContainer" containerID="7f1754ffa44905115da8a0c2e7cb1de3982b1ddd61c7565c905b868984e8e650"
Jul 28 00:04:17.701689 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:17.701655    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"]
Jul 28 00:04:17.891218 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:17.891189    2559 scope.go:117] "RemoveContainer" containerID="a45e799c13fb1057bad60a77a9c7a721af431a8863e5e0d43913cfdb92b250b4"
Jul 28 00:04:30.642398 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.642332    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"]
Jul 28 00:04:30.642929 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.642742    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1386c35-ad17-494a-9b57-aa9c3729968d" containerName="cleanup"
Jul 28 00:04:30.642929 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.642754    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1386c35-ad17-494a-9b57-aa9c3729968d" containerName="cleanup"
Jul 28 00:04:30.642929 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.642763    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="009101ee-24c6-4e5b-bf21-4315e55a655f" containerName="cleanup"
Jul 28 00:04:30.642929 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.642769    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="009101ee-24c6-4e5b-bf21-4315e55a655f" containerName="cleanup"
Jul 28 00:04:30.642929 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.642786    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="254effe6-6f66-41ac-b99a-59a443838587" containerName="cleanup"
Jul 28 00:04:30.642929 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.642792    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="254effe6-6f66-41ac-b99a-59a443838587" containerName="cleanup"
Jul 28 00:04:30.642929 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.642851    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="009101ee-24c6-4e5b-bf21-4315e55a655f" containerName="cleanup"
Jul 28 00:04:30.642929 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.642859    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1386c35-ad17-494a-9b57-aa9c3729968d" containerName="cleanup"
Jul 28 00:04:30.642929 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.642868    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="254effe6-6f66-41ac-b99a-59a443838587" containerName="cleanup"
Jul 28 00:04:30.645841 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.645822    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.647765 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.647745    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-42d790\"/\"sub-test-model-42d790-kserve-self-signed-certs\""
Jul 28 00:04:30.655191 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.655169    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"]
Jul 28 00:04:30.666295 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.666265    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-dshm\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.666441 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.666298    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-model-cache\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.666441 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.666322    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77862513-38c6-4efc-8e9c-f8903e2f1316-tls-certs\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.666441 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.666344    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-home\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.666441 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.666406    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p959t\" (UniqueName: \"kubernetes.io/projected/77862513-38c6-4efc-8e9c-f8903e2f1316-kube-api-access-p959t\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.666441 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.666433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-tmp-dir\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.766920 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.766885    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-tmp-dir\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.767115 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.766983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-dshm\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.767115 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.767005    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-model-cache\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.767115 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.767028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77862513-38c6-4efc-8e9c-f8903e2f1316-tls-certs\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.767115 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.767055    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-home\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.767115 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.767094    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p959t\" (UniqueName: \"kubernetes.io/projected/77862513-38c6-4efc-8e9c-f8903e2f1316-kube-api-access-p959t\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.767414 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.767352    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-tmp-dir\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.767490 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.767472    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-model-cache\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.767578 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.767555    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-home\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.769558 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.769538    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-dshm\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.769755 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.769741    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77862513-38c6-4efc-8e9c-f8903e2f1316-tls-certs\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.774814 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.774793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p959t\" (UniqueName: \"kubernetes.io/projected/77862513-38c6-4efc-8e9c-f8903e2f1316-kube-api-access-p959t\") pod \"sub-test-model-42d790-kserve-69c646f675-7gc72\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") " pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:30.955526 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:30.955429    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:31.094492 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:31.094462    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"]
Jul 28 00:04:31.096436 ip-10-0-142-53 kubenswrapper[2559]: W0728 00:04:31.096405    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77862513_38c6_4efc_8e9c_f8903e2f1316.slice/crio-2589209ae63cc78f30a8efb5a089ffe78e8f47202309dc062869fd59f2337827 WatchSource:0}: Error finding container 2589209ae63cc78f30a8efb5a089ffe78e8f47202309dc062869fd59f2337827: Status 404 returned error can't find the container with id 2589209ae63cc78f30a8efb5a089ffe78e8f47202309dc062869fd59f2337827
Jul 28 00:04:31.619193 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:31.619154    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72" event={"ID":"77862513-38c6-4efc-8e9c-f8903e2f1316","Type":"ContainerStarted","Data":"fe5501734315b55100f96550aa7408d9ae4217df784ed1b8e7ecc901592ce2d2"}
Jul 28 00:04:31.619400 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:31.619203    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72" event={"ID":"77862513-38c6-4efc-8e9c-f8903e2f1316","Type":"ContainerStarted","Data":"2589209ae63cc78f30a8efb5a089ffe78e8f47202309dc062869fd59f2337827"}
Jul 28 00:04:31.635252 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:31.635186    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72" podStartSLOduration=1.635172806 podStartE2EDuration="1.635172806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:04:31.633365359 +0000 UTC m=+2894.577596107" watchObservedRunningTime="2026-07-28 00:04:31.635172806 +0000 UTC m=+2894.579403553"
Jul 28 00:04:32.721577 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:32.721534    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8" podUID="e32da263-a70a-40f3-b5f8-974eb417f67b" containerName="main" containerID="cri-o://07effb0fcd91b9d6d978a919ad611bb2f6ce019cea6ccf779f9fa1dc54a3789c" gracePeriod=45
Jul 28 00:04:32.981227 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:32.981166    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 28 00:04:33.088280 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.088247    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6w8z\" (UniqueName: \"kubernetes.io/projected/e32da263-a70a-40f3-b5f8-974eb417f67b-kube-api-access-z6w8z\") pod \"e32da263-a70a-40f3-b5f8-974eb417f67b\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") "
Jul 28 00:04:33.088280 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.088288    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-dshm\") pod \"e32da263-a70a-40f3-b5f8-974eb417f67b\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") "
Jul 28 00:04:33.088562 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.088313    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32da263-a70a-40f3-b5f8-974eb417f67b-tls-certs\") pod \"e32da263-a70a-40f3-b5f8-974eb417f67b\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") "
Jul 28 00:04:33.088562 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.088331    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-tmp-dir\") pod \"e32da263-a70a-40f3-b5f8-974eb417f67b\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") "
Jul 28 00:04:33.088680 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.088568    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-model-cache\") pod \"e32da263-a70a-40f3-b5f8-974eb417f67b\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") "
Jul 28 00:04:33.088680 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.088640    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-home\") pod \"e32da263-a70a-40f3-b5f8-974eb417f67b\" (UID: \"e32da263-a70a-40f3-b5f8-974eb417f67b\") "
Jul 28 00:04:33.088680 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.088640    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e32da263-a70a-40f3-b5f8-974eb417f67b" (UID: "e32da263-a70a-40f3-b5f8-974eb417f67b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:04:33.088837 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.088823    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-model-cache" (OuterVolumeSpecName: "model-cache") pod "e32da263-a70a-40f3-b5f8-974eb417f67b" (UID: "e32da263-a70a-40f3-b5f8-974eb417f67b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:04:33.088980 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.088943    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-home" (OuterVolumeSpecName: "home") pod "e32da263-a70a-40f3-b5f8-974eb417f67b" (UID: "e32da263-a70a-40f3-b5f8-974eb417f67b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:04:33.089090 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.089075    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-tmp-dir\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:04:33.089161 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.089095    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-model-cache\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:04:33.089161 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.089109    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-home\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:04:33.090960 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.090926    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32da263-a70a-40f3-b5f8-974eb417f67b-kube-api-access-z6w8z" (OuterVolumeSpecName: "kube-api-access-z6w8z") pod "e32da263-a70a-40f3-b5f8-974eb417f67b" (UID: "e32da263-a70a-40f3-b5f8-974eb417f67b"). InnerVolumeSpecName "kube-api-access-z6w8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:04:33.091138 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.091122    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-dshm" (OuterVolumeSpecName: "dshm") pod "e32da263-a70a-40f3-b5f8-974eb417f67b" (UID: "e32da263-a70a-40f3-b5f8-974eb417f67b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:04:33.091250 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.091232    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32da263-a70a-40f3-b5f8-974eb417f67b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e32da263-a70a-40f3-b5f8-974eb417f67b" (UID: "e32da263-a70a-40f3-b5f8-974eb417f67b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:04:33.189919 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.189880    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6w8z\" (UniqueName: \"kubernetes.io/projected/e32da263-a70a-40f3-b5f8-974eb417f67b-kube-api-access-z6w8z\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:04:33.189919 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.189919    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32da263-a70a-40f3-b5f8-974eb417f67b-dshm\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:04:33.190163 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.189932    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32da263-a70a-40f3-b5f8-974eb417f67b-tls-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:04:33.630585 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.630552    2559 generic.go:356] "Generic (PLEG): container finished" podID="e32da263-a70a-40f3-b5f8-974eb417f67b" containerID="07effb0fcd91b9d6d978a919ad611bb2f6ce019cea6ccf779f9fa1dc54a3789c" exitCode=0
Jul 28 00:04:33.630780 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.630591    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"
Jul 28 00:04:33.630780 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.630636    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8" event={"ID":"e32da263-a70a-40f3-b5f8-974eb417f67b","Type":"ContainerDied","Data":"07effb0fcd91b9d6d978a919ad611bb2f6ce019cea6ccf779f9fa1dc54a3789c"}
Jul 28 00:04:33.630780 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.630681    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8" event={"ID":"e32da263-a70a-40f3-b5f8-974eb417f67b","Type":"ContainerDied","Data":"0b937f8a3639f96769b65e1428fd64a9b1b92f9fb4542bd69900361513bd9810"}
Jul 28 00:04:33.630780 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.630701    2559 scope.go:117] "RemoveContainer" containerID="07effb0fcd91b9d6d978a919ad611bb2f6ce019cea6ccf779f9fa1dc54a3789c"
Jul 28 00:04:33.639825 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.639786    2559 scope.go:117] "RemoveContainer" containerID="07effb0fcd91b9d6d978a919ad611bb2f6ce019cea6ccf779f9fa1dc54a3789c"
Jul 28 00:04:33.640058 ip-10-0-142-53 kubenswrapper[2559]: E0728 00:04:33.640041    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07effb0fcd91b9d6d978a919ad611bb2f6ce019cea6ccf779f9fa1dc54a3789c\": container with ID starting with 07effb0fcd91b9d6d978a919ad611bb2f6ce019cea6ccf779f9fa1dc54a3789c not found: ID does not exist" containerID="07effb0fcd91b9d6d978a919ad611bb2f6ce019cea6ccf779f9fa1dc54a3789c"
Jul 28 00:04:33.640101 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.640068    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07effb0fcd91b9d6d978a919ad611bb2f6ce019cea6ccf779f9fa1dc54a3789c"} err="failed to get container status \"07effb0fcd91b9d6d978a919ad611bb2f6ce019cea6ccf779f9fa1dc54a3789c\": rpc error: code = NotFound desc = could not find container \"07effb0fcd91b9d6d978a919ad611bb2f6ce019cea6ccf779f9fa1dc54a3789c\": container with ID starting with 07effb0fcd91b9d6d978a919ad611bb2f6ce019cea6ccf779f9fa1dc54a3789c not found: ID does not exist"
Jul 28 00:04:33.644912 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.644892    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"]
Jul 28 00:04:33.648463 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:33.648442    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/auth-test-model-42d790-kserve-798d49d5c9-9sjh8"]
Jul 28 00:04:35.563303 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:35.563273    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e32da263-a70a-40f3-b5f8-974eb417f67b" path="/var/lib/kubelet/pods/e32da263-a70a-40f3-b5f8-974eb417f67b/volumes"
Jul 28 00:04:40.956295 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:40.956257    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:40.956295 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:40.956305    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:40.968648 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:40.968622    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:04:41.683350 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:04:41.683320    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:05:17.913425 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:05:17.913365    2559 scope.go:117] "RemoveContainer" containerID="e4023252dc4e95c51cced8fe4c608e6c26d2ff0d0ad636f5fce423ed372e0f96"
Jul 28 00:06:14.458893 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:14.458857    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"]
Jul 28 00:06:17.736479 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:17.736449    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 28 00:06:17.742726 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:17.742703    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 28 00:06:29.478994 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.478946    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72" podUID="77862513-38c6-4efc-8e9c-f8903e2f1316" containerName="main" containerID="cri-o://fe5501734315b55100f96550aa7408d9ae4217df784ed1b8e7ecc901592ce2d2" gracePeriod=45
Jul 28 00:06:29.743350 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.743284    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:06:29.894509 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.894481    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-tmp-dir\") pod \"77862513-38c6-4efc-8e9c-f8903e2f1316\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") "
Jul 28 00:06:29.894708 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.894559    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-model-cache\") pod \"77862513-38c6-4efc-8e9c-f8903e2f1316\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") "
Jul 28 00:06:29.894708 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.894582    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p959t\" (UniqueName: \"kubernetes.io/projected/77862513-38c6-4efc-8e9c-f8903e2f1316-kube-api-access-p959t\") pod \"77862513-38c6-4efc-8e9c-f8903e2f1316\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") "
Jul 28 00:06:29.894708 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.894623    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77862513-38c6-4efc-8e9c-f8903e2f1316-tls-certs\") pod \"77862513-38c6-4efc-8e9c-f8903e2f1316\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") "
Jul 28 00:06:29.894708 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.894651    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-dshm\") pod \"77862513-38c6-4efc-8e9c-f8903e2f1316\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") "
Jul 28 00:06:29.894708 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.894679    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-home\") pod \"77862513-38c6-4efc-8e9c-f8903e2f1316\" (UID: \"77862513-38c6-4efc-8e9c-f8903e2f1316\") "
Jul 28 00:06:29.894988 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.894781    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77862513-38c6-4efc-8e9c-f8903e2f1316" (UID: "77862513-38c6-4efc-8e9c-f8903e2f1316"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:06:29.894988 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.894872    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-model-cache" (OuterVolumeSpecName: "model-cache") pod "77862513-38c6-4efc-8e9c-f8903e2f1316" (UID: "77862513-38c6-4efc-8e9c-f8903e2f1316"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:06:29.895096 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.894993    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-home" (OuterVolumeSpecName: "home") pod "77862513-38c6-4efc-8e9c-f8903e2f1316" (UID: "77862513-38c6-4efc-8e9c-f8903e2f1316"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:06:29.895096 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.895036    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-tmp-dir\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:06:29.895096 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.895048    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-model-cache\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:06:29.897051 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.897025    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77862513-38c6-4efc-8e9c-f8903e2f1316-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77862513-38c6-4efc-8e9c-f8903e2f1316" (UID: "77862513-38c6-4efc-8e9c-f8903e2f1316"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:06:29.897051 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.897034    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-dshm" (OuterVolumeSpecName: "dshm") pod "77862513-38c6-4efc-8e9c-f8903e2f1316" (UID: "77862513-38c6-4efc-8e9c-f8903e2f1316"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:06:29.897201 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.897164    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77862513-38c6-4efc-8e9c-f8903e2f1316-kube-api-access-p959t" (OuterVolumeSpecName: "kube-api-access-p959t") pod "77862513-38c6-4efc-8e9c-f8903e2f1316" (UID: "77862513-38c6-4efc-8e9c-f8903e2f1316"). InnerVolumeSpecName "kube-api-access-p959t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:06:29.995840 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.995741    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p959t\" (UniqueName: \"kubernetes.io/projected/77862513-38c6-4efc-8e9c-f8903e2f1316-kube-api-access-p959t\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:06:29.995840 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.995774    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77862513-38c6-4efc-8e9c-f8903e2f1316-tls-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:06:29.995840 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.995788    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-dshm\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:06:29.995840 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:29.995798    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77862513-38c6-4efc-8e9c-f8903e2f1316-home\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:06:30.228143 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:30.228098    2559 generic.go:356] "Generic (PLEG): container finished" podID="77862513-38c6-4efc-8e9c-f8903e2f1316" containerID="fe5501734315b55100f96550aa7408d9ae4217df784ed1b8e7ecc901592ce2d2" exitCode=0
Jul 28 00:06:30.228348 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:30.228284    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72" event={"ID":"77862513-38c6-4efc-8e9c-f8903e2f1316","Type":"ContainerDied","Data":"fe5501734315b55100f96550aa7408d9ae4217df784ed1b8e7ecc901592ce2d2"}
Jul 28 00:06:30.228348 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:30.228321    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72" event={"ID":"77862513-38c6-4efc-8e9c-f8903e2f1316","Type":"ContainerDied","Data":"2589209ae63cc78f30a8efb5a089ffe78e8f47202309dc062869fd59f2337827"}
Jul 28 00:06:30.228483 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:30.228354    2559 scope.go:117] "RemoveContainer" containerID="fe5501734315b55100f96550aa7408d9ae4217df784ed1b8e7ecc901592ce2d2"
Jul 28 00:06:30.228602 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:30.228588    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"
Jul 28 00:06:30.240659 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:30.240641    2559 scope.go:117] "RemoveContainer" containerID="fe5501734315b55100f96550aa7408d9ae4217df784ed1b8e7ecc901592ce2d2"
Jul 28 00:06:30.240930 ip-10-0-142-53 kubenswrapper[2559]: E0728 00:06:30.240904    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5501734315b55100f96550aa7408d9ae4217df784ed1b8e7ecc901592ce2d2\": container with ID starting with fe5501734315b55100f96550aa7408d9ae4217df784ed1b8e7ecc901592ce2d2 not found: ID does not exist" containerID="fe5501734315b55100f96550aa7408d9ae4217df784ed1b8e7ecc901592ce2d2"
Jul 28 00:06:30.241062 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:30.240936    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5501734315b55100f96550aa7408d9ae4217df784ed1b8e7ecc901592ce2d2"} err="failed to get container status \"fe5501734315b55100f96550aa7408d9ae4217df784ed1b8e7ecc901592ce2d2\": rpc error: code = NotFound desc = could not find container \"fe5501734315b55100f96550aa7408d9ae4217df784ed1b8e7ecc901592ce2d2\": container with ID starting with fe5501734315b55100f96550aa7408d9ae4217df784ed1b8e7ecc901592ce2d2 not found: ID does not exist"
Jul 28 00:06:30.250455 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:30.250400    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"]
Jul 28 00:06:30.254060 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:30.254042    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/sub-test-model-42d790-kserve-69c646f675-7gc72"]
Jul 28 00:06:31.562426 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:31.562383    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77862513-38c6-4efc-8e9c-f8903e2f1316" path="/var/lib/kubelet/pods/77862513-38c6-4efc-8e9c-f8903e2f1316/volumes"
Jul 28 00:06:33.037504 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.037466    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"]
Jul 28 00:06:33.038006 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.037989    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77862513-38c6-4efc-8e9c-f8903e2f1316" containerName="main"
Jul 28 00:06:33.038062 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.038010    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="77862513-38c6-4efc-8e9c-f8903e2f1316" containerName="main"
Jul 28 00:06:33.038062 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.038024    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e32da263-a70a-40f3-b5f8-974eb417f67b" containerName="main"
Jul 28 00:06:33.038062 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.038030    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e32da263-a70a-40f3-b5f8-974eb417f67b" containerName="main"
Jul 28 00:06:33.038162 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.038106    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="77862513-38c6-4efc-8e9c-f8903e2f1316" containerName="main"
Jul 28 00:06:33.038162 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.038115    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e32da263-a70a-40f3-b5f8-974eb417f67b" containerName="main"
Jul 28 00:06:33.042471 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.042454    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.045471 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.045441    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-42d790\"/\"openshift-service-ca.crt\""
Jul 28 00:06:33.045471 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.045466    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-42d790\"/\"default-dockercfg-6nkz8\""
Jul 28 00:06:33.045614 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.045491    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-42d790\"/\"rate-test-model-42d790-kserve-self-signed-certs\""
Jul 28 00:06:33.045614 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.045466    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-42d790\"/\"kube-root-ca.crt\""
Jul 28 00:06:33.049341 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.049318    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"]
Jul 28 00:06:33.123733 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.123696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-tmp-dir\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.123733 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.123733    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nndtk\" (UniqueName: \"kubernetes.io/projected/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-kube-api-access-nndtk\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.123943 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.123761    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-dshm\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.123943 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.123895    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-home\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.124033 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.123952    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-tls-certs\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.124033 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.123996    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-model-cache\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.225313 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.225279    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-tmp-dir\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.225313 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.225314    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nndtk\" (UniqueName: \"kubernetes.io/projected/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-kube-api-access-nndtk\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.225572 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.225333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-dshm\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.225572 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.225420    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-home\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.225572 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.225466    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-tls-certs\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.225572 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.225495    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-model-cache\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.225815 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.225790    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-tmp-dir\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.225869 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.225823    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-model-cache\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.225911 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.225882    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-home\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.227845 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.227827    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-dshm\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.228106 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.228090    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-tls-certs\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.232355 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.232337    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nndtk\" (UniqueName: \"kubernetes.io/projected/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-kube-api-access-nndtk\") pod \"rate-test-model-42d790-kserve-5c5bff6859-g9gj4\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") " pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.352465 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.352360    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:33.493180 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:33.493144    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"]
Jul 28 00:06:33.494861 ip-10-0-142-53 kubenswrapper[2559]: W0728 00:06:33.494835    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e3b9f9e_b255_4446_96e7_18cf4f0a2871.slice/crio-ff7c7abbe94de3e04b5977541d0456b2ded544b90017c5dbb7fab8d055732250 WatchSource:0}: Error finding container ff7c7abbe94de3e04b5977541d0456b2ded544b90017c5dbb7fab8d055732250: Status 404 returned error can't find the container with id ff7c7abbe94de3e04b5977541d0456b2ded544b90017c5dbb7fab8d055732250
Jul 28 00:06:34.254677 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:34.254641    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4" event={"ID":"1e3b9f9e-b255-4446-96e7-18cf4f0a2871","Type":"ContainerStarted","Data":"cd16e225c7647f485ca43428bc3d8778842fa08b2796c5e9277b5bae2cb21cba"}
Jul 28 00:06:34.254677 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:34.254682    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4" event={"ID":"1e3b9f9e-b255-4446-96e7-18cf4f0a2871","Type":"ContainerStarted","Data":"ff7c7abbe94de3e04b5977541d0456b2ded544b90017c5dbb7fab8d055732250"}
Jul 28 00:06:34.274535 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:34.274487    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4" podStartSLOduration=1.2744737289999999 podStartE2EDuration="1.274473729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:06:34.272766271 +0000 UTC m=+3017.216997019" watchObservedRunningTime="2026-07-28 00:06:34.274473729 +0000 UTC m=+3017.218704497"
Jul 28 00:06:43.353201 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:43.353170    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:43.353201 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:43.353210    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:43.365619 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:43.365601    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:06:44.316957 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:06:44.316927    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:08:01.337104 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:01.337066    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"]
Jul 28 00:08:16.356855 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.356810    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4" podUID="1e3b9f9e-b255-4446-96e7-18cf4f0a2871" containerName="main" containerID="cri-o://cd16e225c7647f485ca43428bc3d8778842fa08b2796c5e9277b5bae2cb21cba" gracePeriod=45
Jul 28 00:08:16.610159 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.610094    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:08:16.714646 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.714612    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-home\") pod \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") "
Jul 28 00:08:16.714840 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.714700    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-tmp-dir\") pod \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") "
Jul 28 00:08:16.714840 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.714722    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-dshm\") pod \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") "
Jul 28 00:08:16.714840 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.714763    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-model-cache\") pod \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") "
Jul 28 00:08:16.714840 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.714828    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-tls-certs\") pod \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") "
Jul 28 00:08:16.715043 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.714857    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nndtk\" (UniqueName: \"kubernetes.io/projected/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-kube-api-access-nndtk\") pod \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\" (UID: \"1e3b9f9e-b255-4446-96e7-18cf4f0a2871\") "
Jul 28 00:08:16.715043 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.714960    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-home" (OuterVolumeSpecName: "home") pod "1e3b9f9e-b255-4446-96e7-18cf4f0a2871" (UID: "1e3b9f9e-b255-4446-96e7-18cf4f0a2871"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:08:16.715155 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.715051    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e3b9f9e-b255-4446-96e7-18cf4f0a2871" (UID: "1e3b9f9e-b255-4446-96e7-18cf4f0a2871"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:08:16.715155 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.715123    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e3b9f9e-b255-4446-96e7-18cf4f0a2871" (UID: "1e3b9f9e-b255-4446-96e7-18cf4f0a2871"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:08:16.718413 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.717586    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-dshm" (OuterVolumeSpecName: "dshm") pod "1e3b9f9e-b255-4446-96e7-18cf4f0a2871" (UID: "1e3b9f9e-b255-4446-96e7-18cf4f0a2871"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:08:16.718413 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.717962    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-home\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:08:16.718413 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.717983    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-tmp-dir\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:08:16.718413 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.718002    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-dshm\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:08:16.718413 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.718017    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-model-cache\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:08:16.725224 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.725185    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-kube-api-access-nndtk" (OuterVolumeSpecName: "kube-api-access-nndtk") pod "1e3b9f9e-b255-4446-96e7-18cf4f0a2871" (UID: "1e3b9f9e-b255-4446-96e7-18cf4f0a2871"). InnerVolumeSpecName "kube-api-access-nndtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:08:16.725637 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.725608    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e3b9f9e-b255-4446-96e7-18cf4f0a2871" (UID: "1e3b9f9e-b255-4446-96e7-18cf4f0a2871"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:08:16.790141 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.790103    2559 generic.go:356] "Generic (PLEG): container finished" podID="1e3b9f9e-b255-4446-96e7-18cf4f0a2871" containerID="cd16e225c7647f485ca43428bc3d8778842fa08b2796c5e9277b5bae2cb21cba" exitCode=0
Jul 28 00:08:16.790355 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.790164    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"
Jul 28 00:08:16.790355 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.790191    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4" event={"ID":"1e3b9f9e-b255-4446-96e7-18cf4f0a2871","Type":"ContainerDied","Data":"cd16e225c7647f485ca43428bc3d8778842fa08b2796c5e9277b5bae2cb21cba"}
Jul 28 00:08:16.790355 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.790233    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4" event={"ID":"1e3b9f9e-b255-4446-96e7-18cf4f0a2871","Type":"ContainerDied","Data":"ff7c7abbe94de3e04b5977541d0456b2ded544b90017c5dbb7fab8d055732250"}
Jul 28 00:08:16.790355 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.790249    2559 scope.go:117] "RemoveContainer" containerID="cd16e225c7647f485ca43428bc3d8778842fa08b2796c5e9277b5bae2cb21cba"
Jul 28 00:08:16.800002 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.799984    2559 scope.go:117] "RemoveContainer" containerID="cd16e225c7647f485ca43428bc3d8778842fa08b2796c5e9277b5bae2cb21cba"
Jul 28 00:08:16.800264 ip-10-0-142-53 kubenswrapper[2559]: E0728 00:08:16.800233    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd16e225c7647f485ca43428bc3d8778842fa08b2796c5e9277b5bae2cb21cba\": container with ID starting with cd16e225c7647f485ca43428bc3d8778842fa08b2796c5e9277b5bae2cb21cba not found: ID does not exist" containerID="cd16e225c7647f485ca43428bc3d8778842fa08b2796c5e9277b5bae2cb21cba"
Jul 28 00:08:16.800314 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.800270    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd16e225c7647f485ca43428bc3d8778842fa08b2796c5e9277b5bae2cb21cba"} err="failed to get container status \"cd16e225c7647f485ca43428bc3d8778842fa08b2796c5e9277b5bae2cb21cba\": rpc error: code = NotFound desc = could not find container \"cd16e225c7647f485ca43428bc3d8778842fa08b2796c5e9277b5bae2cb21cba\": container with ID starting with cd16e225c7647f485ca43428bc3d8778842fa08b2796c5e9277b5bae2cb21cba not found: ID does not exist"
Jul 28 00:08:16.810899 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.810872    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"]
Jul 28 00:08:16.815071 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.815038    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-42d790/rate-test-model-42d790-kserve-5c5bff6859-g9gj4"]
Jul 28 00:08:16.819408 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.819360    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-tls-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:08:16.819502 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:16.819416    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nndtk\" (UniqueName: \"kubernetes.io/projected/1e3b9f9e-b255-4446-96e7-18cf4f0a2871-kube-api-access-nndtk\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:08:17.562827 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:17.562788    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e3b9f9e-b255-4446-96e7-18cf4f0a2871" path="/var/lib/kubelet/pods/1e3b9f9e-b255-4446-96e7-18cf4f0a2871/volumes"
Jul 28 00:08:22.546466 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.546434    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"]
Jul 28 00:08:22.546917 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.546839    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e3b9f9e-b255-4446-96e7-18cf4f0a2871" containerName="main"
Jul 28 00:08:22.546917 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.546850    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e3b9f9e-b255-4446-96e7-18cf4f0a2871" containerName="main"
Jul 28 00:08:22.546917 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.546916    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e3b9f9e-b255-4446-96e7-18cf4f0a2871" containerName="main"
Jul 28 00:08:22.551424 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.551406    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.553728 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.553707    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4a6d94-openshift-default-dockercfg-7vtz5\""
Jul 28 00:08:22.560321 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.560299    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"]
Jul 28 00:08:22.671838 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.671806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-token\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.672013 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.671843    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-workload-socket\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.672013 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.671870    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-credential-socket\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.672013 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.671931    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e852dad-40c4-4599-bc05-55fa1f48859a-istiod-ca-cert\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.672013 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.671997    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-data\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.672013 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.672011    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-podinfo\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.672228 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.672073    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-workload-certs\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.672228 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.672114    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwq5q\" (UniqueName: \"kubernetes.io/projected/5e852dad-40c4-4599-bc05-55fa1f48859a-kube-api-access-gwq5q\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.672228 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.672133    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-envoy\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.773479 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.773441    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-data\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.773479 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.773479    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-podinfo\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.773736 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.773511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-workload-certs\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.773736 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.773639    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwq5q\" (UniqueName: \"kubernetes.io/projected/5e852dad-40c4-4599-bc05-55fa1f48859a-kube-api-access-gwq5q\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.773736 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.773676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-envoy\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.773736 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.773706    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-token\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.773956 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.773739    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-workload-socket\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.773956 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.773786    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-credential-socket\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.773956 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.773865    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e852dad-40c4-4599-bc05-55fa1f48859a-istiod-ca-cert\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.773956 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.773876    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-workload-certs\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.773956 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.773919    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-data\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.774203 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.774046    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-workload-socket\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.774496 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.774475    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-credential-socket\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.774549 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.774509    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e852dad-40c4-4599-bc05-55fa1f48859a-istiod-ca-cert\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.776024 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.776004    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-envoy\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.776221 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.776203    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-podinfo\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.788866 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.788836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-token\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.789460 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.789439    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwq5q\" (UniqueName: \"kubernetes.io/projected/5e852dad-40c4-4599-bc05-55fa1f48859a-kube-api-access-gwq5q\") pod \"e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") " pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:22.863411 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:22.863316    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:23.002930 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:23.002900    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"]
Jul 28 00:08:23.004118 ip-10-0-142-53 kubenswrapper[2559]: W0728 00:08:23.004093    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e852dad_40c4_4599_bc05_55fa1f48859a.slice/crio-6816310944f7dd5ccdcceec3645f9043049f27d3c27e110e37aeafb68ffc4b48 WatchSource:0}: Error finding container 6816310944f7dd5ccdcceec3645f9043049f27d3c27e110e37aeafb68ffc4b48: Status 404 returned error can't find the container with id 6816310944f7dd5ccdcceec3645f9043049f27d3c27e110e37aeafb68ffc4b48
Jul 28 00:08:23.006422 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:23.006386    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 00:08:23.006484 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:23.006461    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 00:08:23.006521 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:23.006490    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 00:08:23.832359 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:23.832323    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5" event={"ID":"5e852dad-40c4-4599-bc05-55fa1f48859a","Type":"ContainerStarted","Data":"ce2c18237195ae189279cdc265148af356701ad67786a56d2f0fa899588a8fc3"}
Jul 28 00:08:23.832787 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:23.832396    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5" event={"ID":"5e852dad-40c4-4599-bc05-55fa1f48859a","Type":"ContainerStarted","Data":"6816310944f7dd5ccdcceec3645f9043049f27d3c27e110e37aeafb68ffc4b48"}
Jul 28 00:08:23.849894 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:23.849843    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5" podStartSLOduration=1.849826927 podStartE2EDuration="1.849826927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:08:23.849062798 +0000 UTC m=+3126.793293547" watchObservedRunningTime="2026-07-28 00:08:23.849826927 +0000 UTC m=+3126.794057674"
Jul 28 00:08:23.863560 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:23.863526    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:24.869531 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:24.869497    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:25.841786 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:25.841751    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:25.842590 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:25.842567    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:41.883522 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:41.883491    2559 patch_prober.go:28] interesting pod/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 28 00:08:41.883935 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:41.883548    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5" podUID="5e852dad-40c4-4599-bc05-55fa1f48859a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 00:08:42.939535 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:42.939453    2559 generic.go:356] "Generic (PLEG): container finished" podID="5e852dad-40c4-4599-bc05-55fa1f48859a" containerID="ce2c18237195ae189279cdc265148af356701ad67786a56d2f0fa899588a8fc3" exitCode=0
Jul 28 00:08:42.939535 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:42.939506    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5" event={"ID":"5e852dad-40c4-4599-bc05-55fa1f48859a","Type":"ContainerDied","Data":"ce2c18237195ae189279cdc265148af356701ad67786a56d2f0fa899588a8fc3"}
Jul 28 00:08:42.939967 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:42.939899    2559 scope.go:117] "RemoveContainer" containerID="ce2c18237195ae189279cdc265148af356701ad67786a56d2f0fa899588a8fc3"
Jul 28 00:08:42.941020 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:42.940978    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 00:08:42.941081 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:42.941068    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 00:08:42.941119 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:42.941103    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 00:08:43.946172 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:43.946136    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5" event={"ID":"5e852dad-40c4-4599-bc05-55fa1f48859a","Type":"ContainerStarted","Data":"58dba90bc70ae3d07dec985dd397125fb0ac38809156bc696cc7ee9ca4c44ec2"}
Jul 28 00:08:44.864282 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:44.864245    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:44.864527 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:44.864503    2559 patch_prober.go:28] interesting pod/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 28 00:08:44.864583 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:44.864558    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5" podUID="5e852dad-40c4-4599-bc05-55fa1f48859a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 28 00:08:45.864054 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:45.864014    2559 patch_prober.go:28] interesting pod/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 28 00:08:45.864443 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:45.864079    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5" podUID="5e852dad-40c4-4599-bc05-55fa1f48859a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 28 00:08:46.509398 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.506280    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"]
Jul 28 00:08:46.525406 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.519691    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.525406 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.523799    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-87c90a-openshift-default-dockercfg-9kjhv\""
Jul 28 00:08:46.531141 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.531117    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"]
Jul 28 00:08:46.625986 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.625957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-credential-socket\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.626159 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.626007    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-token\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.626159 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.626034    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zjhl\" (UniqueName: \"kubernetes.io/projected/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-kube-api-access-5zjhl\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.626159 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.626064    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-workload-certs\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.626159 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.626133    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istiod-ca-cert\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.626328 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.626169    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-podinfo\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.626328 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.626225    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-envoy\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.626328 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.626258    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-data\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.626328 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.626297    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-workload-socket\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.727524 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.727471    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-envoy\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.727715 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.727537    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-data\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.727715 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.727566    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-workload-socket\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.727715 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.727616    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-credential-socket\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.727715 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.727669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-token\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.727715 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.727691    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zjhl\" (UniqueName: \"kubernetes.io/projected/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-kube-api-access-5zjhl\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.727978 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.727732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-workload-certs\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.727978 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.727773    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istiod-ca-cert\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.727978 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.727812    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-podinfo\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.728501 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.728476    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-credential-socket\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.728920 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.728860    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-workload-certs\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.729071 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.728875    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-workload-socket\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.729558 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.729532    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istiod-ca-cert\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.729942 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.729918    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-data\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.731166 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.731120    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-podinfo\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.732017 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.731977    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-envoy\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.736316 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.736276    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zjhl\" (UniqueName: \"kubernetes.io/projected/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-kube-api-access-5zjhl\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.736728 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.736713    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-token\") pod \"e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") " pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.837082 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.837053    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:46.863787 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.863754    2559 patch_prober.go:28] interesting pod/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 28 00:08:46.863908 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:46.863828    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5" podUID="5e852dad-40c4-4599-bc05-55fa1f48859a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 28 00:08:47.020218 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:47.020181    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"]
Jul 28 00:08:47.024280 ip-10-0-142-53 kubenswrapper[2559]: W0728 00:08:47.024255    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42e7888_5bfd_4f66_b898_3c75ce4f60b7.slice/crio-f84155249ed88feeb2e26fcd33e4ad80cd905565b74a0c4792593f95870e4b75 WatchSource:0}: Error finding container f84155249ed88feeb2e26fcd33e4ad80cd905565b74a0c4792593f95870e4b75: Status 404 returned error can't find the container with id f84155249ed88feeb2e26fcd33e4ad80cd905565b74a0c4792593f95870e4b75
Jul 28 00:08:47.026911 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:47.026872    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 00:08:47.026987 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:47.026969    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 00:08:47.027035 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:47.027015    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 00:08:47.864586 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:47.864546    2559 patch_prober.go:28] interesting pod/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 28 00:08:47.864786 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:47.864611    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5" podUID="5e852dad-40c4-4599-bc05-55fa1f48859a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 28 00:08:47.969634 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:47.969598    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp" event={"ID":"d42e7888-5bfd-4f66-b898-3c75ce4f60b7","Type":"ContainerStarted","Data":"8f34b5eed7c48248ea7ef2d87d21c77c86c2977250cb3984b47057161b686c7e"}
Jul 28 00:08:47.969816 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:47.969688    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp" event={"ID":"d42e7888-5bfd-4f66-b898-3c75ce4f60b7","Type":"ContainerStarted","Data":"f84155249ed88feeb2e26fcd33e4ad80cd905565b74a0c4792593f95870e4b75"}
Jul 28 00:08:47.990547 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:47.990485    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp" podStartSLOduration=1.990467164 podStartE2EDuration="1.990467164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:08:47.986593061 +0000 UTC m=+3150.930823811" watchObservedRunningTime="2026-07-28 00:08:47.990467164 +0000 UTC m=+3150.934697912"
Jul 28 00:08:48.837706 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:48.837671    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:48.842923 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:48.842893    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:48.867671 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:48.867648    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:48.868015 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:48.867998    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:48.868790 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:48.868773    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:08:48.974898 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:48.974864    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:08:48.975773 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:08:48.975754    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:09:02.048633 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:02.048603    2559 generic.go:356] "Generic (PLEG): container finished" podID="d42e7888-5bfd-4f66-b898-3c75ce4f60b7" containerID="8f34b5eed7c48248ea7ef2d87d21c77c86c2977250cb3984b47057161b686c7e" exitCode=0
Jul 28 00:09:02.049044 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:02.048651    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp" event={"ID":"d42e7888-5bfd-4f66-b898-3c75ce4f60b7","Type":"ContainerDied","Data":"8f34b5eed7c48248ea7ef2d87d21c77c86c2977250cb3984b47057161b686c7e"}
Jul 28 00:09:02.049089 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:02.049054    2559 scope.go:117] "RemoveContainer" containerID="8f34b5eed7c48248ea7ef2d87d21c77c86c2977250cb3984b47057161b686c7e"
Jul 28 00:09:02.050164 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:02.050136    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 00:09:02.050228 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:02.050201    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 00:09:02.050265 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:02.050232    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 00:09:03.056170 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:03.056137    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp" event={"ID":"d42e7888-5bfd-4f66-b898-3c75ce4f60b7","Type":"ContainerStarted","Data":"ff0b039f7a26249526a5d10026065b60ab21209e47df3aeae31258de05a7e0d6"}
Jul 28 00:09:03.837572 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:03.837541    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:09:03.838627 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:03.838601    2559 patch_prober.go:28] interesting pod/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 28 00:09:03.838737 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:03.838653    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp" podUID="d42e7888-5bfd-4f66-b898-3c75ce4f60b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 28 00:09:03.975041 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:03.974987    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:09:04.837620 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:04.837576    2559 patch_prober.go:28] interesting pod/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 28 00:09:04.838041 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:04.837643    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp" podUID="d42e7888-5bfd-4f66-b898-3c75ce4f60b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 28 00:09:05.838332 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:05.838299    2559 patch_prober.go:28] interesting pod/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 28 00:09:05.838724 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:05.838357    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp" podUID="d42e7888-5bfd-4f66-b898-3c75ce4f60b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 28 00:09:06.837509 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:06.837474    2559 patch_prober.go:28] interesting pod/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 28 00:09:06.837689 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:06.837532    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp" podUID="d42e7888-5bfd-4f66-b898-3c75ce4f60b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 28 00:09:07.842184 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:07.842155    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:09:07.843222 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:07.843201    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:09:17.449345 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.449303    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"]
Jul 28 00:09:17.454442 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.454410    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.457570 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.457543    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4a6d94\"/\"kube-root-ca.crt\""
Jul 28 00:09:17.458235 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.458078    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4a6d94\"/\"openshift-service-ca.crt\""
Jul 28 00:09:17.458454 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.457543    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4a6d94\"/\"default-dockercfg-8594q\""
Jul 28 00:09:17.462177 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.462154    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"]
Jul 28 00:09:17.463319 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.457548    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4a6d94\"/\"ipp-route-4a6d94-kserve-self-signed-certs\""
Jul 28 00:09:17.545952 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.545915    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ba0139-ce71-4941-8f07-c8eaa7def1cf-tls-certs\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.546143 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.545975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn8gc\" (UniqueName: \"kubernetes.io/projected/49ba0139-ce71-4941-8f07-c8eaa7def1cf-kube-api-access-hn8gc\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.546143 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.546063    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-model-cache\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.546143 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.546096    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-home\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.546255 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.546167    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-tmp-dir\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.546255 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.546204    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-dshm\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.647050 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.647008    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn8gc\" (UniqueName: \"kubernetes.io/projected/49ba0139-ce71-4941-8f07-c8eaa7def1cf-kube-api-access-hn8gc\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.647246 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.647076    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-model-cache\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.647246 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.647107    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-home\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.647246 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.647135    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-tmp-dir\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.647246 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.647154    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-dshm\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.647246 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.647216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ba0139-ce71-4941-8f07-c8eaa7def1cf-tls-certs\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.647624 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.647603    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-home\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.647673 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.647647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-tmp-dir\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.647712 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.647694    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-model-cache\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.649668 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.649652    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-dshm\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.649908 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.649892    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ba0139-ce71-4941-8f07-c8eaa7def1cf-tls-certs\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.654701 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.654678    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn8gc\" (UniqueName: \"kubernetes.io/projected/49ba0139-ce71-4941-8f07-c8eaa7def1cf-kube-api-access-hn8gc\") pod \"ipp-route-4a6d94-kserve-7b5b687467-lb4qm\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") " pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.768478 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.768393    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:17.912432 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:17.912401    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"]
Jul 28 00:09:17.914480 ip-10-0-142-53 kubenswrapper[2559]: W0728 00:09:17.914453    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49ba0139_ce71_4941_8f07_c8eaa7def1cf.slice/crio-a12772750199ac2c99ea67757170b1b1480db1bfda23056610234ee7f79ca8f2 WatchSource:0}: Error finding container a12772750199ac2c99ea67757170b1b1480db1bfda23056610234ee7f79ca8f2: Status 404 returned error can't find the container with id a12772750199ac2c99ea67757170b1b1480db1bfda23056610234ee7f79ca8f2
Jul 28 00:09:18.145417 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:18.145346    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm" event={"ID":"49ba0139-ce71-4941-8f07-c8eaa7def1cf","Type":"ContainerStarted","Data":"6cf1a8a3b32107ab49736dcf691c5a30e7a3d30a4464f9a1b263d483e1f34c98"}
Jul 28 00:09:18.145631 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:18.145434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm" event={"ID":"49ba0139-ce71-4941-8f07-c8eaa7def1cf","Type":"ContainerStarted","Data":"a12772750199ac2c99ea67757170b1b1480db1bfda23056610234ee7f79ca8f2"}
Jul 28 00:09:18.160417 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:18.160340    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm" podStartSLOduration=1.160322409 podStartE2EDuration="1.160322409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:09:18.158024079 +0000 UTC m=+3181.102254825" watchObservedRunningTime="2026-07-28 00:09:18.160322409 +0000 UTC m=+3181.104553156"
Jul 28 00:09:27.769249 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:27.769214    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:27.769633 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:27.769262    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:27.781773 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:27.781750    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:09:28.219653 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:09:28.219621    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:10:22.126889 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:22.126806    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv"]
Jul 28 00:10:22.130525 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:22.130508    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv"
Jul 28 00:10:22.132535 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:22.132513    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 28 00:10:22.135754 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:22.135732    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv"]
Jul 28 00:10:22.191684 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:22.191655    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71beedc-5ddc-4bf7-908a-0ba1aafc99da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv\" (UID: \"d71beedc-5ddc-4bf7-908a-0ba1aafc99da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv"
Jul 28 00:10:22.292718 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:22.292676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71beedc-5ddc-4bf7-908a-0ba1aafc99da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv\" (UID: \"d71beedc-5ddc-4bf7-908a-0ba1aafc99da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv"
Jul 28 00:10:22.293301 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:22.293281    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71beedc-5ddc-4bf7-908a-0ba1aafc99da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv\" (UID: \"d71beedc-5ddc-4bf7-908a-0ba1aafc99da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv"
Jul 28 00:10:22.442863 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:22.442779    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv"
Jul 28 00:10:22.588936 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:22.588896    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv"]
Jul 28 00:10:22.595565 ip-10-0-142-53 kubenswrapper[2559]: W0728 00:10:22.595473    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71beedc_5ddc_4bf7_908a_0ba1aafc99da.slice/crio-b25b31b7a1b02af8df1a291b88a291017c79994fcc43fac3c43d40aad29407e2 WatchSource:0}: Error finding container b25b31b7a1b02af8df1a291b88a291017c79994fcc43fac3c43d40aad29407e2: Status 404 returned error can't find the container with id b25b31b7a1b02af8df1a291b88a291017c79994fcc43fac3c43d40aad29407e2
Jul 28 00:10:23.519194 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:23.519159    2559 generic.go:356] "Generic (PLEG): container finished" podID="d71beedc-5ddc-4bf7-908a-0ba1aafc99da" containerID="2a4f830386b58ba4e6114e8cce75b104946c826dd55e59609ce3763452b7b021" exitCode=0
Jul 28 00:10:23.519582 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:23.519243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv" event={"ID":"d71beedc-5ddc-4bf7-908a-0ba1aafc99da","Type":"ContainerDied","Data":"2a4f830386b58ba4e6114e8cce75b104946c826dd55e59609ce3763452b7b021"}
Jul 28 00:10:23.519582 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:23.519293    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv" event={"ID":"d71beedc-5ddc-4bf7-908a-0ba1aafc99da","Type":"ContainerStarted","Data":"b25b31b7a1b02af8df1a291b88a291017c79994fcc43fac3c43d40aad29407e2"}
Jul 28 00:10:24.664581 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:24.664551    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv"
Jul 28 00:10:24.722442 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:24.722413    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71beedc-5ddc-4bf7-908a-0ba1aafc99da-maas-api-service-ca\") pod \"d71beedc-5ddc-4bf7-908a-0ba1aafc99da\" (UID: \"d71beedc-5ddc-4bf7-908a-0ba1aafc99da\") "
Jul 28 00:10:24.722769 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:24.722735    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71beedc-5ddc-4bf7-908a-0ba1aafc99da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d71beedc-5ddc-4bf7-908a-0ba1aafc99da" (UID: "d71beedc-5ddc-4bf7-908a-0ba1aafc99da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:10:24.823805 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:24.823773    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71beedc-5ddc-4bf7-908a-0ba1aafc99da-maas-api-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:10:25.532206 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:25.532168    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv" event={"ID":"d71beedc-5ddc-4bf7-908a-0ba1aafc99da","Type":"ContainerDied","Data":"b25b31b7a1b02af8df1a291b88a291017c79994fcc43fac3c43d40aad29407e2"}
Jul 28 00:10:25.532417 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:25.532261    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b25b31b7a1b02af8df1a291b88a291017c79994fcc43fac3c43d40aad29407e2"
Jul 28 00:10:25.532417 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:25.532184    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv"
Jul 28 00:10:37.196985 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:37.196946    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv"]
Jul 28 00:10:37.200213 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:37.200175    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35fc9-zdkfv"]
Jul 28 00:10:37.563669 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:37.563629    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d71beedc-5ddc-4bf7-908a-0ba1aafc99da" path="/var/lib/kubelet/pods/d71beedc-5ddc-4bf7-908a-0ba1aafc99da/volumes"
Jul 28 00:10:46.988493 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:46.988447    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb"]
Jul 28 00:10:46.988880 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:46.988854    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d71beedc-5ddc-4bf7-908a-0ba1aafc99da" containerName="revoke-keys"
Jul 28 00:10:46.988880 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:46.988865    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d71beedc-5ddc-4bf7-908a-0ba1aafc99da" containerName="revoke-keys"
Jul 28 00:10:46.988952 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:46.988947    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d71beedc-5ddc-4bf7-908a-0ba1aafc99da" containerName="revoke-keys"
Jul 28 00:10:46.992193 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:46.992177    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb"
Jul 28 00:10:46.994255 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:46.994235    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 28 00:10:46.998660 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:46.998637    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb"]
Jul 28 00:10:47.139281 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:47.139244    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28249a10-359f-466a-9e49-5953aabadbb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb\" (UID: \"28249a10-359f-466a-9e49-5953aabadbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb"
Jul 28 00:10:47.240425 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:47.240317    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28249a10-359f-466a-9e49-5953aabadbb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb\" (UID: \"28249a10-359f-466a-9e49-5953aabadbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb"
Jul 28 00:10:47.240937 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:47.240917    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28249a10-359f-466a-9e49-5953aabadbb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb\" (UID: \"28249a10-359f-466a-9e49-5953aabadbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb"
Jul 28 00:10:47.305561 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:47.305522    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb"
Jul 28 00:10:47.459804 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:47.459768    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb"]
Jul 28 00:10:47.461047 ip-10-0-142-53 kubenswrapper[2559]: W0728 00:10:47.461010    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28249a10_359f_466a_9e49_5953aabadbb8.slice/crio-8d5497074bfb40e3e133665d138b6fc8577f7225582ae1bcf03260de2bdaae61 WatchSource:0}: Error finding container 8d5497074bfb40e3e133665d138b6fc8577f7225582ae1bcf03260de2bdaae61: Status 404 returned error can't find the container with id 8d5497074bfb40e3e133665d138b6fc8577f7225582ae1bcf03260de2bdaae61
Jul 28 00:10:47.659776 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:47.659736    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb" event={"ID":"28249a10-359f-466a-9e49-5953aabadbb8","Type":"ContainerStarted","Data":"8d5497074bfb40e3e133665d138b6fc8577f7225582ae1bcf03260de2bdaae61"}
Jul 28 00:10:48.666845 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:48.666750    2559 generic.go:356] "Generic (PLEG): container finished" podID="28249a10-359f-466a-9e49-5953aabadbb8" containerID="8e7cb363d8ebb9d49cea94bc12deefdcffccf4153545d1c9cbe5d7ae0e7d46d1" exitCode=0
Jul 28 00:10:48.666845 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:48.666808    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb" event={"ID":"28249a10-359f-466a-9e49-5953aabadbb8","Type":"ContainerDied","Data":"8e7cb363d8ebb9d49cea94bc12deefdcffccf4153545d1c9cbe5d7ae0e7d46d1"}
Jul 28 00:10:49.816286 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:49.816258    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb"
Jul 28 00:10:49.965288 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:49.965205    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28249a10-359f-466a-9e49-5953aabadbb8-maas-api-service-ca\") pod \"28249a10-359f-466a-9e49-5953aabadbb8\" (UID: \"28249a10-359f-466a-9e49-5953aabadbb8\") "
Jul 28 00:10:49.965602 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:49.965579    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28249a10-359f-466a-9e49-5953aabadbb8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "28249a10-359f-466a-9e49-5953aabadbb8" (UID: "28249a10-359f-466a-9e49-5953aabadbb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:10:50.066923 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:50.066886    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28249a10-359f-466a-9e49-5953aabadbb8-maas-api-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:10:50.678258 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:50.678228    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb"
Jul 28 00:10:50.678484 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:50.678259    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb" event={"ID":"28249a10-359f-466a-9e49-5953aabadbb8","Type":"ContainerDied","Data":"8d5497074bfb40e3e133665d138b6fc8577f7225582ae1bcf03260de2bdaae61"}
Jul 28 00:10:50.678484 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:10:50.678309    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d5497074bfb40e3e133665d138b6fc8577f7225582ae1bcf03260de2bdaae61"
Jul 28 00:11:07.057893 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:07.057851    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb"]
Jul 28 00:11:07.060144 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:07.060118    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4a6d94-bg5bb"]
Jul 28 00:11:07.564792 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:07.564764    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28249a10-359f-466a-9e49-5953aabadbb8" path="/var/lib/kubelet/pods/28249a10-359f-466a-9e49-5953aabadbb8/volumes"
Jul 28 00:11:12.424750 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:12.424711    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"]
Jul 28 00:11:17.774895 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:17.774868    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 28 00:11:17.781221 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:17.781200    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c27p5_5eedceb6-d520-4108-8b3d-5c6b933d1fb1/ovn-acl-logging/0.log"
Jul 28 00:11:27.445224 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.445161    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm" podUID="49ba0139-ce71-4941-8f07-c8eaa7def1cf" containerName="main" containerID="cri-o://6cf1a8a3b32107ab49736dcf691c5a30e7a3d30a4464f9a1b263d483e1f34c98" gracePeriod=45
Jul 28 00:11:27.708250 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.708197    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:11:27.848663 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.848636    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn8gc\" (UniqueName: \"kubernetes.io/projected/49ba0139-ce71-4941-8f07-c8eaa7def1cf-kube-api-access-hn8gc\") pod \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") "
Jul 28 00:11:27.848841 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.848711    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-model-cache\") pod \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") "
Jul 28 00:11:27.848841 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.848733    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-tmp-dir\") pod \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") "
Jul 28 00:11:27.848841 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.848754    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-dshm\") pod \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") "
Jul 28 00:11:27.848841 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.848775    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ba0139-ce71-4941-8f07-c8eaa7def1cf-tls-certs\") pod \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") "
Jul 28 00:11:27.848841 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.848827    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-home\") pod \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\" (UID: \"49ba0139-ce71-4941-8f07-c8eaa7def1cf\") "
Jul 28 00:11:27.849091 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.848983    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "49ba0139-ce71-4941-8f07-c8eaa7def1cf" (UID: "49ba0139-ce71-4941-8f07-c8eaa7def1cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:11:27.849158 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.849087    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "49ba0139-ce71-4941-8f07-c8eaa7def1cf" (UID: "49ba0139-ce71-4941-8f07-c8eaa7def1cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:11:27.849214 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.849158    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-home" (OuterVolumeSpecName: "home") pod "49ba0139-ce71-4941-8f07-c8eaa7def1cf" (UID: "49ba0139-ce71-4941-8f07-c8eaa7def1cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:11:27.849265 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.849243    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-model-cache\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:11:27.849265 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.849257    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-tmp-dir\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:11:27.849330 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.849265    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-home\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:11:27.851057 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.851030    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ba0139-ce71-4941-8f07-c8eaa7def1cf-kube-api-access-hn8gc" (OuterVolumeSpecName: "kube-api-access-hn8gc") pod "49ba0139-ce71-4941-8f07-c8eaa7def1cf" (UID: "49ba0139-ce71-4941-8f07-c8eaa7def1cf"). InnerVolumeSpecName "kube-api-access-hn8gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:11:27.851454 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.851431    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ba0139-ce71-4941-8f07-c8eaa7def1cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49ba0139-ce71-4941-8f07-c8eaa7def1cf" (UID: "49ba0139-ce71-4941-8f07-c8eaa7def1cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:11:27.851454 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.851438    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-dshm" (OuterVolumeSpecName: "dshm") pod "49ba0139-ce71-4941-8f07-c8eaa7def1cf" (UID: "49ba0139-ce71-4941-8f07-c8eaa7def1cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:11:27.889418 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.889389    2559 generic.go:356] "Generic (PLEG): container finished" podID="49ba0139-ce71-4941-8f07-c8eaa7def1cf" containerID="6cf1a8a3b32107ab49736dcf691c5a30e7a3d30a4464f9a1b263d483e1f34c98" exitCode=0
Jul 28 00:11:27.889552 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.889447    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"
Jul 28 00:11:27.889552 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.889463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm" event={"ID":"49ba0139-ce71-4941-8f07-c8eaa7def1cf","Type":"ContainerDied","Data":"6cf1a8a3b32107ab49736dcf691c5a30e7a3d30a4464f9a1b263d483e1f34c98"}
Jul 28 00:11:27.889552 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.889505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm" event={"ID":"49ba0139-ce71-4941-8f07-c8eaa7def1cf","Type":"ContainerDied","Data":"a12772750199ac2c99ea67757170b1b1480db1bfda23056610234ee7f79ca8f2"}
Jul 28 00:11:27.889552 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.889523    2559 scope.go:117] "RemoveContainer" containerID="6cf1a8a3b32107ab49736dcf691c5a30e7a3d30a4464f9a1b263d483e1f34c98"
Jul 28 00:11:27.904305 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.904283    2559 scope.go:117] "RemoveContainer" containerID="6cf1a8a3b32107ab49736dcf691c5a30e7a3d30a4464f9a1b263d483e1f34c98"
Jul 28 00:11:27.904605 ip-10-0-142-53 kubenswrapper[2559]: E0728 00:11:27.904585    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf1a8a3b32107ab49736dcf691c5a30e7a3d30a4464f9a1b263d483e1f34c98\": container with ID starting with 6cf1a8a3b32107ab49736dcf691c5a30e7a3d30a4464f9a1b263d483e1f34c98 not found: ID does not exist" containerID="6cf1a8a3b32107ab49736dcf691c5a30e7a3d30a4464f9a1b263d483e1f34c98"
Jul 28 00:11:27.904662 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.904614    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf1a8a3b32107ab49736dcf691c5a30e7a3d30a4464f9a1b263d483e1f34c98"} err="failed to get container status \"6cf1a8a3b32107ab49736dcf691c5a30e7a3d30a4464f9a1b263d483e1f34c98\": rpc error: code = NotFound desc = could not find container \"6cf1a8a3b32107ab49736dcf691c5a30e7a3d30a4464f9a1b263d483e1f34c98\": container with ID starting with 6cf1a8a3b32107ab49736dcf691c5a30e7a3d30a4464f9a1b263d483e1f34c98 not found: ID does not exist"
Jul 28 00:11:27.913843 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.913815    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"]
Jul 28 00:11:27.917319 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.917294    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-4a6d94/ipp-route-4a6d94-kserve-7b5b687467-lb4qm"]
Jul 28 00:11:27.950254 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.950228    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn8gc\" (UniqueName: \"kubernetes.io/projected/49ba0139-ce71-4941-8f07-c8eaa7def1cf-kube-api-access-hn8gc\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:11:27.950254 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.950251    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ba0139-ce71-4941-8f07-c8eaa7def1cf-dshm\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:11:27.950416 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:27.950262    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ba0139-ce71-4941-8f07-c8eaa7def1cf-tls-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:11:29.563828 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:29.563795    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49ba0139-ce71-4941-8f07-c8eaa7def1cf" path="/var/lib/kubelet/pods/49ba0139-ce71-4941-8f07-c8eaa7def1cf/volumes"
Jul 28 00:11:55.085576 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.085488    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"]
Jul 28 00:11:55.086152 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.085803    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5" podUID="5e852dad-40c4-4599-bc05-55fa1f48859a" containerName="istio-proxy" containerID="cri-o://58dba90bc70ae3d07dec985dd397125fb0ac38809156bc696cc7ee9ca4c44ec2" gracePeriod=30
Jul 28 00:11:55.470358 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.470269    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb"]
Jul 28 00:11:55.470744 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.470730    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28249a10-359f-466a-9e49-5953aabadbb8" containerName="revoke-keys"
Jul 28 00:11:55.470796 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.470745    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="28249a10-359f-466a-9e49-5953aabadbb8" containerName="revoke-keys"
Jul 28 00:11:55.470796 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.470775    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ba0139-ce71-4941-8f07-c8eaa7def1cf" containerName="main"
Jul 28 00:11:55.470796 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.470784    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ba0139-ce71-4941-8f07-c8eaa7def1cf" containerName="main"
Jul 28 00:11:55.470889 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.470852    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ba0139-ce71-4941-8f07-c8eaa7def1cf" containerName="main"
Jul 28 00:11:55.470889 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.470867    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="28249a10-359f-466a-9e49-5953aabadbb8" containerName="revoke-keys"
Jul 28 00:11:55.474527 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.474511    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb"
Jul 28 00:11:55.476526 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.476497    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 28 00:11:55.480572 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.480549    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb"]
Jul 28 00:11:55.621856 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.621818    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b23677-ca03-4427-b62e-35820ce53e17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb\" (UID: \"e5b23677-ca03-4427-b62e-35820ce53e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb"
Jul 28 00:11:55.722826 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.722730    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b23677-ca03-4427-b62e-35820ce53e17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb\" (UID: \"e5b23677-ca03-4427-b62e-35820ce53e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb"
Jul 28 00:11:55.723359 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.723338    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b23677-ca03-4427-b62e-35820ce53e17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb\" (UID: \"e5b23677-ca03-4427-b62e-35820ce53e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb"
Jul 28 00:11:55.788264 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.788227    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb"
Jul 28 00:11:55.927454 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:55.927421    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb"]
Jul 28 00:11:55.929350 ip-10-0-142-53 kubenswrapper[2559]: W0728 00:11:55.929316    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b23677_ca03_4427_b62e_35820ce53e17.slice/crio-22326e55d94c3b496e1d4839a6de6fcc055931c34b6307d23cf4a72f0d98757b WatchSource:0}: Error finding container 22326e55d94c3b496e1d4839a6de6fcc055931c34b6307d23cf4a72f0d98757b: Status 404 returned error can't find the container with id 22326e55d94c3b496e1d4839a6de6fcc055931c34b6307d23cf4a72f0d98757b
Jul 28 00:11:56.050192 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:56.050141    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb" event={"ID":"e5b23677-ca03-4427-b62e-35820ce53e17","Type":"ContainerStarted","Data":"22326e55d94c3b496e1d4839a6de6fcc055931c34b6307d23cf4a72f0d98757b"}
Jul 28 00:11:57.057480 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:57.057442    2559 generic.go:356] "Generic (PLEG): container finished" podID="e5b23677-ca03-4427-b62e-35820ce53e17" containerID="07a361e5b9e98db7512248424eeacc2f0a087b06bba6f08e17124478df3a26cf" exitCode=0
Jul 28 00:11:57.057912 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:57.057512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb" event={"ID":"e5b23677-ca03-4427-b62e-35820ce53e17","Type":"ContainerDied","Data":"07a361e5b9e98db7512248424eeacc2f0a087b06bba6f08e17124478df3a26cf"}
Jul 28 00:11:58.206312 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:58.206283    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb"
Jul 28 00:11:58.347400 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:58.347288    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b23677-ca03-4427-b62e-35820ce53e17-maas-api-service-ca\") pod \"e5b23677-ca03-4427-b62e-35820ce53e17\" (UID: \"e5b23677-ca03-4427-b62e-35820ce53e17\") "
Jul 28 00:11:58.347596 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:58.347576    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b23677-ca03-4427-b62e-35820ce53e17-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5b23677-ca03-4427-b62e-35820ce53e17" (UID: "e5b23677-ca03-4427-b62e-35820ce53e17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:11:58.448479 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:58.448428    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b23677-ca03-4427-b62e-35820ce53e17-maas-api-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:11:59.069476 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:59.069443    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb" event={"ID":"e5b23677-ca03-4427-b62e-35820ce53e17","Type":"ContainerDied","Data":"22326e55d94c3b496e1d4839a6de6fcc055931c34b6307d23cf4a72f0d98757b"}
Jul 28 00:11:59.069476 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:59.069461    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb"
Jul 28 00:11:59.069689 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:11:59.069491    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22326e55d94c3b496e1d4839a6de6fcc055931c34b6307d23cf4a72f0d98757b"
Jul 28 00:12:00.352052 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.351987    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:12:00.468778 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.468741    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-credential-socket\") pod \"5e852dad-40c4-4599-bc05-55fa1f48859a\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") "
Jul 28 00:12:00.468956 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.468844    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwq5q\" (UniqueName: \"kubernetes.io/projected/5e852dad-40c4-4599-bc05-55fa1f48859a-kube-api-access-gwq5q\") pod \"5e852dad-40c4-4599-bc05-55fa1f48859a\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") "
Jul 28 00:12:00.468956 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.468876    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-token\") pod \"5e852dad-40c4-4599-bc05-55fa1f48859a\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") "
Jul 28 00:12:00.468956 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.468894    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-data\") pod \"5e852dad-40c4-4599-bc05-55fa1f48859a\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") "
Jul 28 00:12:00.468956 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.468941    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e852dad-40c4-4599-bc05-55fa1f48859a-istiod-ca-cert\") pod \"5e852dad-40c4-4599-bc05-55fa1f48859a\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") "
Jul 28 00:12:00.469128 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.468974    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-envoy\") pod \"5e852dad-40c4-4599-bc05-55fa1f48859a\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") "
Jul 28 00:12:00.469128 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.469014    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-podinfo\") pod \"5e852dad-40c4-4599-bc05-55fa1f48859a\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") "
Jul 28 00:12:00.469128 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.469040    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-workload-socket\") pod \"5e852dad-40c4-4599-bc05-55fa1f48859a\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") "
Jul 28 00:12:00.469128 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.469038    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e852dad-40c4-4599-bc05-55fa1f48859a" (UID: "5e852dad-40c4-4599-bc05-55fa1f48859a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:00.469128 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.469061    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-workload-certs\") pod \"5e852dad-40c4-4599-bc05-55fa1f48859a\" (UID: \"5e852dad-40c4-4599-bc05-55fa1f48859a\") "
Jul 28 00:12:00.469449 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.469415    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e852dad-40c4-4599-bc05-55fa1f48859a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e852dad-40c4-4599-bc05-55fa1f48859a" (UID: "5e852dad-40c4-4599-bc05-55fa1f48859a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:12:00.469642 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.469609    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e852dad-40c4-4599-bc05-55fa1f48859a" (UID: "5e852dad-40c4-4599-bc05-55fa1f48859a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:00.469968 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.469897    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-credential-socket\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:00.470054 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.469966    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e852dad-40c4-4599-bc05-55fa1f48859a" (UID: "5e852dad-40c4-4599-bc05-55fa1f48859a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:00.470194 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.470168    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e852dad-40c4-4599-bc05-55fa1f48859a" (UID: "5e852dad-40c4-4599-bc05-55fa1f48859a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:00.471354 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.471324    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e852dad-40c4-4599-bc05-55fa1f48859a" (UID: "5e852dad-40c4-4599-bc05-55fa1f48859a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:12:00.471479 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.471411    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e852dad-40c4-4599-bc05-55fa1f48859a-kube-api-access-gwq5q" (OuterVolumeSpecName: "kube-api-access-gwq5q") pod "5e852dad-40c4-4599-bc05-55fa1f48859a" (UID: "5e852dad-40c4-4599-bc05-55fa1f48859a"). InnerVolumeSpecName "kube-api-access-gwq5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:12:00.471479 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.471410    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e852dad-40c4-4599-bc05-55fa1f48859a" (UID: "5e852dad-40c4-4599-bc05-55fa1f48859a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:00.471479 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.471455    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e852dad-40c4-4599-bc05-55fa1f48859a" (UID: "5e852dad-40c4-4599-bc05-55fa1f48859a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 00:12:00.570846 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.570804    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwq5q\" (UniqueName: \"kubernetes.io/projected/5e852dad-40c4-4599-bc05-55fa1f48859a-kube-api-access-gwq5q\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:00.570846 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.570840    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-token\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:00.570846 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.570850    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-data\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:00.570846 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.570858    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e852dad-40c4-4599-bc05-55fa1f48859a-istiod-ca-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:00.571119 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.570869    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-envoy\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:00.571119 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.570876    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e852dad-40c4-4599-bc05-55fa1f48859a-istio-podinfo\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:00.571119 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.570884    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-workload-socket\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:00.571119 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:00.570892    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e852dad-40c4-4599-bc05-55fa1f48859a-workload-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:01.081465 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:01.081431    2559 generic.go:356] "Generic (PLEG): container finished" podID="5e852dad-40c4-4599-bc05-55fa1f48859a" containerID="58dba90bc70ae3d07dec985dd397125fb0ac38809156bc696cc7ee9ca4c44ec2" exitCode=0
Jul 28 00:12:01.081634 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:01.081481    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"
Jul 28 00:12:01.081634 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:01.081492    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5" event={"ID":"5e852dad-40c4-4599-bc05-55fa1f48859a","Type":"ContainerDied","Data":"58dba90bc70ae3d07dec985dd397125fb0ac38809156bc696cc7ee9ca4c44ec2"}
Jul 28 00:12:01.081634 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:01.081535    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5" event={"ID":"5e852dad-40c4-4599-bc05-55fa1f48859a","Type":"ContainerDied","Data":"6816310944f7dd5ccdcceec3645f9043049f27d3c27e110e37aeafb68ffc4b48"}
Jul 28 00:12:01.081634 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:01.081556    2559 scope.go:117] "RemoveContainer" containerID="58dba90bc70ae3d07dec985dd397125fb0ac38809156bc696cc7ee9ca4c44ec2"
Jul 28 00:12:01.096805 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:01.096783    2559 scope.go:117] "RemoveContainer" containerID="ce2c18237195ae189279cdc265148af356701ad67786a56d2f0fa899588a8fc3"
Jul 28 00:12:01.106186 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:01.106167    2559 scope.go:117] "RemoveContainer" containerID="58dba90bc70ae3d07dec985dd397125fb0ac38809156bc696cc7ee9ca4c44ec2"
Jul 28 00:12:01.106459 ip-10-0-142-53 kubenswrapper[2559]: E0728 00:12:01.106432    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58dba90bc70ae3d07dec985dd397125fb0ac38809156bc696cc7ee9ca4c44ec2\": container with ID starting with 58dba90bc70ae3d07dec985dd397125fb0ac38809156bc696cc7ee9ca4c44ec2 not found: ID does not exist" containerID="58dba90bc70ae3d07dec985dd397125fb0ac38809156bc696cc7ee9ca4c44ec2"
Jul 28 00:12:01.106530 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:01.106469    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58dba90bc70ae3d07dec985dd397125fb0ac38809156bc696cc7ee9ca4c44ec2"} err="failed to get container status \"58dba90bc70ae3d07dec985dd397125fb0ac38809156bc696cc7ee9ca4c44ec2\": rpc error: code = NotFound desc = could not find container \"58dba90bc70ae3d07dec985dd397125fb0ac38809156bc696cc7ee9ca4c44ec2\": container with ID starting with 58dba90bc70ae3d07dec985dd397125fb0ac38809156bc696cc7ee9ca4c44ec2 not found: ID does not exist"
Jul 28 00:12:01.106530 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:01.106486    2559 scope.go:117] "RemoveContainer" containerID="ce2c18237195ae189279cdc265148af356701ad67786a56d2f0fa899588a8fc3"
Jul 28 00:12:01.106735 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:01.106711    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"]
Jul 28 00:12:01.106817 ip-10-0-142-53 kubenswrapper[2559]: E0728 00:12:01.106711    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2c18237195ae189279cdc265148af356701ad67786a56d2f0fa899588a8fc3\": container with ID starting with ce2c18237195ae189279cdc265148af356701ad67786a56d2f0fa899588a8fc3 not found: ID does not exist" containerID="ce2c18237195ae189279cdc265148af356701ad67786a56d2f0fa899588a8fc3"
Jul 28 00:12:01.106817 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:01.106768    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2c18237195ae189279cdc265148af356701ad67786a56d2f0fa899588a8fc3"} err="failed to get container status \"ce2c18237195ae189279cdc265148af356701ad67786a56d2f0fa899588a8fc3\": rpc error: code = NotFound desc = could not find container \"ce2c18237195ae189279cdc265148af356701ad67786a56d2f0fa899588a8fc3\": container with ID starting with ce2c18237195ae189279cdc265148af356701ad67786a56d2f0fa899588a8fc3 not found: ID does not exist"
Jul 28 00:12:01.109469 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:01.109439    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4a6d94-openshift-default-7df8cc5c4b-grlt5"]
Jul 28 00:12:01.564656 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:01.564624    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e852dad-40c4-4599-bc05-55fa1f48859a" path="/var/lib/kubelet/pods/5e852dad-40c4-4599-bc05-55fa1f48859a/volumes"
Jul 28 00:12:10.549975 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:10.549945    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb"]
Jul 28 00:12:10.552802 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:10.552776    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-87c90a-dvftb"]
Jul 28 00:12:11.563584 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:11.563551    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5b23677-ca03-4427-b62e-35820ce53e17" path="/var/lib/kubelet/pods/e5b23677-ca03-4427-b62e-35820ce53e17/volumes"
Jul 28 00:12:16.941567 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:16.941535    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"]
Jul 28 00:12:16.942054 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:16.941746    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp" podUID="d42e7888-5bfd-4f66-b898-3c75ce4f60b7" containerName="istio-proxy" containerID="cri-o://ff0b039f7a26249526a5d10026065b60ab21209e47df3aeae31258de05a7e0d6" gracePeriod=30
Jul 28 00:12:22.205400 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.205316    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:12:22.207270 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.207237    2559 generic.go:356] "Generic (PLEG): container finished" podID="d42e7888-5bfd-4f66-b898-3c75ce4f60b7" containerID="ff0b039f7a26249526a5d10026065b60ab21209e47df3aeae31258de05a7e0d6" exitCode=0
Jul 28 00:12:22.207425 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.207282    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"
Jul 28 00:12:22.207425 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.207289    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp" event={"ID":"d42e7888-5bfd-4f66-b898-3c75ce4f60b7","Type":"ContainerDied","Data":"ff0b039f7a26249526a5d10026065b60ab21209e47df3aeae31258de05a7e0d6"}
Jul 28 00:12:22.207425 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.207354    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp" event={"ID":"d42e7888-5bfd-4f66-b898-3c75ce4f60b7","Type":"ContainerDied","Data":"f84155249ed88feeb2e26fcd33e4ad80cd905565b74a0c4792593f95870e4b75"}
Jul 28 00:12:22.207425 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.207395    2559 scope.go:117] "RemoveContainer" containerID="ff0b039f7a26249526a5d10026065b60ab21209e47df3aeae31258de05a7e0d6"
Jul 28 00:12:22.216181 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.216162    2559 scope.go:117] "RemoveContainer" containerID="8f34b5eed7c48248ea7ef2d87d21c77c86c2977250cb3984b47057161b686c7e"
Jul 28 00:12:22.225822 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.225726    2559 scope.go:117] "RemoveContainer" containerID="ff0b039f7a26249526a5d10026065b60ab21209e47df3aeae31258de05a7e0d6"
Jul 28 00:12:22.226061 ip-10-0-142-53 kubenswrapper[2559]: E0728 00:12:22.226039    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0b039f7a26249526a5d10026065b60ab21209e47df3aeae31258de05a7e0d6\": container with ID starting with ff0b039f7a26249526a5d10026065b60ab21209e47df3aeae31258de05a7e0d6 not found: ID does not exist" containerID="ff0b039f7a26249526a5d10026065b60ab21209e47df3aeae31258de05a7e0d6"
Jul 28 00:12:22.226153 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.226070    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0b039f7a26249526a5d10026065b60ab21209e47df3aeae31258de05a7e0d6"} err="failed to get container status \"ff0b039f7a26249526a5d10026065b60ab21209e47df3aeae31258de05a7e0d6\": rpc error: code = NotFound desc = could not find container \"ff0b039f7a26249526a5d10026065b60ab21209e47df3aeae31258de05a7e0d6\": container with ID starting with ff0b039f7a26249526a5d10026065b60ab21209e47df3aeae31258de05a7e0d6 not found: ID does not exist"
Jul 28 00:12:22.226153 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.226088    2559 scope.go:117] "RemoveContainer" containerID="8f34b5eed7c48248ea7ef2d87d21c77c86c2977250cb3984b47057161b686c7e"
Jul 28 00:12:22.226404 ip-10-0-142-53 kubenswrapper[2559]: E0728 00:12:22.226357    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f34b5eed7c48248ea7ef2d87d21c77c86c2977250cb3984b47057161b686c7e\": container with ID starting with 8f34b5eed7c48248ea7ef2d87d21c77c86c2977250cb3984b47057161b686c7e not found: ID does not exist" containerID="8f34b5eed7c48248ea7ef2d87d21c77c86c2977250cb3984b47057161b686c7e"
Jul 28 00:12:22.226564 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.226536    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f34b5eed7c48248ea7ef2d87d21c77c86c2977250cb3984b47057161b686c7e"} err="failed to get container status \"8f34b5eed7c48248ea7ef2d87d21c77c86c2977250cb3984b47057161b686c7e\": rpc error: code = NotFound desc = could not find container \"8f34b5eed7c48248ea7ef2d87d21c77c86c2977250cb3984b47057161b686c7e\": container with ID starting with 8f34b5eed7c48248ea7ef2d87d21c77c86c2977250cb3984b47057161b686c7e not found: ID does not exist"
Jul 28 00:12:22.262705 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.262673    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-token\") pod \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") "
Jul 28 00:12:22.262890 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.262749    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-podinfo\") pod \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") "
Jul 28 00:12:22.262890 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.262784    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-envoy\") pod \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") "
Jul 28 00:12:22.262890 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.262804    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-workload-certs\") pod \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") "
Jul 28 00:12:22.262890 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.262828    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-workload-socket\") pod \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") "
Jul 28 00:12:22.262890 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.262858    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zjhl\" (UniqueName: \"kubernetes.io/projected/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-kube-api-access-5zjhl\") pod \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") "
Jul 28 00:12:22.262890 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.262888    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-data\") pod \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") "
Jul 28 00:12:22.263207 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.262911    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istiod-ca-cert\") pod \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") "
Jul 28 00:12:22.263207 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.262939    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-credential-socket\") pod \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\" (UID: \"d42e7888-5bfd-4f66-b898-3c75ce4f60b7\") "
Jul 28 00:12:22.263207 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.263125    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d42e7888-5bfd-4f66-b898-3c75ce4f60b7" (UID: "d42e7888-5bfd-4f66-b898-3c75ce4f60b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:22.263365 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.263290    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-workload-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:22.263365 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.263336    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d42e7888-5bfd-4f66-b898-3c75ce4f60b7" (UID: "d42e7888-5bfd-4f66-b898-3c75ce4f60b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:22.263530 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.263358    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d42e7888-5bfd-4f66-b898-3c75ce4f60b7" (UID: "d42e7888-5bfd-4f66-b898-3c75ce4f60b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:22.263793 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.263764    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "d42e7888-5bfd-4f66-b898-3c75ce4f60b7" (UID: "d42e7888-5bfd-4f66-b898-3c75ce4f60b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:22.263912 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.263797    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d42e7888-5bfd-4f66-b898-3c75ce4f60b7" (UID: "d42e7888-5bfd-4f66-b898-3c75ce4f60b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:12:22.265167 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.265142    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "d42e7888-5bfd-4f66-b898-3c75ce4f60b7" (UID: "d42e7888-5bfd-4f66-b898-3c75ce4f60b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:12:22.265330 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.265311    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-kube-api-access-5zjhl" (OuterVolumeSpecName: "kube-api-access-5zjhl") pod "d42e7888-5bfd-4f66-b898-3c75ce4f60b7" (UID: "d42e7888-5bfd-4f66-b898-3c75ce4f60b7"). InnerVolumeSpecName "kube-api-access-5zjhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:12:22.265616 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.265600    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d42e7888-5bfd-4f66-b898-3c75ce4f60b7" (UID: "d42e7888-5bfd-4f66-b898-3c75ce4f60b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:22.265690 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.265619    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d42e7888-5bfd-4f66-b898-3c75ce4f60b7" (UID: "d42e7888-5bfd-4f66-b898-3c75ce4f60b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 00:12:22.364198 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.364160    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-workload-socket\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:22.364198 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.364190    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zjhl\" (UniqueName: \"kubernetes.io/projected/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-kube-api-access-5zjhl\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:22.364198 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.364200    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-data\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:22.364198 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.364210    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istiod-ca-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:22.364523 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.364218    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-credential-socket\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:22.364523 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.364226    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-token\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:22.364523 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.364234    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-podinfo\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:22.364523 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.364241    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42e7888-5bfd-4f66-b898-3c75ce4f60b7-istio-envoy\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:22.528859 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.528831    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"]
Jul 28 00:12:22.532522 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:22.532497    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-87c90a-openshift-default-79b65dfd6d-zvfbp"]
Jul 28 00:12:23.562809 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:23.562774    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d42e7888-5bfd-4f66-b898-3c75ce4f60b7" path="/var/lib/kubelet/pods/d42e7888-5bfd-4f66-b898-3c75ce4f60b7/volumes"
Jul 28 00:12:27.795285 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:27.795251    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-42d790-29753280-c6h7b"]
Jul 28 00:12:27.798884 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:27.798853    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-42d790-29753280-c6h7b"]
Jul 28 00:12:29.563152 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:29.563117    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="254effe6-6f66-41ac-b99a-59a443838587" path="/var/lib/kubelet/pods/254effe6-6f66-41ac-b99a-59a443838587/volumes"
Jul 28 00:12:39.130136 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.130095    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9"]
Jul 28 00:12:39.130629 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.130490    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e852dad-40c4-4599-bc05-55fa1f48859a" containerName="istio-proxy"
Jul 28 00:12:39.130629 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.130501    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e852dad-40c4-4599-bc05-55fa1f48859a" containerName="istio-proxy"
Jul 28 00:12:39.130629 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.130514    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5b23677-ca03-4427-b62e-35820ce53e17" containerName="revoke-keys"
Jul 28 00:12:39.130629 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.130520    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5b23677-ca03-4427-b62e-35820ce53e17" containerName="revoke-keys"
Jul 28 00:12:39.130629 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.130529    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d42e7888-5bfd-4f66-b898-3c75ce4f60b7" containerName="istio-proxy"
Jul 28 00:12:39.130629 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.130540    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d42e7888-5bfd-4f66-b898-3c75ce4f60b7" containerName="istio-proxy"
Jul 28 00:12:39.130889 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.130644    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e852dad-40c4-4599-bc05-55fa1f48859a" containerName="istio-proxy"
Jul 28 00:12:39.130889 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.130658    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5b23677-ca03-4427-b62e-35820ce53e17" containerName="revoke-keys"
Jul 28 00:12:39.130889 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.130671    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d42e7888-5bfd-4f66-b898-3c75ce4f60b7" containerName="istio-proxy"
Jul 28 00:12:39.130889 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.130682    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d42e7888-5bfd-4f66-b898-3c75ce4f60b7" containerName="istio-proxy"
Jul 28 00:12:39.134113 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.134095    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9"
Jul 28 00:12:39.136094 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.136075    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 28 00:12:39.139697 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.139674    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9"]
Jul 28 00:12:39.226216 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.226182    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f66c6f-fb53-44ee-884f-0477fa72e6a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9\" (UID: \"16f66c6f-fb53-44ee-884f-0477fa72e6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9"
Jul 28 00:12:39.327182 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.327145    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f66c6f-fb53-44ee-884f-0477fa72e6a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9\" (UID: \"16f66c6f-fb53-44ee-884f-0477fa72e6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9"
Jul 28 00:12:39.327810 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.327789    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f66c6f-fb53-44ee-884f-0477fa72e6a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9\" (UID: \"16f66c6f-fb53-44ee-884f-0477fa72e6a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9"
Jul 28 00:12:39.445631 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.445531    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9"
Jul 28 00:12:39.588016 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:39.587962    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9"]
Jul 28 00:12:39.590462 ip-10-0-142-53 kubenswrapper[2559]: W0728 00:12:39.590431    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f66c6f_fb53_44ee_884f_0477fa72e6a1.slice/crio-1c23ff650681d521a8a9d93d060414eea5d50e08714dd1190af6d383f538bbc5 WatchSource:0}: Error finding container 1c23ff650681d521a8a9d93d060414eea5d50e08714dd1190af6d383f538bbc5: Status 404 returned error can't find the container with id 1c23ff650681d521a8a9d93d060414eea5d50e08714dd1190af6d383f538bbc5
Jul 28 00:12:40.300195 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:40.300164    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9" event={"ID":"16f66c6f-fb53-44ee-884f-0477fa72e6a1","Type":"ContainerStarted","Data":"438be637ff1690643e70738affe1c5f98c0891b988f97e76e02484fb9379a994"}
Jul 28 00:12:40.300528 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:40.300223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9" event={"ID":"16f66c6f-fb53-44ee-884f-0477fa72e6a1","Type":"ContainerStarted","Data":"1c23ff650681d521a8a9d93d060414eea5d50e08714dd1190af6d383f538bbc5"}
Jul 28 00:12:41.306772 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:41.306717    2559 generic.go:356] "Generic (PLEG): container finished" podID="16f66c6f-fb53-44ee-884f-0477fa72e6a1" containerID="438be637ff1690643e70738affe1c5f98c0891b988f97e76e02484fb9379a994" exitCode=0
Jul 28 00:12:41.306772 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:41.306767    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9" event={"ID":"16f66c6f-fb53-44ee-884f-0477fa72e6a1","Type":"ContainerDied","Data":"438be637ff1690643e70738affe1c5f98c0891b988f97e76e02484fb9379a994"}
Jul 28 00:12:41.454469 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:41.454439    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9"
Jul 28 00:12:41.550447 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:41.550408    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f66c6f-fb53-44ee-884f-0477fa72e6a1-maas-api-service-ca\") pod \"16f66c6f-fb53-44ee-884f-0477fa72e6a1\" (UID: \"16f66c6f-fb53-44ee-884f-0477fa72e6a1\") "
Jul 28 00:12:41.550733 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:41.550712    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f66c6f-fb53-44ee-884f-0477fa72e6a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16f66c6f-fb53-44ee-884f-0477fa72e6a1" (UID: "16f66c6f-fb53-44ee-884f-0477fa72e6a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:12:41.651098 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:41.651064    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16f66c6f-fb53-44ee-884f-0477fa72e6a1-maas-api-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:12:42.313471 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:42.313436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9" event={"ID":"16f66c6f-fb53-44ee-884f-0477fa72e6a1","Type":"ContainerDied","Data":"1c23ff650681d521a8a9d93d060414eea5d50e08714dd1190af6d383f538bbc5"}
Jul 28 00:12:42.313471 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:42.313465    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9"
Jul 28 00:12:42.313907 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:42.313487    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c23ff650681d521a8a9d93d060414eea5d50e08714dd1190af6d383f538bbc5"
Jul 28 00:12:49.517741 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:49.517701    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a8122-29753280-gdk9j"]
Jul 28 00:12:49.522159 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:49.522129    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4a8122-29753280-gdk9j"]
Jul 28 00:12:49.564436 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:49.564402    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="009101ee-24c6-4e5b-bf21-4315e55a655f" path="/var/lib/kubelet/pods/009101ee-24c6-4e5b-bf21-4315e55a655f/volumes"
Jul 28 00:12:54.206311 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:54.206272    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9"]
Jul 28 00:12:54.208346 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:54.208315    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4a8122-kv8v9"]
Jul 28 00:12:55.564315 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:12:55.564285    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16f66c6f-fb53-44ee-884f-0477fa72e6a1" path="/var/lib/kubelet/pods/16f66c6f-fb53-44ee-884f-0477fa72e6a1/volumes"
Jul 28 00:13:00.723871 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:00.719264    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"]
Jul 28 00:13:00.723871 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:00.719645    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn" podUID="3aa6fdee-2279-43ac-94fb-1a8d3b657a24" containerName="istio-proxy" containerID="cri-o://767fba1f53708c8fe519423b962447e9c9c142c55e9838f74df814742f1929b6" gracePeriod=30
Jul 28 00:13:06.004447 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.004390    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 28 00:13:06.075366 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.075328    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istiod-ca-cert\") pod \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") "
Jul 28 00:13:06.075564 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.075396    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-workload-certs\") pod \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") "
Jul 28 00:13:06.075564 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.075423    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-token\") pod \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") "
Jul 28 00:13:06.075564 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.075475    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-workload-socket\") pod \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") "
Jul 28 00:13:06.075564 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.075496    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-data\") pod \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") "
Jul 28 00:13:06.075564 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.075531    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-envoy\") pod \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") "
Jul 28 00:13:06.075830 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.075573    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-credential-socket\") pod \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") "
Jul 28 00:13:06.075830 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.075592    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfbsp\" (UniqueName: \"kubernetes.io/projected/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-kube-api-access-nfbsp\") pod \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") "
Jul 28 00:13:06.075830 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.075645    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-podinfo\") pod \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\" (UID: \"3aa6fdee-2279-43ac-94fb-1a8d3b657a24\") "
Jul 28 00:13:06.075830 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.075726    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3aa6fdee-2279-43ac-94fb-1a8d3b657a24" (UID: "3aa6fdee-2279-43ac-94fb-1a8d3b657a24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:13:06.075830 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.075764    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3aa6fdee-2279-43ac-94fb-1a8d3b657a24" (UID: "3aa6fdee-2279-43ac-94fb-1a8d3b657a24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:13:06.075830 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.075785    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3aa6fdee-2279-43ac-94fb-1a8d3b657a24" (UID: "3aa6fdee-2279-43ac-94fb-1a8d3b657a24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:13:06.076119 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.075974    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3aa6fdee-2279-43ac-94fb-1a8d3b657a24" (UID: "3aa6fdee-2279-43ac-94fb-1a8d3b657a24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:13:06.076119 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.076077    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-workload-socket\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:13:06.076119 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.076097    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-credential-socket\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:13:06.076119 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.076110    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istiod-ca-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:13:06.076309 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.076123    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-workload-certs\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:13:06.076620 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.076595    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-data" (OuterVolumeSpecName: "istio-data") pod "3aa6fdee-2279-43ac-94fb-1a8d3b657a24" (UID: "3aa6fdee-2279-43ac-94fb-1a8d3b657a24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:13:06.077861 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.077839    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-token" (OuterVolumeSpecName: "istio-token") pod "3aa6fdee-2279-43ac-94fb-1a8d3b657a24" (UID: "3aa6fdee-2279-43ac-94fb-1a8d3b657a24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:13:06.078103 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.078078    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3aa6fdee-2279-43ac-94fb-1a8d3b657a24" (UID: "3aa6fdee-2279-43ac-94fb-1a8d3b657a24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:13:06.078399 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.078363    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-kube-api-access-nfbsp" (OuterVolumeSpecName: "kube-api-access-nfbsp") pod "3aa6fdee-2279-43ac-94fb-1a8d3b657a24" (UID: "3aa6fdee-2279-43ac-94fb-1a8d3b657a24"). InnerVolumeSpecName "kube-api-access-nfbsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:13:06.078399 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.078385    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3aa6fdee-2279-43ac-94fb-1a8d3b657a24" (UID: "3aa6fdee-2279-43ac-94fb-1a8d3b657a24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 00:13:06.176963 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.176912    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-envoy\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:13:06.176963 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.176955    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfbsp\" (UniqueName: \"kubernetes.io/projected/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-kube-api-access-nfbsp\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:13:06.176963 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.176967    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-podinfo\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:13:06.176963 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.176975    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-token\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:13:06.176963 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.176983    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa6fdee-2279-43ac-94fb-1a8d3b657a24-istio-data\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
Jul 28 00:13:06.405288 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.405255    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-77g9r_faf1a861-2bf2-4abf-8ac9-be78538a6360/manager/0.log"
Jul 28 00:13:06.446207 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.446170    2559 generic.go:356] "Generic (PLEG): container finished" podID="3aa6fdee-2279-43ac-94fb-1a8d3b657a24" containerID="767fba1f53708c8fe519423b962447e9c9c142c55e9838f74df814742f1929b6" exitCode=0
Jul 28 00:13:06.446386 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.446213    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"
Jul 28 00:13:06.446386 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.446220    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn" event={"ID":"3aa6fdee-2279-43ac-94fb-1a8d3b657a24","Type":"ContainerDied","Data":"767fba1f53708c8fe519423b962447e9c9c142c55e9838f74df814742f1929b6"}
Jul 28 00:13:06.446386 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.446272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn" event={"ID":"3aa6fdee-2279-43ac-94fb-1a8d3b657a24","Type":"ContainerDied","Data":"2f33ef24f7b297d0d9c07d4993150a3c4ec13f1f9fca501a433b68df2b589a94"}
Jul 28 00:13:06.446386 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.446297    2559 scope.go:117] "RemoveContainer" containerID="767fba1f53708c8fe519423b962447e9c9c142c55e9838f74df814742f1929b6"
Jul 28 00:13:06.456016 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.455996    2559 scope.go:117] "RemoveContainer" containerID="7935c265bc6e52545fd68562743a09c16104c2c66c552e986fc40e4440f1ddea"
Jul 28 00:13:06.467195 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.467108    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"]
Jul 28 00:13:06.469447 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.469426    2559 scope.go:117] "RemoveContainer" containerID="767fba1f53708c8fe519423b962447e9c9c142c55e9838f74df814742f1929b6"
Jul 28 00:13:06.469760 ip-10-0-142-53 kubenswrapper[2559]: E0728 00:13:06.469739    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767fba1f53708c8fe519423b962447e9c9c142c55e9838f74df814742f1929b6\": container with ID starting with 767fba1f53708c8fe519423b962447e9c9c142c55e9838f74df814742f1929b6 not found: ID does not exist" containerID="767fba1f53708c8fe519423b962447e9c9c142c55e9838f74df814742f1929b6"
Jul 28 00:13:06.469831 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.469769    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767fba1f53708c8fe519423b962447e9c9c142c55e9838f74df814742f1929b6"} err="failed to get container status \"767fba1f53708c8fe519423b962447e9c9c142c55e9838f74df814742f1929b6\": rpc error: code = NotFound desc = could not find container \"767fba1f53708c8fe519423b962447e9c9c142c55e9838f74df814742f1929b6\": container with ID starting with 767fba1f53708c8fe519423b962447e9c9c142c55e9838f74df814742f1929b6 not found: ID does not exist"
Jul 28 00:13:06.469831 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.469789    2559 scope.go:117] "RemoveContainer" containerID="7935c265bc6e52545fd68562743a09c16104c2c66c552e986fc40e4440f1ddea"
Jul 28 00:13:06.470056 ip-10-0-142-53 kubenswrapper[2559]: E0728 00:13:06.470027    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7935c265bc6e52545fd68562743a09c16104c2c66c552e986fc40e4440f1ddea\": container with ID starting with 7935c265bc6e52545fd68562743a09c16104c2c66c552e986fc40e4440f1ddea not found: ID does not exist" containerID="7935c265bc6e52545fd68562743a09c16104c2c66c552e986fc40e4440f1ddea"
Jul 28 00:13:06.470126 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.470051    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7935c265bc6e52545fd68562743a09c16104c2c66c552e986fc40e4440f1ddea"} err="failed to get container status \"7935c265bc6e52545fd68562743a09c16104c2c66c552e986fc40e4440f1ddea\": rpc error: code = NotFound desc = could not find container \"7935c265bc6e52545fd68562743a09c16104c2c66c552e986fc40e4440f1ddea\": container with ID starting with 7935c265bc6e52545fd68562743a09c16104c2c66c552e986fc40e4440f1ddea not found: ID does not exist"
Jul 28 00:13:06.470834 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.470810    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-4a8122-openshift-default-867974454d-bk6vn"]
Jul 28 00:13:06.516628 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.516602    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5f5cm_423b58a7-f70f-4da3-8923-f4ac83305728/manager/0.log"
Jul 28 00:13:06.747509 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.747423    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dwjk9_1ed194b2-4778-4b8e-b542-0e503143d00d/server/0.log"
Jul 28 00:13:06.863771 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.863742    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kdl5f_701ecf62-8af6-466f-b84b-f29df06b55fb/manager/0.log"
Jul 28 00:13:06.984588 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.984549    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9cf867746-hb2hn_2f6fc328-0e1c-4e46-8a6d-81460e187c3d/manager/0.log"
Jul 28 00:13:06.996070 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:06.996046    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9cf867746-hb2hn_2f6fc328-0e1c-4e46-8a6d-81460e187c3d/copy-manifests/0.log"
Jul 28 00:13:07.562956 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:07.562921    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aa6fdee-2279-43ac-94fb-1a8d3b657a24" path="/var/lib/kubelet/pods/3aa6fdee-2279-43ac-94fb-1a8d3b657a24/volumes"
Jul 28 00:13:07.886283 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:07.886203    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753280-698wv_e1386c35-ad17-494a-9b57-aa9c3729968d/cleanup/0.log"
Jul 28 00:13:07.986779 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:07.986740    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-k7xmj_a0a555a0-6df4-448d-9a64-4d5aae82b5f4/postgres/0.log"
Jul 28 00:13:09.325771 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:09.325732    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-nsdgv_fb75458a-d253-4bea-a666-344d36bee972/manager/0.log"
Jul 28 00:13:09.433443 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:09.433405    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-qqv9h_605263d2-8841-417a-974e-640173936e64/manager/0.log"
Jul 28 00:13:09.647589 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:09.647509    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-5fksn_2468e0bf-d810-4b49-9ea8-c48e2d0f9ed0/registry-server/0.log"
Jul 28 00:13:09.753846 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:09.753815    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-q87m7_a536fc25-5bcf-4327-8659-a8b1867dbe93/manager/0.log"
Jul 28 00:13:09.972553 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:09.972477    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-k46lx_36f61d75-c765-4434-a582-344825e5cd38/manager/0.log"
Jul 28 00:13:10.301481 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:10.301458    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps_008c5090-63d0-4f0e-9f7a-ef065dfbdb6e/istio-proxy/0.log"
Jul 28 00:13:10.518212 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:10.518181    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5946b997db-8t42c_69d6b5c0-df07-4c77-97c8-8a5cb5f70520/kube-auth-proxy/0.log"
Jul 28 00:13:10.729184 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:10.729103    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hbnzn_d706cd9a-be1b-4544-9740-12f63c54c73e/istio-proxy/1.log"
Jul 28 00:13:11.605877 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:11.605847    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-glx4q_6224811f-02b3-4951-8227-fc2329e9971a/main/0.log"
Jul 28 00:13:11.707111 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:11.707080    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f95l4s_419fad95-9938-4b45-aa29-3081ef8a8b4e/main/0.log"
Jul 28 00:13:11.920807 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:11.920722    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-pqh5t_d0a63841-fa2a-482b-a58f-1933106becbe/main/0.log"
Jul 28 00:13:18.014640 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:18.014613    2559 scope.go:117] "RemoveContainer" containerID="d223cc517943ca01108dd669165d3134f389669f68bf417d34ecf69fc899009b"
Jul 28 00:13:18.023073 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:18.023052    2559 scope.go:117] "RemoveContainer" containerID="e2f1a51e3d2653fe6b3a826564c6726c5d30992fbad2aee3dcab876d06de8d64"
Jul 28 00:13:19.650083 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:19.650051    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fhqmk_03e4d553-7ad4-4072-acf6-1d8424cbff5b/global-pull-secret-syncer/0.log"
Jul 28 00:13:19.848015 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:19.847980    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bcgpc_4cf21fbf-dc73-405a-836a-f5cf4a7c64d8/konnectivity-agent/0.log"
Jul 28 00:13:19.911730 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:19.911660    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-53.ec2.internal_a64776cf4968745c952331caeb9a6baf/haproxy/0.log"
Jul 28 00:13:25.160352 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:25.160302    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-nsdgv_fb75458a-d253-4bea-a666-344d36bee972/manager/0.log"
Jul 28 00:13:25.187024 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:25.186991    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-qqv9h_605263d2-8841-417a-974e-640173936e64/manager/0.log"
Jul 28 00:13:25.254845 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:25.254819    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-5fksn_2468e0bf-d810-4b49-9ea8-c48e2d0f9ed0/registry-server/0.log"
Jul 28 00:13:25.418289 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:25.418191    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-q87m7_a536fc25-5bcf-4327-8659-a8b1867dbe93/manager/0.log"
Jul 28 00:13:25.518269 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:25.518239    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-k46lx_36f61d75-c765-4434-a582-344825e5cd38/manager/0.log"
Jul 28 00:13:27.524361 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:27.524328    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kxntv_181cca67-ce91-4b5b-b2dc-d905813b1b16/node-exporter/0.log"
Jul 28 00:13:27.542399 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:27.542357    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kxntv_181cca67-ce91-4b5b-b2dc-d905813b1b16/kube-rbac-proxy/0.log"
Jul 28 00:13:27.565309 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:27.565286    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kxntv_181cca67-ce91-4b5b-b2dc-d905813b1b16/init-textfile/0.log"
Jul 28 00:13:27.666932 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:27.666908    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-dskc8_21dfaac0-d939-42c1-a6d7-8259fd89dc40/kube-rbac-proxy-main/0.log"
Jul 28 00:13:27.685990 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:27.685932    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-dskc8_21dfaac0-d939-42c1-a6d7-8259fd89dc40/kube-rbac-proxy-self/0.log"
Jul 28 00:13:27.704302 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:27.704275    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-dskc8_21dfaac0-d939-42c1-a6d7-8259fd89dc40/openshift-state-metrics/0.log"
Jul 28 00:13:27.738957 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:27.738939    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a07de815-9624-4abe-bbf5-00fcf93109fd/prometheus/0.log"
Jul 28 00:13:27.762184 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:27.762167    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a07de815-9624-4abe-bbf5-00fcf93109fd/config-reloader/0.log"
Jul 28 00:13:27.789702 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:27.789682    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a07de815-9624-4abe-bbf5-00fcf93109fd/thanos-sidecar/0.log"
Jul 28 00:13:27.810586 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:27.810558    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a07de815-9624-4abe-bbf5-00fcf93109fd/kube-rbac-proxy-web/0.log"
Jul 28 00:13:27.830011 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:27.829982    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a07de815-9624-4abe-bbf5-00fcf93109fd/kube-rbac-proxy/0.log"
Jul 28 00:13:27.848964 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:27.848943    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a07de815-9624-4abe-bbf5-00fcf93109fd/kube-rbac-proxy-thanos/0.log"
Jul 28 00:13:27.867732 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:27.867705    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a07de815-9624-4abe-bbf5-00fcf93109fd/init-config-reloader/0.log"
Jul 28 00:13:27.938543 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:27.938475    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-gpjrq_3545a128-2826-409f-a4bc-a91cc78d34a2/prometheus-operator-admission-webhook/0.log"
Jul 28 00:13:30.047554 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.047515    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"]
Jul 28 00:13:30.047996 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.047971    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16f66c6f-fb53-44ee-884f-0477fa72e6a1" containerName="revoke-keys"
Jul 28 00:13:30.047996 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.047988    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="16f66c6f-fb53-44ee-884f-0477fa72e6a1" containerName="revoke-keys"
Jul 28 00:13:30.048090 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.048007    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e852dad-40c4-4599-bc05-55fa1f48859a" containerName="istio-proxy"
Jul 28 00:13:30.048090 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.048016    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e852dad-40c4-4599-bc05-55fa1f48859a" containerName="istio-proxy"
Jul 28 00:13:30.048090 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.048031    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa6fdee-2279-43ac-94fb-1a8d3b657a24" containerName="istio-proxy"
Jul 28 00:13:30.048090 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.048037    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa6fdee-2279-43ac-94fb-1a8d3b657a24" containerName="istio-proxy"
Jul 28 00:13:30.048090 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.048049    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d42e7888-5bfd-4f66-b898-3c75ce4f60b7" containerName="istio-proxy"
Jul 28 00:13:30.048090 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.048054    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d42e7888-5bfd-4f66-b898-3c75ce4f60b7" containerName="istio-proxy"
Jul 28 00:13:30.048268 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.048134    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa6fdee-2279-43ac-94fb-1a8d3b657a24" containerName="istio-proxy"
Jul 28 00:13:30.048268 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.048144    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa6fdee-2279-43ac-94fb-1a8d3b657a24" containerName="istio-proxy"
Jul 28 00:13:30.048268 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.048149    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e852dad-40c4-4599-bc05-55fa1f48859a" containerName="istio-proxy"
Jul 28 00:13:30.048268 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.048157    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="16f66c6f-fb53-44ee-884f-0477fa72e6a1" containerName="revoke-keys"
Jul 28 00:13:30.051271 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.051254    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.053164 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.053136    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9lrsk\"/\"kube-root-ca.crt\""
Jul 28 00:13:30.053486 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.053469    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9lrsk\"/\"openshift-service-ca.crt\""
Jul 28 00:13:30.053912 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.053900    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9lrsk\"/\"default-dockercfg-d9m9z\""
Jul 28 00:13:30.057653 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.057631    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"]
Jul 28 00:13:30.101515 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.101475    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4016e777-30ea-4911-b016-0a7af3b6a5fe-proc\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.101702 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.101578    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4016e777-30ea-4911-b016-0a7af3b6a5fe-lib-modules\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.101702 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.101647    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htf4b\" (UniqueName: \"kubernetes.io/projected/4016e777-30ea-4911-b016-0a7af3b6a5fe-kube-api-access-htf4b\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.101810 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.101723    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4016e777-30ea-4911-b016-0a7af3b6a5fe-podres\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.101810 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.101766    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4016e777-30ea-4911-b016-0a7af3b6a5fe-sys\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.202265 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.202221    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4016e777-30ea-4911-b016-0a7af3b6a5fe-proc\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.202509 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.202285    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4016e777-30ea-4911-b016-0a7af3b6a5fe-lib-modules\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.202509 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.202337    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htf4b\" (UniqueName: \"kubernetes.io/projected/4016e777-30ea-4911-b016-0a7af3b6a5fe-kube-api-access-htf4b\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.202509 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.202357    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4016e777-30ea-4911-b016-0a7af3b6a5fe-proc\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.202509 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.202468    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4016e777-30ea-4911-b016-0a7af3b6a5fe-podres\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.202509 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.202494    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4016e777-30ea-4911-b016-0a7af3b6a5fe-sys\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.202735 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.202510    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4016e777-30ea-4911-b016-0a7af3b6a5fe-lib-modules\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.202735 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.202592    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4016e777-30ea-4911-b016-0a7af3b6a5fe-sys\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.202735 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.202678    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4016e777-30ea-4911-b016-0a7af3b6a5fe-podres\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.209241 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.209211    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htf4b\" (UniqueName: \"kubernetes.io/projected/4016e777-30ea-4911-b016-0a7af3b6a5fe-kube-api-access-htf4b\") pod \"perf-node-gather-daemonset-g5wsr\" (UID: \"4016e777-30ea-4911-b016-0a7af3b6a5fe\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.362243 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.362150    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:30.434839 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.434801    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-nrjl2_2423c014-fc07-4851-addd-b65230b2631e/download-server/0.log"
Jul 28 00:13:30.500108 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.500082    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"]
Jul 28 00:13:30.502066 ip-10-0-142-53 kubenswrapper[2559]: W0728 00:13:30.502030    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4016e777_30ea_4911_b016_0a7af3b6a5fe.slice/crio-fb74b4a35c8b539f264b50500024bf211c11046af4c439555cb4903e727c40ff WatchSource:0}: Error finding container fb74b4a35c8b539f264b50500024bf211c11046af4c439555cb4903e727c40ff: Status 404 returned error can't find the container with id fb74b4a35c8b539f264b50500024bf211c11046af4c439555cb4903e727c40ff
Jul 28 00:13:30.576347 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:30.576315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr" event={"ID":"4016e777-30ea-4911-b016-0a7af3b6a5fe","Type":"ContainerStarted","Data":"fb74b4a35c8b539f264b50500024bf211c11046af4c439555cb4903e727c40ff"}
Jul 28 00:13:31.581985 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:31.581946    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr" event={"ID":"4016e777-30ea-4911-b016-0a7af3b6a5fe","Type":"ContainerStarted","Data":"4416b44a93f4de33a548ccb2e73a326ed2e22333c917e8a57845392da7234c8a"}
Jul 28 00:13:31.582448 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:31.582076    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:31.596121 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:31.596069    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr" podStartSLOduration=1.59605595 podStartE2EDuration="1.59605595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:13:31.594647686 +0000 UTC m=+3434.538878433" watchObservedRunningTime="2026-07-28 00:13:31.59605595 +0000 UTC m=+3434.540286697"
Jul 28 00:13:32.094063 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:32.094032    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cgwgh_4e467024-a191-49ad-a8ec-29e7923d652e/dns/0.log"
Jul 28 00:13:32.112355 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:32.112319    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cgwgh_4e467024-a191-49ad-a8ec-29e7923d652e/kube-rbac-proxy/0.log"
Jul 28 00:13:32.217864 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:32.217819    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h8v75_7ec18511-c498-4407-93eb-d0edabcd4311/dns-node-resolver/0.log"
Jul 28 00:13:32.819571 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:32.819534    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mzw9d_906d2765-8007-4aed-8c0d-2aa12031fa5a/node-ca/0.log"
Jul 28 00:13:33.746910 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:33.746883    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps_008c5090-63d0-4f0e-9f7a-ef065dfbdb6e/istio-proxy/0.log"
Jul 28 00:13:34.047070 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:34.047045    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5946b997db-8t42c_69d6b5c0-df07-4c77-97c8-8a5cb5f70520/kube-auth-proxy/0.log"
Jul 28 00:13:34.097057 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:34.097023    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hbnzn_d706cd9a-be1b-4544-9740-12f63c54c73e/istio-proxy/0.log"
Jul 28 00:13:34.122641 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:34.122610    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hbnzn_d706cd9a-be1b-4544-9740-12f63c54c73e/istio-proxy/1.log"
Jul 28 00:13:35.042476 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:35.042444    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rcxmb_6cb94d17-291c-4e88-8164-7e4664815fdf/serve-healthcheck-canary/0.log"
Jul 28 00:13:35.650505 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:35.650474    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-brxvq_45968a96-5d68-474f-8cc5-2a1520d7fbcc/kube-rbac-proxy/0.log"
Jul 28 00:13:35.669735 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:35.669709    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-brxvq_45968a96-5d68-474f-8cc5-2a1520d7fbcc/exporter/0.log"
Jul 28 00:13:35.689058 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:35.689025    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-brxvq_45968a96-5d68-474f-8cc5-2a1520d7fbcc/extractor/0.log"
Jul 28 00:13:37.598123 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:37.598095    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-g5wsr"
Jul 28 00:13:37.758424 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:37.758392    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-77g9r_faf1a861-2bf2-4abf-8ac9-be78538a6360/manager/0.log"
Jul 28 00:13:37.964116 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:37.964008    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5f5cm_423b58a7-f70f-4da3-8923-f4ac83305728/manager/0.log"
Jul 28 00:13:38.214815 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:38.214710    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dwjk9_1ed194b2-4778-4b8e-b542-0e503143d00d/server/0.log"
Jul 28 00:13:38.286920 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:38.286891    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kdl5f_701ecf62-8af6-466f-b84b-f29df06b55fb/manager/0.log"
Jul 28 00:13:38.371053 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:38.371014    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9cf867746-hb2hn_2f6fc328-0e1c-4e46-8a6d-81460e187c3d/manager/0.log"
Jul 28 00:13:38.387632 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:38.387609    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9cf867746-hb2hn_2f6fc328-0e1c-4e46-8a6d-81460e187c3d/copy-manifests/0.log"
Jul 28 00:13:40.356337 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:40.356255    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps_008c5090-63d0-4f0e-9f7a-ef065dfbdb6e/istio-proxy/0.log"
Jul 28 00:13:40.630799 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:40.630706    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5946b997db-8t42c_69d6b5c0-df07-4c77-97c8-8a5cb5f70520/kube-auth-proxy/0.log"
Jul 28 00:13:40.662998 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:40.662966    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hbnzn_d706cd9a-be1b-4544-9740-12f63c54c73e/istio-proxy/0.log"
Jul 28 00:13:40.688812 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:40.688785    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hbnzn_d706cd9a-be1b-4544-9740-12f63c54c73e/istio-proxy/1.log"
Jul 28 00:13:42.235937 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:42.235905    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-n47ps_008c5090-63d0-4f0e-9f7a-ef065dfbdb6e/istio-proxy/0.log"
Jul 28 00:13:42.509745 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:42.509664    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5946b997db-8t42c_69d6b5c0-df07-4c77-97c8-8a5cb5f70520/kube-auth-proxy/0.log"
Jul 28 00:13:42.544200 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:42.544169    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hbnzn_d706cd9a-be1b-4544-9740-12f63c54c73e/istio-proxy/0.log"
Jul 28 00:13:42.573313 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:42.573279    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hbnzn_d706cd9a-be1b-4544-9740-12f63c54c73e/istio-proxy/1.log"
Jul 28 00:13:46.499639 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:46.499566    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nxt99_549494f8-ca88-4fde-84cd-06f3af312efc/kube-multus-additional-cni-plugins/0.log"
Jul 28 00:13:46.518239 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:46.518213    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nxt99_549494f8-ca88-4fde-84cd-06f3af312efc/egress-router-binary-copy/0.log"
Jul 28 00:13:46.536637 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:46.536617    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nxt99_549494f8-ca88-4fde-84cd-06f3af312efc/cni-plugins/0.log"
Jul 28 00:13:46.556325 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:46.556304    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nxt99_549494f8-ca88-4fde-84cd-06f3af312efc/bond-cni-plugin/0.log"
Jul 28 00:13:46.575774 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:46.575753    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nxt99_549494f8-ca88-4fde-84cd-06f3af312efc/routeoverride-cni/0.log"
Jul 28 00:13:46.594800 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:46.594781    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nxt99_549494f8-ca88-4fde-84cd-06f3af312efc/whereabouts-cni-bincopy/0.log"
Jul 28 00:13:46.614568 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:46.614543    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nxt99_549494f8-ca88-4fde-84cd-06f3af312efc/whereabouts-cni/0.log"
Jul 28 00:13:46.650943 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:46.650925    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n75xf_01afb354-a5aa-41d8-ae0a-5028e75ae3e9/kube-multus/0.log"
Jul 28 00:13:46.704466 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:46.704449    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hrrmv_ee785ab1-83ca-4254-b144-c99f9114660f/network-metrics-daemon/0.log"
Jul 28 00:13:46.721127 ip-10-0-142-53 kubenswrapper[2559]: I0728 00:13:46.721107    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hrrmv_ee785ab1-83ca-4254-b144-c99f9114660f/kube-rbac-proxy/0.log"