Jul 27 19:17:35.624940 ip-10-0-129-103 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 19:17:37.828280 ip-10-0-129-103 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 19:17:37.828280 ip-10-0-129-103 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 19:17:37.828280 ip-10-0-129-103 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 19:17:37.828280 ip-10-0-129-103 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 19:17:37.828280 ip-10-0-129-103 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 19:17:37.828917 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.828355    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 19:17:37.835388 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835346    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:17:37.835496 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835486    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:17:37.835496 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835495    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835499    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835502    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835505    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835508    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835511    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835516    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835518    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835521    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835525    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835528    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835531    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835533    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835536    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835539    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835541    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835544    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835547    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835550    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835553    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:17:37.835559 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835555    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835559    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835562    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835565    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835568    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835573    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835577    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835580    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835583    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835587    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835590    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835593    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835596    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835598    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835601    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835604    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835607    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835609    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835612    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835615    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:17:37.836035 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835617    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835620    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835623    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835625    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835628    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835632    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835635    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835638    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835640    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835644    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835646    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835649    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835651    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835656    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835661    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835665    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835669    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835673    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835676    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:17:37.836538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835679    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835682    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835685    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835688    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835691    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835694    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835696    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835699    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835704    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835706    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835709    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835712    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835714    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835717    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835719    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835722    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835724    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835727    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835730    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835732    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:17:37.837036 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835735    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835737    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835741    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835743    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835746    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835750    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835753    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835756    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835760    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835763    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835765    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835768    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835771    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835773    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835776    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835778    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835781    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.835784    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836495    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:17:37.837534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836502    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836505    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836508    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836510    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836513    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836517    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836520    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836523    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836526    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836530    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836534    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836536    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836539    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836542    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836545    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836548    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836551    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836554    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836557    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:17:37.838063 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836560    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836563    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836566    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836568    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836571    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836574    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836576    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836579    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836581    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836584    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836587    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836589    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836592    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836594    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836597    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836600    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836602    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836605    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836608    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836611    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:17:37.838552 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836614    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836616    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836619    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836621    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836624    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836627    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836629    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836632    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836634    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836637    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836641    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836643    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836646    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836648    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836651    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836654    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836657    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836660    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836662    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836672    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:17:37.839040 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836675    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836678    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836680    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836683    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836685    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836688    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836690    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836693    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836695    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836698    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836700    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836703    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836707    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836711    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836715    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836718    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836720    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836723    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836726    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836728    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:17:37.839534 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836731    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836733    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836736    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836739    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836742    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836745    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836748    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836753    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836755    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836758    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836760    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836763    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836765    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836768    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836770    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836773    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836775    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836778    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.836780    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837152    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:17:37.840012 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837157    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837160    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837163    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837165    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837168    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837170    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837173    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837177    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837180    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837183    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837186    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837189    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837191    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837194    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837197    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837204    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837210    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837213    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837216    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:17:37.840514 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837221    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837224    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837227    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837230    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837233    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837235    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837257    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837260    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837263    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837265    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837268    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837270    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837273    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837276    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837279    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837281    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837284    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837288    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837291    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:17:37.840996 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837294    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837297    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837299    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837302    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837305    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837307    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837309    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837312    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837314    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837317    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837321    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837324    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837327    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837331    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837334    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837336    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837339    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837341    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837344    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837346    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:17:37.841470 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837349    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837352    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837354    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837356    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837359    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837361    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837364    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837367    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837369    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837371    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837374    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837376    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837379    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837381    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837384    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837387    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837389    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837391    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837394    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837396    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:17:37.841978 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837399    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837401    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837404    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837407    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837410    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837413    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837416    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837418    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837422    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837425    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837427    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837430    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837432    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837435    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837437    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837440    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837442    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837445    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837447    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.837449    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:17:37.842479 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837515    2580 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837522    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837528    2580 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837532    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837536    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837539    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837543    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837547    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837550    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837553    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837557    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837561    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837564    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837567    2580 flags.go:64] FLAG: --cgroup-root=""
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837570    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837573    2580 flags.go:64] FLAG: --client-ca-file=""
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837577    2580 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837595    2580 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837601    2580 flags.go:64] FLAG: --cluster-domain=""
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837605    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837608    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837611    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837615    2580 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837618    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 19:17:37.842951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837621    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837624    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837627    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837630    2580 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837633    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837636    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837639    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837642    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837646    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837649    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837652    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837655    2580 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837657    2580 flags.go:64] FLAG: --enable-server="true"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837660    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837664    2580 flags.go:64] FLAG: --event-burst="100"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837668    2580 flags.go:64] FLAG: --event-qps="50"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837670    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837673    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837676    2580 flags.go:64] FLAG: --eviction-hard=""
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837683    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837687    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837690    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837693    2580 flags.go:64] FLAG: --eviction-soft=""
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837696    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837699    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 19:17:37.843533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837703    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837706    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837709    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837712    2580 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837715    2580 flags.go:64] FLAG: --feature-gates=""
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837718    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837721    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837724    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837727    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837730    2580 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837733    2580 flags.go:64] FLAG: --help="false"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837736    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-129-103.ec2.internal"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837739    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837742    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837745    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837748    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837751    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837754    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837757    2580 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837760    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837762    2580 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837765    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837768    2580 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837771    2580 flags.go:64] FLAG: --kube-reserved=""
Jul 27 19:17:37.844149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837774    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837776    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837780    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837783    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837786    2580 flags.go:64] FLAG: --lock-file=""
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837789    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837792    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837795    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837801    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837805    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837808    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837811    2580 flags.go:64] FLAG: --logging-format="text"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837814    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837817    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837819    2580 flags.go:64] FLAG: --manifest-url=""
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837822    2580 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837826    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837829    2580 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837834    2580 flags.go:64] FLAG: --max-pods="110"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837836    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837839    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837842    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837845    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837848    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837851    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 19:17:37.845038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837854    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837861    2580 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837864    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837867    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837869    2580 flags.go:64] FLAG: --pod-cidr=""
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837872    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837877    2580 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837880    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837883    2580 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837886    2580 flags.go:64] FLAG: --port="10250"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837889    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837893    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f97c78ba82199d0c"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837896    2580 flags.go:64] FLAG: --qos-reserved=""
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837899    2580 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837902    2580 flags.go:64] FLAG: --register-node="true"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837906    2580 flags.go:64] FLAG: --register-with-taints=""
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837909    2580 flags.go:64] FLAG: --registry-burst="10"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837913    2580 flags.go:64] FLAG: --registry-qps="5"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837916    2580 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837919    2580 flags.go:64] FLAG: --reserved-memory=""
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837922    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837925    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837928    2580 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837931    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837934    2580 flags.go:64] FLAG: --runonce="false"
Jul 27 19:17:37.846138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837937    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837939    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837942    2580 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837945    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837948    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837951    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837954    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837957    2580 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837960    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837963    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837965    2580 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837968    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837971    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837974    2580 flags.go:64] FLAG: --system-cgroups=""
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837977    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837982    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837984    2580 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837989    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837993    2580 flags.go:64] FLAG: --tls-min-version=""
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837996    2580 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.837998    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.838001    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.838005    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.838008    2580 flags.go:64] FLAG: --v="2"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.838012    2580 flags.go:64] FLAG: --version="false"
Jul 27 19:17:37.847015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.838016    2580 flags.go:64] FLAG: --vmodule=""
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.838019    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.838023    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838147    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838152    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838154    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838157    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838160    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838163    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838165    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838168    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838171    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838173    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838176    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838178    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838181    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838184    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838186    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838189    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838192    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:17:37.847671 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838194    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838197    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838199    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838202    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838206    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838208    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838211    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838213    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838216    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838219    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838222    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838224    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838227    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838230    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838232    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838235    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838250    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838253    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838256    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838259    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:17:37.848364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838262    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838264    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838267    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838269    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838272    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838275    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838278    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838280    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838283    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838285    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838289    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838293    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838296    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838299    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838302    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838306    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838310    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838313    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838316    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:17:37.849261 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838318    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838321    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838324    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838327    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838329    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838332    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838335    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838338    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838340    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838343    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838346    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838348    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838351    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838354    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838356    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838359    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838361    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838364    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838366    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838369    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:17:37.849807 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838371    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838374    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838376    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838379    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838382    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838386    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838389    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838392    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838395    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838400    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838403    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838405    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838408    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838411    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838414    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838417    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838419    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838422    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838424    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:17:37.850581 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838427    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838430    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838433    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.838435    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.839109    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.846673    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.846692    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.847929    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.847941    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848023    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848030    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848035    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848041    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848045    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848050    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:17:37.851372 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848054    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848059    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848063    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848067    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848071    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848076    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848080    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848084    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848088    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848093    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848097    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848101    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848106    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848110    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848114    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848118    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848123    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848128    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848133    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848137    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:17:37.851749 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848141    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848145    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848149    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848154    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848159    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848163    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848167    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848171    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848176    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848180    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848184    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848188    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848193    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848198    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848204    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848210    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848215    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848219    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848223    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:17:37.852323 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848228    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848232    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848236    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848256    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848260    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848264    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848269    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848273    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848277    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848282    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848286    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848290    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848295    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848299    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848303    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848308    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848312    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848317    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848323    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848329    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:17:37.852902 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848333    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848337    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848341    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848346    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848350    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848354    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848359    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848363    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848367    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848372    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848376    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848380    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848383    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848387    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848391    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848396    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848400    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848403    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848408    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:17:37.853474 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848413    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848417    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848421    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848425    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848430    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848434    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848439    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848443    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848447    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848452    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848456    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848460    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848464    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848468    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848474    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:17:37.853941 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.848483    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848675    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848683    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848688    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848692    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848697    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848701    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848705    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848710    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848714    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848718    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848722    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848727    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848731    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848735    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848740    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848744    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848748    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848752    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848757    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:17:37.854464 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848761    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848765    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848770    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848774    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848779    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848783    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848787    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848791    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848796    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848799    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848804    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848808    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848812    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848816    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848820    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848824    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848828    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848832    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848836    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848840    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:17:37.854949 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848844    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848849    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848856    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848863    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848868    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848873    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848878    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848883    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848887    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848892    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848896    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848900    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848905    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848909    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848913    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848917    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848921    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848926    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848931    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:17:37.855439 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848935    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848940    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848944    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848948    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848952    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848956    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848960    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848964    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848969    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848973    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848977    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848981    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848985    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848989    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.848995    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849000    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849005    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849009    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849014    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849018    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:17:37.855904 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849023    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849027    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849033    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849039    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849043    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849049    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849053    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849058    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849062    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849066    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849071    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849076    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849081    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849085    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849089    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849093    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849097    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849101    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849106    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:17:37.856411 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849110    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:17:37.856922 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:37.849114    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:17:37.856922 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.849123    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 19:17:37.856922 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.849279    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 19:17:37.856922 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.852649    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 19:17:37.856922 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.854173    2580 server.go:1019] "Starting client certificate rotation"
Jul 27 19:17:37.856922 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.854276    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 19:17:37.856922 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.854305    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 19:17:37.858120 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.858105    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 19:17:37.859835 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.859819    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 19:17:37.874313 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.874294    2580 log.go:25] "Validated CRI v1 runtime API"
Jul 27 19:17:37.876207 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.876194    2580 log.go:25] "Validated CRI v1 image API"
Jul 27 19:17:37.877094 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.877076    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 19:17:37.879767 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.879751    2580 fs.go:135] Filesystem UUIDs: map[385d5ebe-22fd-4fc1-9de7-ea247fc14095:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b1cc4650-7073-4a97-b1e8-00692aae84e4:/dev/nvme0n1p4]
Jul 27 19:17:37.879836 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.879767    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 19:17:37.885537 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.885520    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 19:17:37.886192 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.886091    2580 manager.go:216] Machine: {Timestamp:2026-07-27 19:17:37.88319302 +0000 UTC m=+0.398897514 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101074 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ff33e6e1230261be3478f9294ba34 SystemUUID:ec2ff33e-6e12-3026-1be3-478f9294ba34 BootID:4b09f849-1580-4496-83c6-abab9a843ac3 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9c:8f:72:a3:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9c:8f:72:a3:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:22:70:e3:ef:8c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 19:17:37.886192 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.886188    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 19:17:37.886305 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.886271    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 19:17:37.889614 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.889591    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 19:17:37.889748 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.889617    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-103.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 19:17:37.890433 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.890424    2580 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 19:17:37.890463 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.890435    2580 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 19:17:37.890463 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.890443    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 19:17:37.891289 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.891279    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 19:17:37.892223 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.892214    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 19:17:37.895045 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.895031    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 19:17:37.895158 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.895149    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 19:17:37.895201 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.895178    2580 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 19:17:37.895201 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.895187    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 19:17:37.895201 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.895200    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 19:17:37.895322 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.895209    2580 kubelet.go:398] "Adding apiserver pod source"
Jul 27 19:17:37.895322 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.895218    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 19:17:37.895591 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.895579    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 19:17:37.895627 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.895599    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 19:17:37.898455 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.898437    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 19:17:37.900454 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.900441    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 19:17:37.900506 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.900471    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 19:17:37.902057 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.902045    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 19:17:37.902110 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.902063    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 19:17:37.902110 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.902073    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 19:17:37.902110 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.902082    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 19:17:37.902110 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.902095    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 19:17:37.902110 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.902104    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 19:17:37.902251 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.902117    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 19:17:37.902251 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.902126    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 19:17:37.902251 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.902136    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 19:17:37.902251 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.902146    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 19:17:37.902251 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.902158    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 19:17:37.902251 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.902167    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 19:17:37.903076 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.903064    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 19:17:37.903107 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.903078    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 19:17:37.906062 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:37.906039    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 19:17:37.906062 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.906048    2580 server.go:1268] "Started kubelet"
Jul 27 19:17:37.906177 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:37.906082    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 19:17:37.906177 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.906081    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 19:17:37.906177 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.906129    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-103.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 19:17:37.906177 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.906137    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 19:17:37.906177 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.906146    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 19:17:37.906979 ip-10-0-129-103 systemd[1]: Started Kubernetes Kubelet.
Jul 27 19:17:37.908422 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.908405    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 19:17:37.908816 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.908804    2580 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 19:17:37.912589 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:37.912565    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 19:17:37.913086 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.913066    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t9q55"
Jul 27 19:17:37.918553 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.918538    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t9q55"
Jul 27 19:17:37.920609 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.920596    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 19:17:37.920713 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.920687    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 19:17:37.920825 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:37.920721    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Jul 27 19:17:37.921286 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.921267    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 19:17:37.921286 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.921279    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 19:17:37.921412 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.921296    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 19:17:37.921455 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.921413    2580 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 19:17:37.921455 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.921421    2580 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 19:17:37.921741 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.921563    2580 factory.go:55] Registering systemd factory
Jul 27 19:17:37.921741 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.921602    2580 factory.go:223] Registration of the systemd container factory successfully
Jul 27 19:17:37.922087 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.922074    2580 factory.go:153] Registering CRI-O factory
Jul 27 19:17:37.922187 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.922169    2580 factory.go:223] Registration of the crio container factory successfully
Jul 27 19:17:37.922308 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.922231    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 19:17:37.922308 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.922268    2580 factory.go:103] Registering Raw factory
Jul 27 19:17:37.922308 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.922284    2580 manager.go:1196] Started watching for new ooms in manager
Jul 27 19:17:37.922761 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.922747    2580 manager.go:349] Starting recovery of all containers
Jul 27 19:17:37.924831 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:37.924804    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 19:17:37.924970 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:37.924945    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 19:17:37.926137 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:37.925012    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-103.ec2.internal.18c63b9f5aabb820  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-103.ec2.internal,UID:ip-10-0-129-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-103.ec2.internal,},FirstTimestamp:2026-07-27 19:17:37.906022432 +0000 UTC m=+0.421726929,LastTimestamp:2026-07-27 19:17:37.906022432 +0000 UTC m=+0.421726929,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-103.ec2.internal,}"
Jul 27 19:17:37.931984 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.931969    2580 manager.go:354] Recovery completed
Jul 27 19:17:37.933564 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:37.933540    2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 27 19:17:37.936788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.936776    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:17:37.938702 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.938687    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:17:37.938758 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.938720    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:17:37.938758 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.938731    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:17:37.939232 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.939218    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 19:17:37.939302 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.939232    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 19:17:37.939302 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.939234    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:17:37.939302 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.939265    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 19:17:37.939880 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.939866    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:17:37.939927 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.939890    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:17:37.939927 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.939899    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:17:37.940948 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:37.940879    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-103.ec2.internal.18c63b9f5c9e5a3c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-103.ec2.internal,UID:ip-10-0-129-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-103.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-103.ec2.internal,},FirstTimestamp:2026-07-27 19:17:37.93870086 +0000 UTC m=+0.454405358,LastTimestamp:2026-07-27 19:17:37.93870086 +0000 UTC m=+0.454405358,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-103.ec2.internal,}"
Jul 27 19:17:37.941013 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.941002    2580 policy_none.go:49] "None policy: Start"
Jul 27 19:17:37.941052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.941020    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 19:17:37.941052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.941038    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 19:17:37.942191 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.942179    2580 policy_none.go:47] "Start"
Jul 27 19:17:37.983849 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.983723    2580 manager.go:337] "Starting Device Plugin manager"
Jul 27 19:17:37.983941 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:37.983872    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 19:17:37.983941 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.983885    2580 server.go:87] "Starting device plugin registration server"
Jul 27 19:17:37.984101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.984090    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 19:17:37.984132 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.984104    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 19:17:37.984289 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.984186    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 19:17:37.984289 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.984279    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 19:17:37.984289 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:37.984288    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 19:17:37.984765 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:37.984747    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 19:17:37.984821 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:37.984793    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-103.ec2.internal\" not found"
Jul 27 19:17:38.078375 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.078343    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 19:17:38.079594 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.079574    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 19:17:38.079651 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.079618    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 19:17:38.079651 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.079645    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 19:17:38.079714 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:38.079682    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 19:17:38.081929 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.081907    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:17:38.084652 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.084639    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:17:38.085998 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.085982    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:17:38.086063 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.086011    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:17:38.086063 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.086022    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:17:38.086063 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.086043    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.094443 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.094428    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.094509 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:38.094448    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-103.ec2.internal\": node \"ip-10-0-129-103.ec2.internal\" not found"
Jul 27 19:17:38.109452 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:38.109433    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Jul 27 19:17:38.180282 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.180255    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal"]
Jul 27 19:17:38.180361 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.180340    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:17:38.181306 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.181292    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:17:38.181363 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.181321    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:17:38.181363 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.181330    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:17:38.183286 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.183274    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:17:38.183382 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.183367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.183435 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.183401    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:17:38.183995 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.183979    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:17:38.184076 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.184010    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:17:38.184076 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.184023    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:17:38.184076 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.183980    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:17:38.184076 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.184076    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:17:38.184267 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.184086    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:17:38.185488 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.185470    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.185582 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.185501    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:17:38.186151 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.186137    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:17:38.186230 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.186164    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:17:38.186230 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.186177    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:17:38.208866 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:38.208848    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-103.ec2.internal\" not found" node="ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.209897 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:38.209873    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Jul 27 19:17:38.212187 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:38.212171    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-103.ec2.internal\" not found" node="ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.222923 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.222908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ed3ed7be1c2695b0c97f92390b9feef-config\") pod \"kube-apiserver-proxy-ip-10-0-129-103.ec2.internal\" (UID: \"1ed3ed7be1c2695b0c97f92390b9feef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.222983 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.222929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b5e70bad49fe9daa9a739742c8d5fb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal\" (UID: \"2b5e70bad49fe9daa9a739742c8d5fb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.222983 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.222954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b5e70bad49fe9daa9a739742c8d5fb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal\" (UID: \"2b5e70bad49fe9daa9a739742c8d5fb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.310713 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:38.310681    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Jul 27 19:17:38.324019 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.323992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ed3ed7be1c2695b0c97f92390b9feef-config\") pod \"kube-apiserver-proxy-ip-10-0-129-103.ec2.internal\" (UID: \"1ed3ed7be1c2695b0c97f92390b9feef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.324081 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.324063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b5e70bad49fe9daa9a739742c8d5fb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal\" (UID: \"2b5e70bad49fe9daa9a739742c8d5fb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.324114 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.324074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b5e70bad49fe9daa9a739742c8d5fb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal\" (UID: \"2b5e70bad49fe9daa9a739742c8d5fb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.324114 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.324101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ed3ed7be1c2695b0c97f92390b9feef-config\") pod \"kube-apiserver-proxy-ip-10-0-129-103.ec2.internal\" (UID: \"1ed3ed7be1c2695b0c97f92390b9feef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.324179 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.324121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b5e70bad49fe9daa9a739742c8d5fb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal\" (UID: \"2b5e70bad49fe9daa9a739742c8d5fb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.324179 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.324158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b5e70bad49fe9daa9a739742c8d5fb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal\" (UID: \"2b5e70bad49fe9daa9a739742c8d5fb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.411457 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:38.411395    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Jul 27 19:17:38.511912 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:38.511881    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Jul 27 19:17:38.511975 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.511918    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.517147 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.516615    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:38.612333 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:38.612302    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Jul 27 19:17:38.712832 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:38.712771    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Jul 27 19:17:38.813296 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:38.813263    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Jul 27 19:17:38.853715 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.853688    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 19:17:38.854141 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.853814    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 19:17:38.913513 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:38.913492    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Jul 27 19:17:38.932025 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.931995    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 19:12:37 +0000 UTC" deadline="2028-03-16 02:21:32.244185499 +0000 UTC"
Jul 27 19:17:38.932025 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:38.932022    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14335h3m53.312166914s"
Jul 27 19:17:39.001525 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:39.001497    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5e70bad49fe9daa9a739742c8d5fb2.slice/crio-acfaf26fdcc2a7ccc96a0c4f379bdc352320e11105cc3398e803c216d03f8acb WatchSource:0}: Error finding container acfaf26fdcc2a7ccc96a0c4f379bdc352320e11105cc3398e803c216d03f8acb: Status 404 returned error can't find the container with id acfaf26fdcc2a7ccc96a0c4f379bdc352320e11105cc3398e803c216d03f8acb
Jul 27 19:17:39.004292 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.004277    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:17:39.014327 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:39.014310    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-103.ec2.internal\" not found"
Jul 27 19:17:39.018653 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.018635    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:17:39.021286 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.021269    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:39.034284 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.034268    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 19:17:39.036204 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.036191    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal"
Jul 27 19:17:39.044533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.044516    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 19:17:39.082657 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.082622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal" event={"ID":"2b5e70bad49fe9daa9a739742c8d5fb2","Type":"ContainerStarted","Data":"acfaf26fdcc2a7ccc96a0c4f379bdc352320e11105cc3398e803c216d03f8acb"}
Jul 27 19:17:39.204320 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:39.204294    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed3ed7be1c2695b0c97f92390b9feef.slice/crio-5cff1a08d3ab20edc816572746b6ff1d9c86f2ec9e1340115310ae57c9b47ee6 WatchSource:0}: Error finding container 5cff1a08d3ab20edc816572746b6ff1d9c86f2ec9e1340115310ae57c9b47ee6: Status 404 returned error can't find the container with id 5cff1a08d3ab20edc816572746b6ff1d9c86f2ec9e1340115310ae57c9b47ee6
Jul 27 19:17:39.274570 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.274509    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:17:39.355493 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.355472    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:17:39.841955 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.841923    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:17:39.896829 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.896779    2580 apiserver.go:52] "Watching apiserver"
Jul 27 19:17:39.906587 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.906546    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 19:17:39.908290 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.908263    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal","openshift-multus/multus-additional-cni-plugins-krd7c","openshift-ovn-kubernetes/ovnkube-node-m552v","kube-system/konnectivity-agent-mwdjc","openshift-cluster-node-tuning-operator/tuned-hxrvx","openshift-image-registry/node-ca-8rsqx","openshift-multus/multus-d8sfk","openshift-multus/network-metrics-daemon-nr4fm","openshift-network-diagnostics/network-check-target-7vhjf","openshift-network-operator/iptables-alerter-8nw28","kube-system/global-pull-secret-syncer-2f68s","kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld","openshift-dns/node-resolver-w9hmm"]
Jul 27 19:17:39.910536 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.910516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.911928 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.911907    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:39.913377 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.913350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.914223 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.914034    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 19:17:39.914479 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.914460    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nzmm5\""
Jul 27 19:17:39.914562 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.914535    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 19:17:39.914844 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.914826    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 19:17:39.914956 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.914939    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 19:17:39.915113 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.915080    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 19:17:39.915179 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.915135    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 19:17:39.915811 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.915367    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mnbx4\""
Jul 27 19:17:39.915811 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.915568    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 19:17:39.915811 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.915589    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mwdjc"
Jul 27 19:17:39.915994 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.915918    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 19:17:39.915994 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.915954    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bg7k7\""
Jul 27 19:17:39.916782 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.916764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.917652 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.917436    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 19:17:39.917652 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.917479    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 19:17:39.917807 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.917695    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 19:17:39.917807 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.917742    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 19:17:39.918135 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.917927    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 19:17:39.918135 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.918131    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 19:17:39.918530 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.918406    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8rsqx"
Jul 27 19:17:39.918922 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.918808    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wjl2k\""
Jul 27 19:17:39.918922 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.918809    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 19:17:39.919075 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.919008    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k4z9d\""
Jul 27 19:17:39.919129 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.919109    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 19:17:39.919417 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.919308    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 19:17:39.920301 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.919988    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:39.920301 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:39.920054    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:17:39.921378 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.921259    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 19:17:39.921923 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.921681    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wz2l\""
Jul 27 19:17:39.922093 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.922074    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 19:17:39.922292 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.922275    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 19:17:39.922381 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.922342    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 19:17:39.922591 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.922574    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w9hmm"
Jul 27 19:17:39.923611 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.923591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:39.925100 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.924795    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8nw28"
Jul 27 19:17:39.926895 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.926236    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 19:17:39.926895 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.926818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:39.926895 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:39.926876    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:17:39.927851 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.927488    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 19:17:39.927851 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.927736    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bfdth\""
Jul 27 19:17:39.928334 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.928206    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:39.928334 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:39.928282    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:17:39.928896 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.928565    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dxmng\""
Jul 27 19:17:39.928896 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.928628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q9rgl\""
Jul 27 19:17:39.928896 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.928723    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 19:17:39.928896 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.928781    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 19:17:39.928896 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.928798    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 19:17:39.928896 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.928869    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 19:17:39.929217 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.928966    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 19:17:39.931571 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-hostroot\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.931571 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-registration-dir\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:39.931709 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-multus-socket-dir-parent\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.931709 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16340d27-1b23-4e10-be1e-7328952623c0-agent-certs\") pod \"konnectivity-agent-mwdjc\" (UID: \"16340d27-1b23-4e10-be1e-7328952623c0\") " pod="kube-system/konnectivity-agent-mwdjc"
Jul 27 19:17:39.931709 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6a22011-4f70-455b-9a51-964e395b2a2c-cnibin\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:39.931709 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-run\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.931709 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-var-lib-kubelet\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.931709 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-run-k8s-cni-cncf-io\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.931709 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931668    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbzwd\" (UniqueName: \"kubernetes.io/projected/94d97464-c11b-47a8-af6b-4c0f4d531794-kube-api-access-tbzwd\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.931709 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzdkb\" (UniqueName: \"kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb\") pod \"network-check-target-7vhjf\" (UID: \"58348a13-fcc8-4f55-9975-1b653ec5a0ad\") " pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:39.931709 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4n28\" (UniqueName: \"kubernetes.io/projected/268e36f2-3edd-4577-bd4d-7742d6859334-kube-api-access-v4n28\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-tuned\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-cni-netd\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/94d97464-c11b-47a8-af6b-4c0f4d531794-ovnkube-script-lib\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3-serviceca\") pod \"node-ca-8rsqx\" (UID: \"dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3\") " pod="openshift-image-registry/node-ca-8rsqx"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6a22011-4f70-455b-9a51-964e395b2a2c-cni-binary-copy\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-kubernetes\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-host\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-system-cni-dir\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6a22011-4f70-455b-9a51-964e395b2a2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931874    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-device-dir\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-etc-selinux\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/379c2805-26eb-4141-bd63-49ce6ad17fee-hosts-file\") pod \"node-resolver-w9hmm\" (UID: \"379c2805-26eb-4141-bd63-49ce6ad17fee\") " pod="openshift-dns/node-resolver-w9hmm"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-var-lib-kubelet\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-var-lib-openvswitch\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-etc-openvswitch\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.932112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-run-openvswitch\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931964    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-node-log\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-sysconfig\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.931989    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-sys\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjlz\" (UniqueName: \"kubernetes.io/projected/54edc824-8cb5-4d35-b193-c87fde5c11a9-kube-api-access-6xjlz\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-kubelet\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx87w\" (UniqueName: \"kubernetes.io/projected/d6a22011-4f70-455b-9a51-964e395b2a2c-kube-api-access-xx87w\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g46th\" (UniqueName: \"kubernetes.io/projected/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-kube-api-access-g46th\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-run-ovn\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-var-lib-cni-bin\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-modprobe-d\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-cnibin\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-run-ovn-kubernetes\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-socket-dir\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-tmp\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-cni-bin\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.932925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932180    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/94d97464-c11b-47a8-af6b-4c0f4d531794-ovnkube-config\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/94d97464-c11b-47a8-af6b-4c0f4d531794-env-overrides\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16340d27-1b23-4e10-be1e-7328952623c0-konnectivity-ca\") pod \"konnectivity-agent-mwdjc\" (UID: \"16340d27-1b23-4e10-be1e-7328952623c0\") " pod="kube-system/konnectivity-agent-mwdjc"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379c2805-26eb-4141-bd63-49ce6ad17fee-tmp-dir\") pod \"node-resolver-w9hmm\" (UID: \"379c2805-26eb-4141-bd63-49ce6ad17fee\") " pod="openshift-dns/node-resolver-w9hmm"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-var-lib-cni-multus\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-run-systemd\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6a22011-4f70-455b-9a51-964e395b2a2c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-systemd\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-os-release\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54edc824-8cb5-4d35-b193-c87fde5c11a9-cni-binary-copy\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-run-netns\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3-host\") pod \"node-ca-8rsqx\" (UID: \"dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3\") " pod="openshift-image-registry/node-ca-8rsqx"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b52k\" (UniqueName: \"kubernetes.io/projected/dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3-kube-api-access-4b52k\") pod \"node-ca-8rsqx\" (UID: \"dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3\") " pod="openshift-image-registry/node-ca-8rsqx"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6a22011-4f70-455b-9a51-964e395b2a2c-system-cni-dir\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d6a22011-4f70-455b-9a51-964e395b2a2c-os-release\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:39.933673 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-kubelet-dir\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-lib-modules\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932465    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-multus-cni-dir\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-systemd-units\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-log-socket\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/94d97464-c11b-47a8-af6b-4c0f4d531794-ovn-node-metrics-cert\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-sys-fs\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-sysctl-conf\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td8fb\" (UniqueName: \"kubernetes.io/projected/379c2805-26eb-4141-bd63-49ce6ad17fee-kube-api-access-td8fb\") pod \"node-resolver-w9hmm\" (UID: \"379c2805-26eb-4141-bd63-49ce6ad17fee\") " pod="openshift-dns/node-resolver-w9hmm"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-run-multus-certs\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-slash\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6a22011-4f70-455b-9a51-964e395b2a2c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.932997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-sysctl-d\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.933012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-multus-conf-dir\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.933026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54edc824-8cb5-4d35-b193-c87fde5c11a9-multus-daemon-config\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.933040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-etc-kubernetes\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:39.934392 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.933053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-run-netns\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:39.936383 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.936285    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 19:17:39.956508 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.956489    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-22grf"
Jul 27 19:17:39.966631 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:39.965610    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-22grf"
Jul 27 19:17:40.022811 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.022791    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 19:17:40.033547 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3-serviceca\") pod \"node-ca-8rsqx\" (UID: \"dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3\") " pod="openshift-image-registry/node-ca-8rsqx"
Jul 27 19:17:40.033658 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6a22011-4f70-455b-9a51-964e395b2a2c-cni-binary-copy\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.033658 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-kubernetes\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.033658 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-host\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.033658 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-system-cni-dir\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.033856 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6a22011-4f70-455b-9a51-964e395b2a2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.033856 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-device-dir\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.033856 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-etc-selinux\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.033856 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/379c2805-26eb-4141-bd63-49ce6ad17fee-hosts-file\") pod \"node-resolver-w9hmm\" (UID: \"379c2805-26eb-4141-bd63-49ce6ad17fee\") " pod="openshift-dns/node-resolver-w9hmm"
Jul 27 19:17:40.033856 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-var-lib-kubelet\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.033856 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-var-lib-openvswitch\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.033856 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-etc-openvswitch\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.033856 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-run-openvswitch\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-node-log\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-sysconfig\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-sys\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjlz\" (UniqueName: \"kubernetes.io/projected/54edc824-8cb5-4d35-b193-c87fde5c11a9-kube-api-access-6xjlz\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-kubelet\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.033977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx87w\" (UniqueName: \"kubernetes.io/projected/d6a22011-4f70-455b-9a51-964e395b2a2c-kube-api-access-xx87w\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h4cn\" (UniqueName: \"kubernetes.io/projected/bd4ebac6-0e11-4e10-a788-04aef97f902f-kube-api-access-2h4cn\") pod \"network-metrics-daemon-nr4fm\" (UID: \"bd4ebac6-0e11-4e10-a788-04aef97f902f\") " pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/267672f4-3a5e-4af6-b2b5-c2e3653f4fcc-iptables-alerter-script\") pod \"iptables-alerter-8nw28\" (UID: \"267672f4-3a5e-4af6-b2b5-c2e3653f4fcc\") " pod="openshift-network-operator/iptables-alerter-8nw28"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3-serviceca\") pod \"node-ca-8rsqx\" (UID: \"dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3\") " pod="openshift-image-registry/node-ca-8rsqx"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46th\" (UniqueName: \"kubernetes.io/projected/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-kube-api-access-g46th\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-run-ovn\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-var-lib-cni-bin\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6a22011-4f70-455b-9a51-964e395b2a2c-cni-binary-copy\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-run-openvswitch\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.034210 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/267672f4-3a5e-4af6-b2b5-c2e3653f4fcc-host-slash\") pod \"iptables-alerter-8nw28\" (UID: \"267672f4-3a5e-4af6-b2b5-c2e3653f4fcc\") " pod="openshift-network-operator/iptables-alerter-8nw28"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-node-log\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-modprobe-d\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-cnibin\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-kubernetes\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-var-lib-kubelet\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-etc-selinux\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-var-lib-openvswitch\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-etc-openvswitch\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-sysconfig\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-run-ovn-kubernetes\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-run-ovn-kubernetes\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-socket-dir\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-device-dir\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-host\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/379c2805-26eb-4141-bd63-49ce6ad17fee-hosts-file\") pod \"node-resolver-w9hmm\" (UID: \"379c2805-26eb-4141-bd63-49ce6ad17fee\") " pod="openshift-dns/node-resolver-w9hmm"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-cnibin\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6a22011-4f70-455b-9a51-964e395b2a2c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-run-ovn\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034539    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-modprobe-d\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-tmp\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-socket-dir\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-sys\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-cni-bin\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-system-cni-dir\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-var-lib-cni-bin\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-kubelet\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/94d97464-c11b-47a8-af6b-4c0f4d531794-ovnkube-config\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/94d97464-c11b-47a8-af6b-4c0f4d531794-env-overrides\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16340d27-1b23-4e10-be1e-7328952623c0-konnectivity-ca\") pod \"konnectivity-agent-mwdjc\" (UID: \"16340d27-1b23-4e10-be1e-7328952623c0\") " pod="kube-system/konnectivity-agent-mwdjc"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rgck\" (UniqueName: \"kubernetes.io/projected/267672f4-3a5e-4af6-b2b5-c2e3653f4fcc-kube-api-access-5rgck\") pod \"iptables-alerter-8nw28\" (UID: \"267672f4-3a5e-4af6-b2b5-c2e3653f4fcc\") " pod="openshift-network-operator/iptables-alerter-8nw28"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379c2805-26eb-4141-bd63-49ce6ad17fee-tmp-dir\") pod \"node-resolver-w9hmm\" (UID: \"379c2805-26eb-4141-bd63-49ce6ad17fee\") " pod="openshift-dns/node-resolver-w9hmm"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-var-lib-cni-multus\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034931    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 19:17:40.035788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-run-systemd\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.034995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6a22011-4f70-455b-9a51-964e395b2a2c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-systemd\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-os-release\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54edc824-8cb5-4d35-b193-c87fde5c11a9-cni-binary-copy\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-run-netns\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3-host\") pod \"node-ca-8rsqx\" (UID: \"dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3\") " pod="openshift-image-registry/node-ca-8rsqx"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b52k\" (UniqueName: \"kubernetes.io/projected/dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3-kube-api-access-4b52k\") pod \"node-ca-8rsqx\" (UID: \"dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3\") " pod="openshift-image-registry/node-ca-8rsqx"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6a22011-4f70-455b-9a51-964e395b2a2c-system-cni-dir\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d6a22011-4f70-455b-9a51-964e395b2a2c-os-release\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/94d97464-c11b-47a8-af6b-4c0f4d531794-ovnkube-config\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035278    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-kubelet-dir\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035302    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-lib-modules\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-cni-bin\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-multus-cni-dir\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/94d97464-c11b-47a8-af6b-4c0f4d531794-env-overrides\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16340d27-1b23-4e10-be1e-7328952623c0-konnectivity-ca\") pod \"konnectivity-agent-mwdjc\" (UID: \"16340d27-1b23-4e10-be1e-7328952623c0\") " pod="kube-system/konnectivity-agent-mwdjc"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-lib-modules\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.036546 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-multus-cni-dir\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-var-lib-cni-multus\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6a22011-4f70-455b-9a51-964e395b2a2c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6a22011-4f70-455b-9a51-964e395b2a2c-system-cni-dir\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-os-release\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-systemd-units\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-run-netns\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-log-socket\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3-host\") pod \"node-ca-8rsqx\" (UID: \"dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3\") " pod="openshift-image-registry/node-ca-8rsqx"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/94d97464-c11b-47a8-af6b-4c0f4d531794-ovn-node-metrics-cert\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-systemd-units\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/379c2805-26eb-4141-bd63-49ce6ad17fee-tmp-dir\") pod \"node-resolver-w9hmm\" (UID: \"379c2805-26eb-4141-bd63-49ce6ad17fee\") " pod="openshift-dns/node-resolver-w9hmm"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/19c1c8b7-cc69-4cad-aa77-67102d7353d8-dbus\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035856    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54edc824-8cb5-4d35-b193-c87fde5c11a9-cni-binary-copy\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-log-socket\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-systemd\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs\") pod \"network-metrics-daemon-nr4fm\" (UID: \"bd4ebac6-0e11-4e10-a788-04aef97f902f\") " pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-kubelet-dir\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.037403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-sys-fs\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d6a22011-4f70-455b-9a51-964e395b2a2c-os-release\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-run-systemd\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.035969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-sysctl-conf\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td8fb\" (UniqueName: \"kubernetes.io/projected/379c2805-26eb-4141-bd63-49ce6ad17fee-kube-api-access-td8fb\") pod \"node-resolver-w9hmm\" (UID: \"379c2805-26eb-4141-bd63-49ce6ad17fee\") " pod="openshift-dns/node-resolver-w9hmm"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-sys-fs\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-run-multus-certs\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-slash\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-sysctl-conf\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6a22011-4f70-455b-9a51-964e395b2a2c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-slash\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-run-multus-certs\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-sysctl-d\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-multus-conf-dir\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54edc824-8cb5-4d35-b193-c87fde5c11a9-multus-daemon-config\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-etc-kubernetes\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-run-netns\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-hostroot\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.038184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036425    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-sysctl-d\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-registration-dir\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-multus-socket-dir-parent\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-multus-conf-dir\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16340d27-1b23-4e10-be1e-7328952623c0-agent-certs\") pod \"konnectivity-agent-mwdjc\" (UID: \"16340d27-1b23-4e10-be1e-7328952623c0\") " pod="kube-system/konnectivity-agent-mwdjc"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036530    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-hostroot\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-run-netns\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6a22011-4f70-455b-9a51-964e395b2a2c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-multus-socket-dir-parent\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-etc-kubernetes\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/268e36f2-3edd-4577-bd4d-7742d6859334-registration-dir\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6a22011-4f70-455b-9a51-964e395b2a2c-cnibin\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/19c1c8b7-cc69-4cad-aa77-67102d7353d8-kubelet-config\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54edc824-8cb5-4d35-b193-c87fde5c11a9-multus-daemon-config\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.036981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-run\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.037003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-var-lib-kubelet\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.037015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6a22011-4f70-455b-9a51-964e395b2a2c-cnibin\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.039052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.037032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-run\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.039902 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.037027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-run-k8s-cni-cncf-io\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.039902 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.037070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbzwd\" (UniqueName: \"kubernetes.io/projected/94d97464-c11b-47a8-af6b-4c0f4d531794-kube-api-access-tbzwd\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.039902 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.037072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-var-lib-kubelet\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.039902 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.037073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54edc824-8cb5-4d35-b193-c87fde5c11a9-host-run-k8s-cni-cncf-io\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.039902 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.037094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdkb\" (UniqueName: \"kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb\") pod \"network-check-target-7vhjf\" (UID: \"58348a13-fcc8-4f55-9975-1b653ec5a0ad\") " pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:40.039902 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.037113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4n28\" (UniqueName: \"kubernetes.io/projected/268e36f2-3edd-4577-bd4d-7742d6859334-kube-api-access-v4n28\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.039902 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.037133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-tuned\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.039902 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.037153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-cni-netd\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.039902 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.037176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/94d97464-c11b-47a8-af6b-4c0f4d531794-ovnkube-script-lib\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.039902 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.037416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/94d97464-c11b-47a8-af6b-4c0f4d531794-host-cni-netd\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.039902 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.037649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/94d97464-c11b-47a8-af6b-4c0f4d531794-ovnkube-script-lib\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.041521 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.040740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/94d97464-c11b-47a8-af6b-4c0f4d531794-ovn-node-metrics-cert\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.041521 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.040744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-etc-tuned\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.041521 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.041071    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16340d27-1b23-4e10-be1e-7328952623c0-agent-certs\") pod \"konnectivity-agent-mwdjc\" (UID: \"16340d27-1b23-4e10-be1e-7328952623c0\") " pod="kube-system/konnectivity-agent-mwdjc"
Jul 27 19:17:40.044952 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.044814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-tmp\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.045067 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.044976    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:17:40.045067 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.044996    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:17:40.045067 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.045010    2580 projected.go:196] Error preparing data for projected volume kube-api-access-gzdkb for pod openshift-network-diagnostics/network-check-target-7vhjf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:17:40.045230 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.045069    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb podName:58348a13-fcc8-4f55-9975-1b653ec5a0ad nodeName:}" failed. No retries permitted until 2026-07-27 19:17:40.545047626 +0000 UTC m=+3.060752120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gzdkb" (UniqueName: "kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb") pod "network-check-target-7vhjf" (UID: "58348a13-fcc8-4f55-9975-1b653ec5a0ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:17:40.048326 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.048305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx87w\" (UniqueName: \"kubernetes.io/projected/d6a22011-4f70-455b-9a51-964e395b2a2c-kube-api-access-xx87w\") pod \"multus-additional-cni-plugins-krd7c\" (UID: \"d6a22011-4f70-455b-9a51-964e395b2a2c\") " pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.048524 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.048506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4n28\" (UniqueName: \"kubernetes.io/projected/268e36f2-3edd-4577-bd4d-7742d6859334-kube-api-access-v4n28\") pod \"aws-ebs-csi-driver-node-blnld\" (UID: \"268e36f2-3edd-4577-bd4d-7742d6859334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.049842 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.049819    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b52k\" (UniqueName: \"kubernetes.io/projected/dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3-kube-api-access-4b52k\") pod \"node-ca-8rsqx\" (UID: \"dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3\") " pod="openshift-image-registry/node-ca-8rsqx"
Jul 27 19:17:40.050325 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.050305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td8fb\" (UniqueName: \"kubernetes.io/projected/379c2805-26eb-4141-bd63-49ce6ad17fee-kube-api-access-td8fb\") pod \"node-resolver-w9hmm\" (UID: \"379c2805-26eb-4141-bd63-49ce6ad17fee\") " pod="openshift-dns/node-resolver-w9hmm"
Jul 27 19:17:40.050417 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.050358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g46th\" (UniqueName: \"kubernetes.io/projected/5bf0b05e-4725-4ca2-aa8e-9d5b093413ee-kube-api-access-g46th\") pod \"tuned-hxrvx\" (UID: \"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee\") " pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.050929 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.050906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjlz\" (UniqueName: \"kubernetes.io/projected/54edc824-8cb5-4d35-b193-c87fde5c11a9-kube-api-access-6xjlz\") pod \"multus-d8sfk\" (UID: \"54edc824-8cb5-4d35-b193-c87fde5c11a9\") " pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.052693 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.052672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbzwd\" (UniqueName: \"kubernetes.io/projected/94d97464-c11b-47a8-af6b-4c0f4d531794-kube-api-access-tbzwd\") pod \"ovnkube-node-m552v\" (UID: \"94d97464-c11b-47a8-af6b-4c0f4d531794\") " pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.084943 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.084909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal" event={"ID":"1ed3ed7be1c2695b0c97f92390b9feef","Type":"ContainerStarted","Data":"5cff1a08d3ab20edc816572746b6ff1d9c86f2ec9e1340115310ae57c9b47ee6"}
Jul 27 19:17:40.138196 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.137925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rgck\" (UniqueName: \"kubernetes.io/projected/267672f4-3a5e-4af6-b2b5-c2e3653f4fcc-kube-api-access-5rgck\") pod \"iptables-alerter-8nw28\" (UID: \"267672f4-3a5e-4af6-b2b5-c2e3653f4fcc\") " pod="openshift-network-operator/iptables-alerter-8nw28"
Jul 27 19:17:40.138196 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.137979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/19c1c8b7-cc69-4cad-aa77-67102d7353d8-dbus\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:40.138196 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.138004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs\") pod \"network-metrics-daemon-nr4fm\" (UID: \"bd4ebac6-0e11-4e10-a788-04aef97f902f\") " pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:40.138196 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.138033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/19c1c8b7-cc69-4cad-aa77-67102d7353d8-kubelet-config\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:40.138196 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.138080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:40.138196 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.138105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4cn\" (UniqueName: \"kubernetes.io/projected/bd4ebac6-0e11-4e10-a788-04aef97f902f-kube-api-access-2h4cn\") pod \"network-metrics-daemon-nr4fm\" (UID: \"bd4ebac6-0e11-4e10-a788-04aef97f902f\") " pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:40.138196 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.138129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/267672f4-3a5e-4af6-b2b5-c2e3653f4fcc-iptables-alerter-script\") pod \"iptables-alerter-8nw28\" (UID: \"267672f4-3a5e-4af6-b2b5-c2e3653f4fcc\") " pod="openshift-network-operator/iptables-alerter-8nw28"
Jul 27 19:17:40.138196 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.138075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/19c1c8b7-cc69-4cad-aa77-67102d7353d8-dbus\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:40.138196 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.138154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/267672f4-3a5e-4af6-b2b5-c2e3653f4fcc-host-slash\") pod \"iptables-alerter-8nw28\" (UID: \"267672f4-3a5e-4af6-b2b5-c2e3653f4fcc\") " pod="openshift-network-operator/iptables-alerter-8nw28"
Jul 27 19:17:40.138196 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.138160    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:17:40.138196 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.138200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/267672f4-3a5e-4af6-b2b5-c2e3653f4fcc-host-slash\") pod \"iptables-alerter-8nw28\" (UID: \"267672f4-3a5e-4af6-b2b5-c2e3653f4fcc\") " pod="openshift-network-operator/iptables-alerter-8nw28"
Jul 27 19:17:40.138832 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.138254    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs podName:bd4ebac6-0e11-4e10-a788-04aef97f902f nodeName:}" failed. No retries permitted until 2026-07-27 19:17:40.638224634 +0000 UTC m=+3.153929142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs") pod "network-metrics-daemon-nr4fm" (UID: "bd4ebac6-0e11-4e10-a788-04aef97f902f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:17:40.138832 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.138275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/19c1c8b7-cc69-4cad-aa77-67102d7353d8-kubelet-config\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:40.138832 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.138295    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:17:40.138832 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.138349    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret podName:19c1c8b7-cc69-4cad-aa77-67102d7353d8 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:40.638334411 +0000 UTC m=+3.154038904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret") pod "global-pull-secret-syncer-2f68s" (UID: "19c1c8b7-cc69-4cad-aa77-67102d7353d8") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:17:40.138832 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.138786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/267672f4-3a5e-4af6-b2b5-c2e3653f4fcc-iptables-alerter-script\") pod \"iptables-alerter-8nw28\" (UID: \"267672f4-3a5e-4af6-b2b5-c2e3653f4fcc\") " pod="openshift-network-operator/iptables-alerter-8nw28"
Jul 27 19:17:40.147873 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.147853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rgck\" (UniqueName: \"kubernetes.io/projected/267672f4-3a5e-4af6-b2b5-c2e3653f4fcc-kube-api-access-5rgck\") pod \"iptables-alerter-8nw28\" (UID: \"267672f4-3a5e-4af6-b2b5-c2e3653f4fcc\") " pod="openshift-network-operator/iptables-alerter-8nw28"
Jul 27 19:17:40.148024 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.148002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4cn\" (UniqueName: \"kubernetes.io/projected/bd4ebac6-0e11-4e10-a788-04aef97f902f-kube-api-access-2h4cn\") pod \"network-metrics-daemon-nr4fm\" (UID: \"bd4ebac6-0e11-4e10-a788-04aef97f902f\") " pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:40.225611 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.225579    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d8sfk"
Jul 27 19:17:40.234226 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.234206    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld"
Jul 27 19:17:40.246292 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.246269    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:17:40.251594 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.251574    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mwdjc"
Jul 27 19:17:40.260672 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.260645    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hxrvx"
Jul 27 19:17:40.269540 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.269521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8rsqx"
Jul 27 19:17:40.275385 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.275366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w9hmm"
Jul 27 19:17:40.284092 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.284075    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-krd7c"
Jul 27 19:17:40.289679 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.289652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8nw28"
Jul 27 19:17:40.641139 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.641105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:40.641331 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.641170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs\") pod \"network-metrics-daemon-nr4fm\" (UID: \"bd4ebac6-0e11-4e10-a788-04aef97f902f\") " pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:40.641331 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.641208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdkb\" (UniqueName: \"kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb\") pod \"network-check-target-7vhjf\" (UID: \"58348a13-fcc8-4f55-9975-1b653ec5a0ad\") " pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:40.641331 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.641281    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:17:40.641331 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.641323    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:17:40.641506 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.641337    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:17:40.641506 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.641352    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:17:40.641506 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.641364    2580 projected.go:196] Error preparing data for projected volume kube-api-access-gzdkb for pod openshift-network-diagnostics/network-check-target-7vhjf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:17:40.641506 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.641352    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret podName:19c1c8b7-cc69-4cad-aa77-67102d7353d8 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:41.641330637 +0000 UTC m=+4.157035122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret") pod "global-pull-secret-syncer-2f68s" (UID: "19c1c8b7-cc69-4cad-aa77-67102d7353d8") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:17:40.641506 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.641427    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs podName:bd4ebac6-0e11-4e10-a788-04aef97f902f nodeName:}" failed. No retries permitted until 2026-07-27 19:17:41.641404019 +0000 UTC m=+4.157108517 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs") pod "network-metrics-daemon-nr4fm" (UID: "bd4ebac6-0e11-4e10-a788-04aef97f902f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:17:40.641506 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:40.641450    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb podName:58348a13-fcc8-4f55-9975-1b653ec5a0ad nodeName:}" failed. No retries permitted until 2026-07-27 19:17:41.641439619 +0000 UTC m=+4.157144105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzdkb" (UniqueName: "kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb") pod "network-check-target-7vhjf" (UID: "58348a13-fcc8-4f55-9975-1b653ec5a0ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:17:40.949687 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:40.949657    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16340d27_1b23_4e10_be1e_7328952623c0.slice/crio-c1892c5e2249475d468729dbe2f1c367c7125058ce7f1be74cc88befb193c0e9 WatchSource:0}: Error finding container c1892c5e2249475d468729dbe2f1c367c7125058ce7f1be74cc88befb193c0e9: Status 404 returned error can't find the container with id c1892c5e2249475d468729dbe2f1c367c7125058ce7f1be74cc88befb193c0e9
Jul 27 19:17:40.957395 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:40.957270    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a22011_4f70_455b_9a51_964e395b2a2c.slice/crio-731507eab90879c80f835b8b190736bda54b042abf5a38b4538c5291b7787ab2 WatchSource:0}: Error finding container 731507eab90879c80f835b8b190736bda54b042abf5a38b4538c5291b7787ab2: Status 404 returned error can't find the container with id 731507eab90879c80f835b8b190736bda54b042abf5a38b4538c5291b7787ab2
Jul 27 19:17:40.957989 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:40.957971    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267672f4_3a5e_4af6_b2b5_c2e3653f4fcc.slice/crio-4d1f21029622deab00844de8513614d62ff4b680b7a5f939191c6d5a12d111a7 WatchSource:0}: Error finding container 4d1f21029622deab00844de8513614d62ff4b680b7a5f939191c6d5a12d111a7: Status 404 returned error can't find the container with id 4d1f21029622deab00844de8513614d62ff4b680b7a5f939191c6d5a12d111a7
Jul 27 19:17:40.958970 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:40.958946    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d97464_c11b_47a8_af6b_4c0f4d531794.slice/crio-e441fef5e91bbe8d2606b4cf1a21fb6854033942a5bd57d2a264db47755c2f59 WatchSource:0}: Error finding container e441fef5e91bbe8d2606b4cf1a21fb6854033942a5bd57d2a264db47755c2f59: Status 404 returned error can't find the container with id e441fef5e91bbe8d2606b4cf1a21fb6854033942a5bd57d2a264db47755c2f59
Jul 27 19:17:40.960876 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:40.960854    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268e36f2_3edd_4577_bd4d_7742d6859334.slice/crio-a83c617aa46594adcafdc29169e02db6b8aafa0055880f0a19bddfb9e636ebec WatchSource:0}: Error finding container a83c617aa46594adcafdc29169e02db6b8aafa0055880f0a19bddfb9e636ebec: Status 404 returned error can't find the container with id a83c617aa46594adcafdc29169e02db6b8aafa0055880f0a19bddfb9e636ebec
Jul 27 19:17:40.968467 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.968443    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 19:12:39 +0000 UTC" deadline="2028-04-24 04:23:14.455827888 +0000 UTC"
Jul 27 19:17:40.968467 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:40.968466    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15273h5m33.487365273s"
Jul 27 19:17:40.975538 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:40.975518    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf0b05e_4725_4ca2_aa8e_9d5b093413ee.slice/crio-ce7e75a5ed641f2584ffa859366dcd8cd9629adaddee06b6c3f271d849a6607d WatchSource:0}: Error finding container ce7e75a5ed641f2584ffa859366dcd8cd9629adaddee06b6c3f271d849a6607d: Status 404 returned error can't find the container with id ce7e75a5ed641f2584ffa859366dcd8cd9629adaddee06b6c3f271d849a6607d
Jul 27 19:17:40.976332 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:40.976294    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafcc8b6_00fd_4c7b_adb0_1cd6ba1560f3.slice/crio-4b27e7a62175784847edde32f5e2bc1ac56b53002f328535d97ba7418eedf9aa WatchSource:0}: Error finding container 4b27e7a62175784847edde32f5e2bc1ac56b53002f328535d97ba7418eedf9aa: Status 404 returned error can't find the container with id 4b27e7a62175784847edde32f5e2bc1ac56b53002f328535d97ba7418eedf9aa
Jul 27 19:17:40.977055 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:40.977035    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54edc824_8cb5_4d35_b193_c87fde5c11a9.slice/crio-afe0b6a1ca2f62fc4d3ab1d7094cef702e1e33303d430eb5fc1e51555f44ed86 WatchSource:0}: Error finding container afe0b6a1ca2f62fc4d3ab1d7094cef702e1e33303d430eb5fc1e51555f44ed86: Status 404 returned error can't find the container with id afe0b6a1ca2f62fc4d3ab1d7094cef702e1e33303d430eb5fc1e51555f44ed86
Jul 27 19:17:40.978498 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:17:40.978470    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379c2805_26eb_4141_bd63_49ce6ad17fee.slice/crio-cfe61933c36f4a6d6df0a5f7db240e1b8811905f866ff75172c3d8cf2c9b563c WatchSource:0}: Error finding container cfe61933c36f4a6d6df0a5f7db240e1b8811905f866ff75172c3d8cf2c9b563c: Status 404 returned error can't find the container with id cfe61933c36f4a6d6df0a5f7db240e1b8811905f866ff75172c3d8cf2c9b563c
Jul 27 19:17:41.080916 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.080792    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:41.081014 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:41.080995    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:17:41.088767 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.088742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal" event={"ID":"1ed3ed7be1c2695b0c97f92390b9feef","Type":"ContainerStarted","Data":"0da47ba2a0aa4a5c9edf642eb6cb7c3290fbf1fd231ff195c139c50b00f493ae"}
Jul 27 19:17:41.089943 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.089921    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8nw28" event={"ID":"267672f4-3a5e-4af6-b2b5-c2e3653f4fcc","Type":"ContainerStarted","Data":"4d1f21029622deab00844de8513614d62ff4b680b7a5f939191c6d5a12d111a7"}
Jul 27 19:17:41.090993 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.090971    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8rsqx" event={"ID":"dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3","Type":"ContainerStarted","Data":"4b27e7a62175784847edde32f5e2bc1ac56b53002f328535d97ba7418eedf9aa"}
Jul 27 19:17:41.091971 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.091937    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld" event={"ID":"268e36f2-3edd-4577-bd4d-7742d6859334","Type":"ContainerStarted","Data":"a83c617aa46594adcafdc29169e02db6b8aafa0055880f0a19bddfb9e636ebec"}
Jul 27 19:17:41.093144 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.093126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m552v" event={"ID":"94d97464-c11b-47a8-af6b-4c0f4d531794","Type":"ContainerStarted","Data":"e441fef5e91bbe8d2606b4cf1a21fb6854033942a5bd57d2a264db47755c2f59"}
Jul 27 19:17:41.094216 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.094198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krd7c" event={"ID":"d6a22011-4f70-455b-9a51-964e395b2a2c","Type":"ContainerStarted","Data":"731507eab90879c80f835b8b190736bda54b042abf5a38b4538c5291b7787ab2"}
Jul 27 19:17:41.095317 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.095294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d8sfk" event={"ID":"54edc824-8cb5-4d35-b193-c87fde5c11a9","Type":"ContainerStarted","Data":"afe0b6a1ca2f62fc4d3ab1d7094cef702e1e33303d430eb5fc1e51555f44ed86"}
Jul 27 19:17:41.096285 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.096263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w9hmm" event={"ID":"379c2805-26eb-4141-bd63-49ce6ad17fee","Type":"ContainerStarted","Data":"cfe61933c36f4a6d6df0a5f7db240e1b8811905f866ff75172c3d8cf2c9b563c"}
Jul 27 19:17:41.097267 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.097235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hxrvx" event={"ID":"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee","Type":"ContainerStarted","Data":"ce7e75a5ed641f2584ffa859366dcd8cd9629adaddee06b6c3f271d849a6607d"}
Jul 27 19:17:41.098315 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.098297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mwdjc" event={"ID":"16340d27-1b23-4e10-be1e-7328952623c0","Type":"ContainerStarted","Data":"c1892c5e2249475d468729dbe2f1c367c7125058ce7f1be74cc88befb193c0e9"}
Jul 27 19:17:41.648868 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.648808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdkb\" (UniqueName: \"kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb\") pod \"network-check-target-7vhjf\" (UID: \"58348a13-fcc8-4f55-9975-1b653ec5a0ad\") " pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:41.648868 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.648875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:41.649109 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:41.649081    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:17:41.649109 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:41.649099    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:17:41.649109 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:41.649113    2580 projected.go:196] Error preparing data for projected volume kube-api-access-gzdkb for pod openshift-network-diagnostics/network-check-target-7vhjf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:17:41.649289 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:41.649176    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb podName:58348a13-fcc8-4f55-9975-1b653ec5a0ad nodeName:}" failed. No retries permitted until 2026-07-27 19:17:43.649158459 +0000 UTC m=+6.164862943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzdkb" (UniqueName: "kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb") pod "network-check-target-7vhjf" (UID: "58348a13-fcc8-4f55-9975-1b653ec5a0ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:17:41.649708 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.649684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs\") pod \"network-metrics-daemon-nr4fm\" (UID: \"bd4ebac6-0e11-4e10-a788-04aef97f902f\") " pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:41.649830 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:41.649815    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:17:41.649890 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:41.649867    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs podName:bd4ebac6-0e11-4e10-a788-04aef97f902f nodeName:}" failed. No retries permitted until 2026-07-27 19:17:43.64985277 +0000 UTC m=+6.165557256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs") pod "network-metrics-daemon-nr4fm" (UID: "bd4ebac6-0e11-4e10-a788-04aef97f902f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:17:41.649946 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:41.649894    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:17:41.649946 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:41.649925    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret podName:19c1c8b7-cc69-4cad-aa77-67102d7353d8 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:43.649916096 +0000 UTC m=+6.165620593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret") pod "global-pull-secret-syncer-2f68s" (UID: "19c1c8b7-cc69-4cad-aa77-67102d7353d8") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:17:41.969371 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.969282    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 19:12:39 +0000 UTC" deadline="2028-03-06 11:17:31.032555876 +0000 UTC"
Jul 27 19:17:41.969371 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:41.969325    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14103h59m49.063235385s"
Jul 27 19:17:42.091836 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:42.091784    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:42.092032 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:42.091938    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:17:42.092541 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:42.092519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:42.092688 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:42.092652    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:17:42.117924 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:42.117888    2580 generic.go:356] "Generic (PLEG): container finished" podID="2b5e70bad49fe9daa9a739742c8d5fb2" containerID="d5ede03a11259011563136568d63ebe1e7662e6e25b7ff2427c066a64dfe4fbd" exitCode=0
Jul 27 19:17:42.118930 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:42.118902    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal" event={"ID":"2b5e70bad49fe9daa9a739742c8d5fb2","Type":"ContainerDied","Data":"d5ede03a11259011563136568d63ebe1e7662e6e25b7ff2427c066a64dfe4fbd"}
Jul 27 19:17:42.136087 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:42.136025    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-103.ec2.internal" podStartSLOduration=3.13600643 podStartE2EDuration="3.13600643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:17:41.105309442 +0000 UTC m=+3.621013943" watchObservedRunningTime="2026-07-27 19:17:42.13600643 +0000 UTC m=+4.651710934"
Jul 27 19:17:43.080235 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:43.080201    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:43.080722 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:43.080359    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:17:43.128618 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:43.128580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal" event={"ID":"2b5e70bad49fe9daa9a739742c8d5fb2","Type":"ContainerStarted","Data":"e0b19795ca97f0b809645e113db136f818489c5ef8b1a3e13b98690f81e4d500"}
Jul 27 19:17:43.667837 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:43.667800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdkb\" (UniqueName: \"kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb\") pod \"network-check-target-7vhjf\" (UID: \"58348a13-fcc8-4f55-9975-1b653ec5a0ad\") " pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:43.668012 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:43.667862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:43.668012 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:43.667909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs\") pod \"network-metrics-daemon-nr4fm\" (UID: \"bd4ebac6-0e11-4e10-a788-04aef97f902f\") " pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:43.668167 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:43.668150    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:17:43.668219 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:43.668179    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:17:43.668219 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:43.668194    2580 projected.go:196] Error preparing data for projected volume kube-api-access-gzdkb for pod openshift-network-diagnostics/network-check-target-7vhjf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:17:43.668352 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:43.668265    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb podName:58348a13-fcc8-4f55-9975-1b653ec5a0ad nodeName:}" failed. No retries permitted until 2026-07-27 19:17:47.668232674 +0000 UTC m=+10.183937160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzdkb" (UniqueName: "kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb") pod "network-check-target-7vhjf" (UID: "58348a13-fcc8-4f55-9975-1b653ec5a0ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:17:43.668895 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:43.668632    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:17:43.668895 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:43.668692    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret podName:19c1c8b7-cc69-4cad-aa77-67102d7353d8 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:47.668676888 +0000 UTC m=+10.184381387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret") pod "global-pull-secret-syncer-2f68s" (UID: "19c1c8b7-cc69-4cad-aa77-67102d7353d8") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:17:43.668895 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:43.668767    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:17:43.668895 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:43.668799    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs podName:bd4ebac6-0e11-4e10-a788-04aef97f902f nodeName:}" failed. No retries permitted until 2026-07-27 19:17:47.668788158 +0000 UTC m=+10.184492655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs") pod "network-metrics-daemon-nr4fm" (UID: "bd4ebac6-0e11-4e10-a788-04aef97f902f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:17:44.080907 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:44.080335    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:44.080907 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:44.080464    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:17:44.080907 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:44.080785    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:44.080907 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:44.080870    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:17:45.080192 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:45.080158    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:45.080398 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:45.080298    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:17:46.080762 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:46.080685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:46.080762 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:46.080720    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:46.081270 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:46.080821    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:17:46.081270 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:46.080955    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:17:47.080463 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:47.080429    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:47.080632 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:47.080611    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:17:47.701443 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:47.701404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs\") pod \"network-metrics-daemon-nr4fm\" (UID: \"bd4ebac6-0e11-4e10-a788-04aef97f902f\") " pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:47.701938 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:47.701468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdkb\" (UniqueName: \"kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb\") pod \"network-check-target-7vhjf\" (UID: \"58348a13-fcc8-4f55-9975-1b653ec5a0ad\") " pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:47.701938 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:47.701498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:47.701938 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:47.701587    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:17:47.701938 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:47.701626    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:17:47.701938 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:47.701664    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs podName:bd4ebac6-0e11-4e10-a788-04aef97f902f nodeName:}" failed. No retries permitted until 2026-07-27 19:17:55.701642144 +0000 UTC m=+18.217346630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs") pod "network-metrics-daemon-nr4fm" (UID: "bd4ebac6-0e11-4e10-a788-04aef97f902f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:17:47.701938 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:47.701686    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret podName:19c1c8b7-cc69-4cad-aa77-67102d7353d8 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:55.701675582 +0000 UTC m=+18.217380071 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret") pod "global-pull-secret-syncer-2f68s" (UID: "19c1c8b7-cc69-4cad-aa77-67102d7353d8") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:17:47.701938 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:47.701703    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:17:47.701938 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:47.701713    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:17:47.701938 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:47.701721    2580 projected.go:196] Error preparing data for projected volume kube-api-access-gzdkb for pod openshift-network-diagnostics/network-check-target-7vhjf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:17:47.701938 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:47.701747    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb podName:58348a13-fcc8-4f55-9975-1b653ec5a0ad nodeName:}" failed. No retries permitted until 2026-07-27 19:17:55.701737043 +0000 UTC m=+18.217441524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzdkb" (UniqueName: "kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb") pod "network-check-target-7vhjf" (UID: "58348a13-fcc8-4f55-9975-1b653ec5a0ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:17:48.082602 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:48.081368    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:48.082602 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:48.081474    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:17:48.082602 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:48.081536    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:48.082602 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:48.081602    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:17:49.080530 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:49.080324    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:49.080530 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:49.080467    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:17:50.080800 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:50.080769    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:50.081231 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:50.080770    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:50.081231 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:50.080901    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:17:50.081231 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:50.080994    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:17:51.080628 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:51.080581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:51.080833 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:51.080722    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:17:52.080580 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:52.080547    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:52.080763 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:52.080688    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:17:52.080763 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:52.080718    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:52.081143 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:52.080822    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:17:53.080234 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:53.080200    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:53.080411 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:53.080358    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:17:54.080514 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:54.080485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:54.080985 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:54.080629    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:17:54.080985 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:54.080680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:54.080985 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:54.080780    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:17:55.079842 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:55.079814    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:55.080021 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:55.079925    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:17:55.763019 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:55.762985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs\") pod \"network-metrics-daemon-nr4fm\" (UID: \"bd4ebac6-0e11-4e10-a788-04aef97f902f\") " pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:55.763486 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:55.763037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdkb\" (UniqueName: \"kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb\") pod \"network-check-target-7vhjf\" (UID: \"58348a13-fcc8-4f55-9975-1b653ec5a0ad\") " pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:55.763486 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:55.763150    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:17:55.763486 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:55.763181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:55.763486 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:55.763223    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs podName:bd4ebac6-0e11-4e10-a788-04aef97f902f nodeName:}" failed. No retries permitted until 2026-07-27 19:18:11.763202923 +0000 UTC m=+34.278907409 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs") pod "network-metrics-daemon-nr4fm" (UID: "bd4ebac6-0e11-4e10-a788-04aef97f902f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:17:55.763486 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:55.763152    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:17:55.763486 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:55.763288    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:17:55.763486 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:55.763291    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:17:55.763486 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:55.763302    2580 projected.go:196] Error preparing data for projected volume kube-api-access-gzdkb for pod openshift-network-diagnostics/network-check-target-7vhjf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:17:55.763486 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:55.763353    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret podName:19c1c8b7-cc69-4cad-aa77-67102d7353d8 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:11.763335968 +0000 UTC m=+34.279040452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret") pod "global-pull-secret-syncer-2f68s" (UID: "19c1c8b7-cc69-4cad-aa77-67102d7353d8") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:17:55.763486 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:55.763374    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb podName:58348a13-fcc8-4f55-9975-1b653ec5a0ad nodeName:}" failed. No retries permitted until 2026-07-27 19:18:11.763365145 +0000 UTC m=+34.279069626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzdkb" (UniqueName: "kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb") pod "network-check-target-7vhjf" (UID: "58348a13-fcc8-4f55-9975-1b653ec5a0ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:17:56.080732 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:56.080700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:56.080890 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:56.080830    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:17:56.080890 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:56.080869    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:56.080996 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:56.080942    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:17:57.080728 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:57.080692    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:57.081189 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:57.080831    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:17:58.081038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:58.080996    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:17:58.081517 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:58.081162    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:17:58.081517 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:58.081218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:17:58.081517 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:58.081326    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:17:59.080713 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:17:59.080680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:17:59.080855 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:17:59.080788    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:18:00.080872 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.080556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:18:00.081560 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.080640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:18:00.081560 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:00.080942    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:18:00.081560 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:00.081008    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:18:00.166902 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.166875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d8sfk" event={"ID":"54edc824-8cb5-4d35-b193-c87fde5c11a9","Type":"ContainerStarted","Data":"7392fba256bcc0949472ee560e38d5de26e61a50a53f5afc367b08d05eff96d9"}
Jul 27 19:18:00.168234 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.168206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w9hmm" event={"ID":"379c2805-26eb-4141-bd63-49ce6ad17fee","Type":"ContainerStarted","Data":"cfed49b226d0fc655892700725119afd05fad557864d308cf280a8726a5d38f7"}
Jul 27 19:18:00.169728 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.169594    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hxrvx" event={"ID":"5bf0b05e-4725-4ca2-aa8e-9d5b093413ee","Type":"ContainerStarted","Data":"439a79e412b6bb367e5d9d64278dd6fc29f1295042e0eb74e53646627e7112d8"}
Jul 27 19:18:00.171256 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.171211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mwdjc" event={"ID":"16340d27-1b23-4e10-be1e-7328952623c0","Type":"ContainerStarted","Data":"246877eb29bb8092855352bd301757f18636627f44c8f78b9f9910a6f2794c69"}
Jul 27 19:18:00.172629 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.172609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8rsqx" event={"ID":"dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3","Type":"ContainerStarted","Data":"b2e84f2c662078af8980cd78eecd9820840b06e32a0a69d0a25fb953f2ac2bab"}
Jul 27 19:18:00.174084 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.174064    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld" event={"ID":"268e36f2-3edd-4577-bd4d-7742d6859334","Type":"ContainerStarted","Data":"e7bef4df2d6de7523d4cbfd23076c577cdec0e04917c7818b399d49ac5963704"}
Jul 27 19:18:00.176330 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.176312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m552v" event={"ID":"94d97464-c11b-47a8-af6b-4c0f4d531794","Type":"ContainerStarted","Data":"59141fcf7531989e55ff5919265e6c425881d550fe040649921d080d30828333"}
Jul 27 19:18:00.176415 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.176367    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m552v" event={"ID":"94d97464-c11b-47a8-af6b-4c0f4d531794","Type":"ContainerStarted","Data":"f721129cf4df9bf98cfa2454841cf73b8e91fc677237718b7bfdcdc7bf3822b2"}
Jul 27 19:18:00.176415 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.176399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m552v" event={"ID":"94d97464-c11b-47a8-af6b-4c0f4d531794","Type":"ContainerStarted","Data":"5bfa6afab38eea21b622432aa13003617ca42f92f3b8d7056299cb04443c9b2f"}
Jul 27 19:18:00.177740 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.177722    2580 generic.go:356] "Generic (PLEG): container finished" podID="d6a22011-4f70-455b-9a51-964e395b2a2c" containerID="3139be857aa84e87b816ff91dc4334270adb0005f0ddd02293f8924059628a60" exitCode=0
Jul 27 19:18:00.177812 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.177747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krd7c" event={"ID":"d6a22011-4f70-455b-9a51-964e395b2a2c","Type":"ContainerDied","Data":"3139be857aa84e87b816ff91dc4334270adb0005f0ddd02293f8924059628a60"}
Jul 27 19:18:00.198639 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.198579    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-103.ec2.internal" podStartSLOduration=21.198567373 podStartE2EDuration="21.198567373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:17:43.146962057 +0000 UTC m=+5.662666562" watchObservedRunningTime="2026-07-27 19:18:00.198567373 +0000 UTC m=+22.714271878"
Jul 27 19:18:00.218579 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.218533    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mwdjc" podStartSLOduration=8.067224591 podStartE2EDuration="22.218514837s" totalImagesPullingTime="14.151290246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:00.218152669 +0000 UTC m=+22.733857176" watchObservedRunningTime="2026-07-27 19:18:00.218514837 +0000 UTC m=+22.734219342"
Jul 27 19:18:00.218790 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.218762    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-d8sfk" podStartSLOduration=3.728536677 podStartE2EDuration="22.218756186s" totalImagesPullingTime="18.490219509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:00.198390444 +0000 UTC m=+22.714094948" watchObservedRunningTime="2026-07-27 19:18:00.218756186 +0000 UTC m=+22.734460691"
Jul 27 19:18:00.241520 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.241468    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8rsqx" podStartSLOduration=3.750881328 podStartE2EDuration="22.241449923s" totalImagesPullingTime="18.490568595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:00.24128012 +0000 UTC m=+22.756984624" watchObservedRunningTime="2026-07-27 19:18:00.241449923 +0000 UTC m=+22.757154428"
Jul 27 19:18:00.254996 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.254966    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mwdjc"
Jul 27 19:18:00.281111 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.281072    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hxrvx" podStartSLOduration=3.788716315 podStartE2EDuration="22.281055161s" totalImagesPullingTime="18.492338846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:00.28096966 +0000 UTC m=+22.796674168" watchObservedRunningTime="2026-07-27 19:18:00.281055161 +0000 UTC m=+22.796759666"
Jul 27 19:18:00.300209 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.300158    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w9hmm" podStartSLOduration=3.843358286 podStartE2EDuration="22.300134983s" totalImagesPullingTime="18.456776697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:00.296474673 +0000 UTC m=+22.812179182" watchObservedRunningTime="2026-07-27 19:18:00.300134983 +0000 UTC m=+22.815839493"
Jul 27 19:18:00.364757 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.364719    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mwdjc_16340d27-1b23-4e10-be1e-7328952623c0/konnectivity-agent/0.log" limit=1024
Jul 27 19:18:00.509819 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.509752    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mwdjc_16340d27-1b23-4e10-be1e-7328952623c0/konnectivity-agent/0.log" limit=1024
Jul 27 19:18:00.923494 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.923465    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 19:18:00.996865 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.996700    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T19:18:00.923489433Z","UUID":"6f34174e-dcfe-42fc-822d-2332b6859d68","Handler":null,"Name":"","Endpoint":""}
Jul 27 19:18:00.998300 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.998280    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 19:18:00.998428 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:00.998321    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 19:18:01.080484 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:01.080453    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:18:01.080675 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:01.080590    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:18:01.182166 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:01.182129    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8nw28" event={"ID":"267672f4-3a5e-4af6-b2b5-c2e3653f4fcc","Type":"ContainerStarted","Data":"bb3869b0540641534380996b08cb084611a24581a4e5297c8d8c010783229c58"}
Jul 27 19:18:01.184169 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:01.184141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld" event={"ID":"268e36f2-3edd-4577-bd4d-7742d6859334","Type":"ContainerStarted","Data":"59dc2c9b0fae73cb6d154ced962db4e6e107e4107e02e00e771713c0ce5902f7"}
Jul 27 19:18:01.187642 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:01.187580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m552v" event={"ID":"94d97464-c11b-47a8-af6b-4c0f4d531794","Type":"ContainerStarted","Data":"0731d95fcc236721b541c18e11f4463072b3ce2350019bab132b41adcaf9f36b"}
Jul 27 19:18:01.187741 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:01.187660    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m552v" event={"ID":"94d97464-c11b-47a8-af6b-4c0f4d531794","Type":"ContainerStarted","Data":"7eb90f9dce4da07c37fc7d89ebaa2ccef4e439a4bcb4966d8b9149a9fc03328e"}
Jul 27 19:18:01.187741 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:01.187712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m552v" event={"ID":"94d97464-c11b-47a8-af6b-4c0f4d531794","Type":"ContainerStarted","Data":"ad82e35db1b819ecc6387003fd4a483b88b90aa0e643cdb36ba4d20cad3316f9"}
Jul 27 19:18:01.198596 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:01.198560    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8nw28" podStartSLOduration=9.055248633 podStartE2EDuration="23.198547646s" totalImagesPullingTime="14.143299013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:01.198126312 +0000 UTC m=+23.713830816" watchObservedRunningTime="2026-07-27 19:18:01.198547646 +0000 UTC m=+23.714252149"
Jul 27 19:18:01.326731 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:01.326701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mwdjc_16340d27-1b23-4e10-be1e-7328952623c0/konnectivity-agent/0.log" limit=1024
Jul 27 19:18:01.466928 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:01.466805    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mwdjc_16340d27-1b23-4e10-be1e-7328952623c0/konnectivity-agent/0.log" limit=1024
Jul 27 19:18:01.607163 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:01.607108    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mwdjc"
Jul 27 19:18:01.607843 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:01.607823    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mwdjc"
Jul 27 19:18:02.080448 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:02.080418    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:18:02.080581 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:02.080532    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:18:02.080650 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:02.080621    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:18:02.080754 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:02.080729    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:18:02.192402 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:02.192361    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld" event={"ID":"268e36f2-3edd-4577-bd4d-7742d6859334","Type":"ContainerStarted","Data":"ccee82e291d93eccf650aabf6d1505963c6a0d514fa99946956cf2c66451c81b"}
Jul 27 19:18:02.193045 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:02.193026    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mwdjc"
Jul 27 19:18:02.210275 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:02.210176    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-blnld" podStartSLOduration=3.611624638 podStartE2EDuration="24.2101629s" totalImagesPullingTime="20.598538262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:02.209964558 +0000 UTC m=+24.725669063" watchObservedRunningTime="2026-07-27 19:18:02.2101629 +0000 UTC m=+24.725867404"
Jul 27 19:18:02.339806 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:02.339774    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mwdjc_16340d27-1b23-4e10-be1e-7328952623c0/konnectivity-agent/0.log" limit=1024
Jul 27 19:18:03.080686 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:03.080645    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:18:03.080846 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:03.080794    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:18:03.198394 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:03.198350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m552v" event={"ID":"94d97464-c11b-47a8-af6b-4c0f4d531794","Type":"ContainerStarted","Data":"0c53295c6c36e09eafe7d01cb361b1a71ad3f1ba30e70c5646605918d37fad10"}
Jul 27 19:18:04.080581 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:04.080550    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:18:04.080772 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:04.080550    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:18:04.080772 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:04.080670    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:18:04.080886 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:04.080774    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:18:05.080721 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:05.080550    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:18:05.081414 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:05.080816    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:18:05.205790 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:05.205758    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m552v" event={"ID":"94d97464-c11b-47a8-af6b-4c0f4d531794","Type":"ContainerStarted","Data":"baa48bd3158614d422f377fcbc2d3a35072cf53876c06d075f072ae59ee32e54"}
Jul 27 19:18:05.206059 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:05.206025    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:18:05.206206 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:05.206099    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:18:05.207518 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:05.207489    2580 generic.go:356] "Generic (PLEG): container finished" podID="d6a22011-4f70-455b-9a51-964e395b2a2c" containerID="83592ed0b0a485d287965109aaeea2bac6b0d58c36c185619b7f2f9b31546582" exitCode=0
Jul 27 19:18:05.207615 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:05.207537    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krd7c" event={"ID":"d6a22011-4f70-455b-9a51-964e395b2a2c","Type":"ContainerDied","Data":"83592ed0b0a485d287965109aaeea2bac6b0d58c36c185619b7f2f9b31546582"}
Jul 27 19:18:05.220948 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:05.220927    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:18:05.233666 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:05.233626    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m552v" podStartSLOduration=8.445179274000001 podStartE2EDuration="27.23361293s" totalImagesPullingTime="18.788433656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:05.233056063 +0000 UTC m=+27.748760567" watchObservedRunningTime="2026-07-27 19:18:05.23361293 +0000 UTC m=+27.749317433"
Jul 27 19:18:06.080858 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:06.080678    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:18:06.081390 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:06.080730    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:18:06.081390 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:06.080932    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:18:06.081390 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:06.080996    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:18:06.210576 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:06.210545    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:18:06.225258 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:06.225211    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:18:06.651735 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:06.651656    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7vhjf"]
Jul 27 19:18:06.651920 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:06.651760    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:18:06.651920 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:06.651863    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:18:06.655043 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:06.655018    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nr4fm"]
Jul 27 19:18:06.655179 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:06.655127    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:18:06.655237 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:06.655221    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:18:06.655672 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:06.655646    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2f68s"]
Jul 27 19:18:06.655775 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:06.655727    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:18:06.655825 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:06.655800    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:18:07.216227 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:07.213762    2580 generic.go:356] "Generic (PLEG): container finished" podID="d6a22011-4f70-455b-9a51-964e395b2a2c" containerID="62f0eef8546d3b908042334615eaa9d91bbaac255be83be623ae7d8e597d7165" exitCode=0
Jul 27 19:18:07.216227 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:07.214008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krd7c" event={"ID":"d6a22011-4f70-455b-9a51-964e395b2a2c","Type":"ContainerDied","Data":"62f0eef8546d3b908042334615eaa9d91bbaac255be83be623ae7d8e597d7165"}
Jul 27 19:18:08.081078 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:08.081044    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:18:08.081270 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:08.081136    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:18:08.081270 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:08.081224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:18:08.081391 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:08.081371    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:18:08.081463 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:08.081405    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:18:08.081525 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:08.081466    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:18:09.220203 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:09.220010    2580 generic.go:356] "Generic (PLEG): container finished" podID="d6a22011-4f70-455b-9a51-964e395b2a2c" containerID="c09f7bf6a98df7751c93fd6af0d116de8be2edf8c2da6b6b4f8f506c6e228887" exitCode=0
Jul 27 19:18:09.220203 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:09.220050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krd7c" event={"ID":"d6a22011-4f70-455b-9a51-964e395b2a2c","Type":"ContainerDied","Data":"c09f7bf6a98df7751c93fd6af0d116de8be2edf8c2da6b6b4f8f506c6e228887"}
Jul 27 19:18:10.080610 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:10.080572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:18:10.080810 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:10.080706    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:18:10.080810 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:10.080572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:18:10.080810 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:10.080796    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:18:10.080810 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:10.080572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:18:10.081039 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:10.080870    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:18:11.779606 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:11.779569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs\") pod \"network-metrics-daemon-nr4fm\" (UID: \"bd4ebac6-0e11-4e10-a788-04aef97f902f\") " pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:18:11.779606 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:11.779613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdkb\" (UniqueName: \"kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb\") pod \"network-check-target-7vhjf\" (UID: \"58348a13-fcc8-4f55-9975-1b653ec5a0ad\") " pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:18:11.780215 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:11.779637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:18:11.780215 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:11.779739    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:18:11.780215 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:11.779745    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:18:11.780215 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:11.779764    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:18:11.780215 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:11.779792    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:18:11.780215 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:11.779804    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret podName:19c1c8b7-cc69-4cad-aa77-67102d7353d8 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:43.779785879 +0000 UTC m=+66.295490367 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret") pod "global-pull-secret-syncer-2f68s" (UID: "19c1c8b7-cc69-4cad-aa77-67102d7353d8") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:18:11.780215 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:11.779827    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs podName:bd4ebac6-0e11-4e10-a788-04aef97f902f nodeName:}" failed. No retries permitted until 2026-07-27 19:18:43.779816506 +0000 UTC m=+66.295520987 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs") pod "network-metrics-daemon-nr4fm" (UID: "bd4ebac6-0e11-4e10-a788-04aef97f902f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:18:11.780215 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:11.779806    2580 projected.go:196] Error preparing data for projected volume kube-api-access-gzdkb for pod openshift-network-diagnostics/network-check-target-7vhjf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:18:11.780215 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:11.779860    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb podName:58348a13-fcc8-4f55-9975-1b653ec5a0ad nodeName:}" failed. No retries permitted until 2026-07-27 19:18:43.779854634 +0000 UTC m=+66.295559129 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzdkb" (UniqueName: "kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb") pod "network-check-target-7vhjf" (UID: "58348a13-fcc8-4f55-9975-1b653ec5a0ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:18:12.080675 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.080616    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:18:12.080675 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.080629    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:18:12.080883 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:12.080767    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nr4fm" podUID="bd4ebac6-0e11-4e10-a788-04aef97f902f"
Jul 27 19:18:12.080883 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.080798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:18:12.080994 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:12.080909    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7vhjf" podUID="58348a13-fcc8-4f55-9975-1b653ec5a0ad"
Jul 27 19:18:12.081048 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:12.080998    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f68s" podUID="19c1c8b7-cc69-4cad-aa77-67102d7353d8"
Jul 27 19:18:12.250018 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.249990    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-103.ec2.internal" event="NodeReady"
Jul 27 19:18:12.250199 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.250152    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 19:18:12.313864 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.313828    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"]
Jul 27 19:18:12.316698 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.316673    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.319262 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.319216    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 19:18:12.319938 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.319916    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f5t9g\""
Jul 27 19:18:12.320066 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.320008    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 19:18:12.322338 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.322319    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 19:18:12.329307 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.329263    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 19:18:12.332257 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.332221    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r4tc4"]
Jul 27 19:18:12.333999 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.333982    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r4tc4"
Jul 27 19:18:12.336513 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.336493    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lbsxn"]
Jul 27 19:18:12.336890 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.336870    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 19:18:12.337091 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.337071    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 19:18:12.337165 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.337121    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 19:18:12.337227 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.337073    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dlv9z\""
Jul 27 19:18:12.338628 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.338613    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:12.343356 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.343194    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 19:18:12.343356 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.343315    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m4wg8\""
Jul 27 19:18:12.343505 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.343464    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 19:18:12.344118 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.344089    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"]
Jul 27 19:18:12.354952 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.354931    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r4tc4"]
Jul 27 19:18:12.360892 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.360870    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lbsxn"]
Jul 27 19:18:12.384714 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.384679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.384852 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.384747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be513394-5a1a-435b-87c2-348762f04fe6-installation-pull-secrets\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.384852 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.384836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be513394-5a1a-435b-87c2-348762f04fe6-image-registry-private-configuration\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.384961 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.384861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be513394-5a1a-435b-87c2-348762f04fe6-registry-certificates\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.384961 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.384884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vr6v\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-kube-api-access-9vr6v\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.384961 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.384909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be513394-5a1a-435b-87c2-348762f04fe6-trusted-ca\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.384961 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.384932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-bound-sa-token\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.385103 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.385025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be513394-5a1a-435b-87c2-348762f04fe6-ca-trust-extracted\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.486428 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.486389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be513394-5a1a-435b-87c2-348762f04fe6-ca-trust-extracted\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.486617 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.486439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bfc017-280a-4902-8f87-008710dab93d-config-volume\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:12.486617 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.486464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwsn4\" (UniqueName: \"kubernetes.io/projected/a6bfc017-280a-4902-8f87-008710dab93d-kube-api-access-kwsn4\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:12.486617 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.486533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.486617 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.486597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be513394-5a1a-435b-87c2-348762f04fe6-installation-pull-secrets\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.486871 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.486631    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6wwj\" (UniqueName: \"kubernetes.io/projected/0eaedab2-c519-483f-b30d-8aede92ec1d8-kube-api-access-z6wwj\") pod \"ingress-canary-r4tc4\" (UID: \"0eaedab2-c519-483f-b30d-8aede92ec1d8\") " pod="openshift-ingress-canary/ingress-canary-r4tc4"
Jul 27 19:18:12.486871 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.486659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:12.486871 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.486703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be513394-5a1a-435b-87c2-348762f04fe6-image-registry-private-configuration\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.486871 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:12.486724    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:18:12.486871 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:12.486743    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9b8fc8fd5-kbqcr: secret "image-registry-tls" not found
Jul 27 19:18:12.486871 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:12.486804    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls podName:be513394-5a1a-435b-87c2-348762f04fe6 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:12.986782439 +0000 UTC m=+35.502486928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls") pod "image-registry-9b8fc8fd5-kbqcr" (UID: "be513394-5a1a-435b-87c2-348762f04fe6") : secret "image-registry-tls" not found
Jul 27 19:18:12.487149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.486920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be513394-5a1a-435b-87c2-348762f04fe6-ca-trust-extracted\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.487149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.486730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be513394-5a1a-435b-87c2-348762f04fe6-registry-certificates\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.487149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.487003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vr6v\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-kube-api-access-9vr6v\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.487149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.487044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be513394-5a1a-435b-87c2-348762f04fe6-trusted-ca\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.487149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.487069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-bound-sa-token\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.487149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.487095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0eaedab2-c519-483f-b30d-8aede92ec1d8-cert\") pod \"ingress-canary-r4tc4\" (UID: \"0eaedab2-c519-483f-b30d-8aede92ec1d8\") " pod="openshift-ingress-canary/ingress-canary-r4tc4"
Jul 27 19:18:12.487454 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.487117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6bfc017-280a-4902-8f87-008710dab93d-tmp-dir\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:12.490681 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.487815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be513394-5a1a-435b-87c2-348762f04fe6-registry-certificates\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.490810 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.490780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be513394-5a1a-435b-87c2-348762f04fe6-trusted-ca\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.492737 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.492717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be513394-5a1a-435b-87c2-348762f04fe6-installation-pull-secrets\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.492862 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.492845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be513394-5a1a-435b-87c2-348762f04fe6-image-registry-private-configuration\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.496422 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.496394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-bound-sa-token\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.496555 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.496529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vr6v\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-kube-api-access-9vr6v\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.588776 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.588687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bfc017-280a-4902-8f87-008710dab93d-config-volume\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:12.588776 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.588724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwsn4\" (UniqueName: \"kubernetes.io/projected/a6bfc017-280a-4902-8f87-008710dab93d-kube-api-access-kwsn4\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:12.588776 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.588771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wwj\" (UniqueName: \"kubernetes.io/projected/0eaedab2-c519-483f-b30d-8aede92ec1d8-kube-api-access-z6wwj\") pod \"ingress-canary-r4tc4\" (UID: \"0eaedab2-c519-483f-b30d-8aede92ec1d8\") " pod="openshift-ingress-canary/ingress-canary-r4tc4"
Jul 27 19:18:12.589048 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.588790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:12.589048 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:12.588885    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:18:12.589048 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:12.588945    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls podName:a6bfc017-280a-4902-8f87-008710dab93d nodeName:}" failed. No retries permitted until 2026-07-27 19:18:13.088926427 +0000 UTC m=+35.604630930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls") pod "dns-default-lbsxn" (UID: "a6bfc017-280a-4902-8f87-008710dab93d") : secret "dns-default-metrics-tls" not found
Jul 27 19:18:12.589048 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.589010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0eaedab2-c519-483f-b30d-8aede92ec1d8-cert\") pod \"ingress-canary-r4tc4\" (UID: \"0eaedab2-c519-483f-b30d-8aede92ec1d8\") " pod="openshift-ingress-canary/ingress-canary-r4tc4"
Jul 27 19:18:12.589280 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.589052    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6bfc017-280a-4902-8f87-008710dab93d-tmp-dir\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:12.589428 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.589401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bfc017-280a-4902-8f87-008710dab93d-config-volume\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:12.589428 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.589416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6bfc017-280a-4902-8f87-008710dab93d-tmp-dir\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:12.591853 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.591829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0eaedab2-c519-483f-b30d-8aede92ec1d8-cert\") pod \"ingress-canary-r4tc4\" (UID: \"0eaedab2-c519-483f-b30d-8aede92ec1d8\") " pod="openshift-ingress-canary/ingress-canary-r4tc4"
Jul 27 19:18:12.602382 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.602360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wwj\" (UniqueName: \"kubernetes.io/projected/0eaedab2-c519-483f-b30d-8aede92ec1d8-kube-api-access-z6wwj\") pod \"ingress-canary-r4tc4\" (UID: \"0eaedab2-c519-483f-b30d-8aede92ec1d8\") " pod="openshift-ingress-canary/ingress-canary-r4tc4"
Jul 27 19:18:12.603572 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.603549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwsn4\" (UniqueName: \"kubernetes.io/projected/a6bfc017-280a-4902-8f87-008710dab93d-kube-api-access-kwsn4\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:12.645719 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.645681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r4tc4"
Jul 27 19:18:12.992020 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:12.991941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:12.992607 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:12.992094    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:18:12.992607 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:12.992111    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9b8fc8fd5-kbqcr: secret "image-registry-tls" not found
Jul 27 19:18:12.992607 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:12.992173    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls podName:be513394-5a1a-435b-87c2-348762f04fe6 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:13.992152379 +0000 UTC m=+36.507856865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls") pod "image-registry-9b8fc8fd5-kbqcr" (UID: "be513394-5a1a-435b-87c2-348762f04fe6") : secret "image-registry-tls" not found
Jul 27 19:18:13.092952 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:13.092919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:13.093131 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:13.093072    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:18:13.093187 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:13.093152    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls podName:a6bfc017-280a-4902-8f87-008710dab93d nodeName:}" failed. No retries permitted until 2026-07-27 19:18:14.093130586 +0000 UTC m=+36.608835073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls") pod "dns-default-lbsxn" (UID: "a6bfc017-280a-4902-8f87-008710dab93d") : secret "dns-default-metrics-tls" not found
Jul 27 19:18:13.297749 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:13.297582    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r4tc4"]
Jul 27 19:18:13.301331 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:18:13.301302    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eaedab2_c519_483f_b30d_8aede92ec1d8.slice/crio-739fb4d0812013db711feca4cec39fa0d5e0c2b8a0ad407d1e91e50f0d2d210a WatchSource:0}: Error finding container 739fb4d0812013db711feca4cec39fa0d5e0c2b8a0ad407d1e91e50f0d2d210a: Status 404 returned error can't find the container with id 739fb4d0812013db711feca4cec39fa0d5e0c2b8a0ad407d1e91e50f0d2d210a
Jul 27 19:18:14.000106 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:14.000010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:14.000699 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:14.000179    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:18:14.000699 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:14.000200    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9b8fc8fd5-kbqcr: secret "image-registry-tls" not found
Jul 27 19:18:14.000699 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:14.000293    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls podName:be513394-5a1a-435b-87c2-348762f04fe6 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:16.000268221 +0000 UTC m=+38.515972708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls") pod "image-registry-9b8fc8fd5-kbqcr" (UID: "be513394-5a1a-435b-87c2-348762f04fe6") : secret "image-registry-tls" not found
Jul 27 19:18:14.080664 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:14.080636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:18:14.080811 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:14.080646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:18:14.080811 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:14.080774    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:18:14.084705 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:14.084680    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 19:18:14.084852 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:14.084716    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lvnq9\""
Jul 27 19:18:14.084852 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:14.084744    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 19:18:14.084852 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:14.084716    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 19:18:14.084852 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:14.084790    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d7dln\""
Jul 27 19:18:14.085112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:14.085097    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 19:18:14.100941 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:14.100923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:14.101025 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:14.101018    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:18:14.101066 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:14.101063    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls podName:a6bfc017-280a-4902-8f87-008710dab93d nodeName:}" failed. No retries permitted until 2026-07-27 19:18:16.101049644 +0000 UTC m=+38.616754130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls") pod "dns-default-lbsxn" (UID: "a6bfc017-280a-4902-8f87-008710dab93d") : secret "dns-default-metrics-tls" not found
Jul 27 19:18:14.235196 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:14.235156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r4tc4" event={"ID":"0eaedab2-c519-483f-b30d-8aede92ec1d8","Type":"ContainerStarted","Data":"739fb4d0812013db711feca4cec39fa0d5e0c2b8a0ad407d1e91e50f0d2d210a"}
Jul 27 19:18:14.238287 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:14.238263    2580 generic.go:356] "Generic (PLEG): container finished" podID="d6a22011-4f70-455b-9a51-964e395b2a2c" containerID="ed1a0fe23d9df5e5579b92e331b39facedcf2e48c2c072205866664e6b692d88" exitCode=0
Jul 27 19:18:14.238401 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:14.238294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krd7c" event={"ID":"d6a22011-4f70-455b-9a51-964e395b2a2c","Type":"ContainerDied","Data":"ed1a0fe23d9df5e5579b92e331b39facedcf2e48c2c072205866664e6b692d88"}
Jul 27 19:18:15.243847 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:15.243809    2580 generic.go:356] "Generic (PLEG): container finished" podID="d6a22011-4f70-455b-9a51-964e395b2a2c" containerID="ed45bc72139690362d55b4d5ef539f5b0433e3114db1774c8c31ebf999b5e000" exitCode=0
Jul 27 19:18:15.244313 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:15.243850    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krd7c" event={"ID":"d6a22011-4f70-455b-9a51-964e395b2a2c","Type":"ContainerDied","Data":"ed45bc72139690362d55b4d5ef539f5b0433e3114db1774c8c31ebf999b5e000"}
Jul 27 19:18:16.018046 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:16.017998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:16.018224 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:16.018167    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:18:16.018224 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:16.018190    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9b8fc8fd5-kbqcr: secret "image-registry-tls" not found
Jul 27 19:18:16.018321 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:16.018271    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls podName:be513394-5a1a-435b-87c2-348762f04fe6 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:20.018236287 +0000 UTC m=+42.533940768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls") pod "image-registry-9b8fc8fd5-kbqcr" (UID: "be513394-5a1a-435b-87c2-348762f04fe6") : secret "image-registry-tls" not found
Jul 27 19:18:16.118872 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:16.118833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:16.119051 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:16.119032    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:18:16.119112 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:16.119086    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls podName:a6bfc017-280a-4902-8f87-008710dab93d nodeName:}" failed. No retries permitted until 2026-07-27 19:18:20.119069783 +0000 UTC m=+42.634774285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls") pod "dns-default-lbsxn" (UID: "a6bfc017-280a-4902-8f87-008710dab93d") : secret "dns-default-metrics-tls" not found
Jul 27 19:18:16.250557 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:16.250511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-krd7c" event={"ID":"d6a22011-4f70-455b-9a51-964e395b2a2c","Type":"ContainerStarted","Data":"e7ffd26059ff43daf3f6269a5310dd88804cf3ad10301e50a4b1995e3f2cce1d"}
Jul 27 19:18:16.285903 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:16.285846    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-krd7c" podStartSLOduration=9.270739799 podStartE2EDuration="38.285823226s" totalImagesPullingTime="29.015083427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:16.284469588 +0000 UTC m=+38.800174092" watchObservedRunningTime="2026-07-27 19:18:16.285823226 +0000 UTC m=+38.801527725"
Jul 27 19:18:17.254297 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:17.254072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r4tc4" event={"ID":"0eaedab2-c519-483f-b30d-8aede92ec1d8","Type":"ContainerStarted","Data":"9cac71ce4650e449d41a941b53e0e425922ce7f34f6170e5cfcc4d301ec57cd1"}
Jul 27 19:18:17.289684 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:17.289638    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r4tc4" podStartSLOduration=1.655458114 podStartE2EDuration="5.289622114s" totalImagesPullingTime="3.634164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:17.289421111 +0000 UTC m=+39.805125615" watchObservedRunningTime="2026-07-27 19:18:17.289622114 +0000 UTC m=+39.805326596"
Jul 27 19:18:20.045031 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:20.044996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:20.045513 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:20.045140    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:18:20.045513 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:20.045159    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9b8fc8fd5-kbqcr: secret "image-registry-tls" not found
Jul 27 19:18:20.045513 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:20.045210    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls podName:be513394-5a1a-435b-87c2-348762f04fe6 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:28.045195625 +0000 UTC m=+50.560900111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls") pod "image-registry-9b8fc8fd5-kbqcr" (UID: "be513394-5a1a-435b-87c2-348762f04fe6") : secret "image-registry-tls" not found
Jul 27 19:18:20.146287 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:20.146238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:20.146412 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:20.146343    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:18:20.146412 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:20.146400    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls podName:a6bfc017-280a-4902-8f87-008710dab93d nodeName:}" failed. No retries permitted until 2026-07-27 19:18:28.146386042 +0000 UTC m=+50.662090528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls") pod "dns-default-lbsxn" (UID: "a6bfc017-280a-4902-8f87-008710dab93d") : secret "dns-default-metrics-tls" not found
Jul 27 19:18:21.983642 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:21.983608    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9bx4h"]
Jul 27 19:18:22.005008 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.004986    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.010881 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.010855    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9bx4h"]
Jul 27 19:18:22.012174 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.012150    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-djng7\""
Jul 27 19:18:22.012299 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.012182    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 19:18:22.012877 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.012857    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 19:18:22.012976 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.012860    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 19:18:22.013039 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.012864    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 19:18:22.161297 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.161267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.161297 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.161296    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkftv\" (UniqueName: \"kubernetes.io/projected/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-api-access-gkftv\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.161544 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.161319    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.161544 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.161339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-crio-socket\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.161544 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.161459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-data-volume\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.262557 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.262484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.262557 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.262515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkftv\" (UniqueName: \"kubernetes.io/projected/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-api-access-gkftv\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.262557 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.262537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.262789 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.262578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-crio-socket\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.262789 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:22.262623    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:18:22.262789 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.262655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-data-volume\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.262789 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:22.262681    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:18:22.262789 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:22.262685    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls podName:3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:22.762667245 +0000 UTC m=+45.278371729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9bx4h" (UID: "3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:18:22.262789 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:22.262745    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm podName:3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:22.762729409 +0000 UTC m=+45.278433906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9bx4h" (UID: "3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4") : configmap "kube-rbac-proxy" not found
Jul 27 19:18:22.263034 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.262805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-crio-socket\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.263034 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.262929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-data-volume\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.280953 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.280931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkftv\" (UniqueName: \"kubernetes.io/projected/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-api-access-gkftv\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.765526 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.765485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.765715 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:22.765533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:22.765715 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:22.765613    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:18:22.765715 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:22.765667    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:18:22.765715 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:22.765692    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm podName:3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:23.765674895 +0000 UTC m=+46.281379377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9bx4h" (UID: "3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4") : configmap "kube-rbac-proxy" not found
Jul 27 19:18:22.765715 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:22.765715    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls podName:3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:23.765700888 +0000 UTC m=+46.281405387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9bx4h" (UID: "3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:18:23.773516 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:23.773483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:23.773516 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:23.773515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:23.773910 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:23.773599    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:18:23.773910 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:23.773616    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:18:23.773910 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:23.773656    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm podName:3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:25.77364162 +0000 UTC m=+48.289346102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9bx4h" (UID: "3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4") : configmap "kube-rbac-proxy" not found
Jul 27 19:18:23.773910 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:23.773670    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls podName:3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:25.773663226 +0000 UTC m=+48.289367708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9bx4h" (UID: "3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:18:24.145709 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.145681    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w9hmm_379c2805-26eb-4141-bd63-49ce6ad17fee/dns-node-resolver/0.log"
Jul 27 19:18:24.679147 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.679119    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-spr8r"]
Jul 27 19:18:24.724138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.724110    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-spr8r"]
Jul 27 19:18:24.724291 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.724195    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-spr8r"
Jul 27 19:18:24.727526 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.727505    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 27 19:18:24.727879 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.727864    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 27 19:18:24.728348 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.728329    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 27 19:18:24.728429 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.728381    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 27 19:18:24.728474 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.728445    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-g8g2m\""
Jul 27 19:18:24.753780 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.753762    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8rsqx_dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3/node-ca/0.log"
Jul 27 19:18:24.883965 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.883933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7-signing-cabundle\") pod \"service-ca-658d7b7c78-spr8r\" (UID: \"99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7\") " pod="openshift-service-ca/service-ca-658d7b7c78-spr8r"
Jul 27 19:18:24.884360 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.883970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j54tx\" (UniqueName: \"kubernetes.io/projected/99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7-kube-api-access-j54tx\") pod \"service-ca-658d7b7c78-spr8r\" (UID: \"99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7\") " pod="openshift-service-ca/service-ca-658d7b7c78-spr8r"
Jul 27 19:18:24.884360 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.884026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7-signing-key\") pod \"service-ca-658d7b7c78-spr8r\" (UID: \"99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7\") " pod="openshift-service-ca/service-ca-658d7b7c78-spr8r"
Jul 27 19:18:24.984315 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.984257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7-signing-cabundle\") pod \"service-ca-658d7b7c78-spr8r\" (UID: \"99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7\") " pod="openshift-service-ca/service-ca-658d7b7c78-spr8r"
Jul 27 19:18:24.984315 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.984287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j54tx\" (UniqueName: \"kubernetes.io/projected/99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7-kube-api-access-j54tx\") pod \"service-ca-658d7b7c78-spr8r\" (UID: \"99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7\") " pod="openshift-service-ca/service-ca-658d7b7c78-spr8r"
Jul 27 19:18:24.984459 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.984323    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7-signing-key\") pod \"service-ca-658d7b7c78-spr8r\" (UID: \"99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7\") " pod="openshift-service-ca/service-ca-658d7b7c78-spr8r"
Jul 27 19:18:24.984983 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.984964    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7-signing-cabundle\") pod \"service-ca-658d7b7c78-spr8r\" (UID: \"99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7\") " pod="openshift-service-ca/service-ca-658d7b7c78-spr8r"
Jul 27 19:18:24.992495 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:24.992470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7-signing-key\") pod \"service-ca-658d7b7c78-spr8r\" (UID: \"99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7\") " pod="openshift-service-ca/service-ca-658d7b7c78-spr8r"
Jul 27 19:18:25.002807 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:25.002784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j54tx\" (UniqueName: \"kubernetes.io/projected/99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7-kube-api-access-j54tx\") pod \"service-ca-658d7b7c78-spr8r\" (UID: \"99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7\") " pod="openshift-service-ca/service-ca-658d7b7c78-spr8r"
Jul 27 19:18:25.044001 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:25.043980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-spr8r"
Jul 27 19:18:25.180126 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:25.180101    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-spr8r"]
Jul 27 19:18:25.181719 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:18:25.181693    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c17ac6_7eb6_4f5a_9633_3fa2b8494ae7.slice/crio-0921c0cad41cf721ee65c2844a810d72868afba2c00777636012c58f0a057d5c WatchSource:0}: Error finding container 0921c0cad41cf721ee65c2844a810d72868afba2c00777636012c58f0a057d5c: Status 404 returned error can't find the container with id 0921c0cad41cf721ee65c2844a810d72868afba2c00777636012c58f0a057d5c
Jul 27 19:18:25.273894 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:25.273827    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-spr8r" event={"ID":"99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7","Type":"ContainerStarted","Data":"0921c0cad41cf721ee65c2844a810d72868afba2c00777636012c58f0a057d5c"}
Jul 27 19:18:25.790562 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:25.790531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:25.790562 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:25.790564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:25.790756 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:25.790642    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:18:25.790756 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:25.790663    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:18:25.790756 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:25.790707    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm podName:3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:29.790691549 +0000 UTC m=+52.306396035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9bx4h" (UID: "3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4") : configmap "kube-rbac-proxy" not found
Jul 27 19:18:25.790756 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:25.790721    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls podName:3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:29.790714599 +0000 UTC m=+52.306419081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9bx4h" (UID: "3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:18:25.952853 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:25.952821    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r4tc4_0eaedab2-c519-483f-b30d-8aede92ec1d8/serve-healthcheck-canary/0.log"
Jul 27 19:18:28.108120 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:28.108085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:28.108473 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:28.108208    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:18:28.108473 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:28.108219    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9b8fc8fd5-kbqcr: secret "image-registry-tls" not found
Jul 27 19:18:28.108473 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:28.108283    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls podName:be513394-5a1a-435b-87c2-348762f04fe6 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:44.108268994 +0000 UTC m=+66.623973480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls") pod "image-registry-9b8fc8fd5-kbqcr" (UID: "be513394-5a1a-435b-87c2-348762f04fe6") : secret "image-registry-tls" not found
Jul 27 19:18:28.208934 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:28.208902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:28.209097 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:28.209076    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:18:28.209173 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:28.209155    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls podName:a6bfc017-280a-4902-8f87-008710dab93d nodeName:}" failed. No retries permitted until 2026-07-27 19:18:44.209133988 +0000 UTC m=+66.724838474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls") pod "dns-default-lbsxn" (UID: "a6bfc017-280a-4902-8f87-008710dab93d") : secret "dns-default-metrics-tls" not found
Jul 27 19:18:28.282661 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:28.282631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-spr8r" event={"ID":"99c17ac6-7eb6-4f5a-9633-3fa2b8494ae7","Type":"ContainerStarted","Data":"e573cd911938d6f40b8bd8bcb33bfbc322557f6a6718bbb5d6c21b232d89f170"}
Jul 27 19:18:28.321276 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:28.321216    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-spr8r" podStartSLOduration=2.045731108 podStartE2EDuration="4.321203475s" totalImagesPullingTime="2.275472367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:28.319789477 +0000 UTC m=+50.835493980" watchObservedRunningTime="2026-07-27 19:18:28.321203475 +0000 UTC m=+50.836907979"
Jul 27 19:18:29.823640 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:29.823603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:29.823640 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:29.823640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:29.824068 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:29.823724    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:18:29.824068 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:29.823759    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:18:29.824068 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:29.823786    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm podName:3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:37.823770805 +0000 UTC m=+60.339475289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9bx4h" (UID: "3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4") : configmap "kube-rbac-proxy" not found
Jul 27 19:18:29.824068 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:29.823800    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls podName:3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:37.823794417 +0000 UTC m=+60.339498899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9bx4h" (UID: "3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:18:37.891085 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:37.891049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:37.891085 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:37.891088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:37.891800 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:37.891238    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:18:37.891800 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:18:37.891318    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls podName:3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:53.891304354 +0000 UTC m=+76.407008837 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9bx4h" (UID: "3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:18:37.891800 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:37.891691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:38.233458 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:38.233387    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m552v"
Jul 27 19:18:43.836253 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:43.836192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs\") pod \"network-metrics-daemon-nr4fm\" (UID: \"bd4ebac6-0e11-4e10-a788-04aef97f902f\") " pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:18:43.836253 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:43.836266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdkb\" (UniqueName: \"kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb\") pod \"network-check-target-7vhjf\" (UID: \"58348a13-fcc8-4f55-9975-1b653ec5a0ad\") " pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:18:43.836681 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:43.836307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:18:43.841214 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:43.841195    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 19:18:43.841977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:43.841960    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 19:18:43.842023 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:43.841960    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 19:18:43.848721 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:43.848701    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 19:18:43.848924 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:43.848903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd4ebac6-0e11-4e10-a788-04aef97f902f-metrics-certs\") pod \"network-metrics-daemon-nr4fm\" (UID: \"bd4ebac6-0e11-4e10-a788-04aef97f902f\") " pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:18:43.849015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:43.849000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/19c1c8b7-cc69-4cad-aa77-67102d7353d8-original-pull-secret\") pod \"global-pull-secret-syncer-2f68s\" (UID: \"19c1c8b7-cc69-4cad-aa77-67102d7353d8\") " pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:18:43.859590 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:43.859565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzdkb\" (UniqueName: \"kubernetes.io/projected/58348a13-fcc8-4f55-9975-1b653ec5a0ad-kube-api-access-gzdkb\") pod \"network-check-target-7vhjf\" (UID: \"58348a13-fcc8-4f55-9975-1b653ec5a0ad\") " pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:18:44.091113 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.091040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f68s"
Jul 27 19:18:44.098403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.098388    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d7dln\""
Jul 27 19:18:44.106515 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.106496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nr4fm"
Jul 27 19:18:44.115663 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.115644    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lvnq9\""
Jul 27 19:18:44.121059 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.121040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:18:44.138526 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.138485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:44.141234 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.141211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls\") pod \"image-registry-9b8fc8fd5-kbqcr\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") " pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:44.234362 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.234325    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2f68s"]
Jul 27 19:18:44.237699 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:18:44.237674    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c1c8b7_cc69_4cad_aa77_67102d7353d8.slice/crio-300546dfe4794928b50e2ce5313964d6872503f3f8c430a5ecfc0a88fb4b3b04 WatchSource:0}: Error finding container 300546dfe4794928b50e2ce5313964d6872503f3f8c430a5ecfc0a88fb4b3b04: Status 404 returned error can't find the container with id 300546dfe4794928b50e2ce5313964d6872503f3f8c430a5ecfc0a88fb4b3b04
Jul 27 19:18:44.238794 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.238766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:44.241204 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.241183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6bfc017-280a-4902-8f87-008710dab93d-metrics-tls\") pod \"dns-default-lbsxn\" (UID: \"a6bfc017-280a-4902-8f87-008710dab93d\") " pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:44.250303 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.250221    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nr4fm"]
Jul 27 19:18:44.253737 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:18:44.253713    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4ebac6_0e11_4e10_a788_04aef97f902f.slice/crio-d7584622432515f53d10f632abff0126fa918693a6ded752fe976dc10d3f9ab4 WatchSource:0}: Error finding container d7584622432515f53d10f632abff0126fa918693a6ded752fe976dc10d3f9ab4: Status 404 returned error can't find the container with id d7584622432515f53d10f632abff0126fa918693a6ded752fe976dc10d3f9ab4
Jul 27 19:18:44.281710 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.281687    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7vhjf"]
Jul 27 19:18:44.284729 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:18:44.284709    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58348a13_fcc8_4f55_9975_1b653ec5a0ad.slice/crio-8bb4eb93cd1698c42f117de3e81d30e946f514a4f3d844909103ccfb09daa9b5 WatchSource:0}: Error finding container 8bb4eb93cd1698c42f117de3e81d30e946f514a4f3d844909103ccfb09daa9b5: Status 404 returned error can't find the container with id 8bb4eb93cd1698c42f117de3e81d30e946f514a4f3d844909103ccfb09daa9b5
Jul 27 19:18:44.320095 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.320058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7vhjf" event={"ID":"58348a13-fcc8-4f55-9975-1b653ec5a0ad","Type":"ContainerStarted","Data":"8bb4eb93cd1698c42f117de3e81d30e946f514a4f3d844909103ccfb09daa9b5"}
Jul 27 19:18:44.321112 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.321094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nr4fm" event={"ID":"bd4ebac6-0e11-4e10-a788-04aef97f902f","Type":"ContainerStarted","Data":"d7584622432515f53d10f632abff0126fa918693a6ded752fe976dc10d3f9ab4"}
Jul 27 19:18:44.322142 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.322123    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2f68s" event={"ID":"19c1c8b7-cc69-4cad-aa77-67102d7353d8","Type":"ContainerStarted","Data":"300546dfe4794928b50e2ce5313964d6872503f3f8c430a5ecfc0a88fb4b3b04"}
Jul 27 19:18:44.431028 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.430961    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f5t9g\""
Jul 27 19:18:44.438998 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.438976    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:44.453817 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.453789    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m4wg8\""
Jul 27 19:18:44.461653 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.461632    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:44.606388 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.606361    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"]
Jul 27 19:18:44.609831 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:18:44.609806    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe513394_5a1a_435b_87c2_348762f04fe6.slice/crio-93b1433cea2c22fc653f81b4591900af41e21bece8bee7063dec6378048acf50 WatchSource:0}: Error finding container 93b1433cea2c22fc653f81b4591900af41e21bece8bee7063dec6378048acf50: Status 404 returned error can't find the container with id 93b1433cea2c22fc653f81b4591900af41e21bece8bee7063dec6378048acf50
Jul 27 19:18:44.625108 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:44.625086    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lbsxn"]
Jul 27 19:18:44.627539 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:18:44.627518    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6bfc017_280a_4902_8f87_008710dab93d.slice/crio-44b2b4e80e64a3d2ae079e2921a375791ad5403dd1f4dfcc9317d6b4f3ed70c2 WatchSource:0}: Error finding container 44b2b4e80e64a3d2ae079e2921a375791ad5403dd1f4dfcc9317d6b4f3ed70c2: Status 404 returned error can't find the container with id 44b2b4e80e64a3d2ae079e2921a375791ad5403dd1f4dfcc9317d6b4f3ed70c2
Jul 27 19:18:45.326115 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:45.326081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lbsxn" event={"ID":"a6bfc017-280a-4902-8f87-008710dab93d","Type":"ContainerStarted","Data":"44b2b4e80e64a3d2ae079e2921a375791ad5403dd1f4dfcc9317d6b4f3ed70c2"}
Jul 27 19:18:45.330979 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:45.330950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr" event={"ID":"be513394-5a1a-435b-87c2-348762f04fe6","Type":"ContainerStarted","Data":"6c791d009c18a0e8b9239e87199d7c4c3b5313f301c57a11f4110eb20af0db81"}
Jul 27 19:18:45.331160 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:45.331002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr" event={"ID":"be513394-5a1a-435b-87c2-348762f04fe6","Type":"ContainerStarted","Data":"93b1433cea2c22fc653f81b4591900af41e21bece8bee7063dec6378048acf50"}
Jul 27 19:18:45.331637 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:45.331619    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:18:45.358817 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:45.358762    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr" podStartSLOduration=34.358743175 podStartE2EDuration="34.358743175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:45.357458499 +0000 UTC m=+67.873163004" watchObservedRunningTime="2026-07-27 19:18:45.358743175 +0000 UTC m=+67.874447679"
Jul 27 19:18:46.338396 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:46.338349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nr4fm" event={"ID":"bd4ebac6-0e11-4e10-a788-04aef97f902f","Type":"ContainerStarted","Data":"5af2e4708ce71739125e65bef90bac44793050ab36077c603b64188eb4165f7b"}
Jul 27 19:18:46.338396 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:46.338401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nr4fm" event={"ID":"bd4ebac6-0e11-4e10-a788-04aef97f902f","Type":"ContainerStarted","Data":"a6c893bf7941fc51fe0e11724acbfe61e8b1afdcf49959b6a1eaf8a27a5e678f"}
Jul 27 19:18:46.356951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:46.356896    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nr4fm" podStartSLOduration=66.923982051 podStartE2EDuration="1m8.356875745s" totalImagesPullingTime="1.432893694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:46.356620385 +0000 UTC m=+68.872324915" watchObservedRunningTime="2026-07-27 19:18:46.356875745 +0000 UTC m=+68.872580253"
Jul 27 19:18:50.350651 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:50.350611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lbsxn" event={"ID":"a6bfc017-280a-4902-8f87-008710dab93d","Type":"ContainerStarted","Data":"6c32df9158519be991ff75cb3d5f43c231e45f0c314c7c1ea1fac17bf217cb8d"}
Jul 27 19:18:50.351123 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:50.350677    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:18:50.351123 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:50.350698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lbsxn" event={"ID":"a6bfc017-280a-4902-8f87-008710dab93d","Type":"ContainerStarted","Data":"8b2c441ed906d569a0d1c83ada37025271f6c37402582a10d8ac4e2712645310"}
Jul 27 19:18:50.351935 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:50.351907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7vhjf" event={"ID":"58348a13-fcc8-4f55-9975-1b653ec5a0ad","Type":"ContainerStarted","Data":"0bdea7b33d57bc4797158bfd8a3904546be9ce38ba5747578c21be3db472dc08"}
Jul 27 19:18:50.352054 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:50.351957    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:18:50.353126 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:50.353094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2f68s" event={"ID":"19c1c8b7-cc69-4cad-aa77-67102d7353d8","Type":"ContainerStarted","Data":"882fb86e31394d1201ede7437e28628709d09ddbf202d295961a1555d51a3cdf"}
Jul 27 19:18:50.374784 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:50.374735    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lbsxn" podStartSLOduration=33.511212947 podStartE2EDuration="38.374720937s" totalImagesPullingTime="4.86350799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:50.374444071 +0000 UTC m=+72.890148578" watchObservedRunningTime="2026-07-27 19:18:50.374720937 +0000 UTC m=+72.890425440"
Jul 27 19:18:50.390573 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:50.390531    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7vhjf" podStartSLOduration=67.187283638 podStartE2EDuration="1m12.390515826s" totalImagesPullingTime="5.203232188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:50.389724484 +0000 UTC m=+72.905428988" watchObservedRunningTime="2026-07-27 19:18:50.390515826 +0000 UTC m=+72.906220375"
Jul 27 19:18:50.405254 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:50.405197    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2f68s" podStartSLOduration=67.142641903 podStartE2EDuration="1m12.405183769s" totalImagesPullingTime="5.262541866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:50.404138106 +0000 UTC m=+72.919842610" watchObservedRunningTime="2026-07-27 19:18:50.405183769 +0000 UTC m=+72.920888274"
Jul 27 19:18:53.913317 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:53.913276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:53.915633 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:53.915608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9bx4h\" (UID: \"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4\") " pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:54.117265 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:54.117215    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-djng7\""
Jul 27 19:18:54.125209 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:54.125188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9bx4h"
Jul 27 19:18:54.246087 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:54.246054    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9bx4h"]
Jul 27 19:18:54.253588 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:18:54.253563    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe8111c_eeef_4de3_af3a_f6fa03cbc3a4.slice/crio-7fd82398489c4cabfdf5ee073b63b021256df4994ed2fa2214445bd14a4f14c9 WatchSource:0}: Error finding container 7fd82398489c4cabfdf5ee073b63b021256df4994ed2fa2214445bd14a4f14c9: Status 404 returned error can't find the container with id 7fd82398489c4cabfdf5ee073b63b021256df4994ed2fa2214445bd14a4f14c9
Jul 27 19:18:54.367538 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:54.367509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9bx4h" event={"ID":"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4","Type":"ContainerStarted","Data":"7fd82398489c4cabfdf5ee073b63b021256df4994ed2fa2214445bd14a4f14c9"}
Jul 27 19:18:56.621645 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.621611    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"]
Jul 27 19:18:56.623820 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.623797    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"
Jul 27 19:18:56.628468 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.628446    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 27 19:18:56.628573 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.628451    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 19:18:56.629075 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.629053    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 19:18:56.633119 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.633102    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 19:18:56.649284 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.649238    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"]
Jul 27 19:18:56.695130 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.695100    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"]
Jul 27 19:18:56.729046 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.729012    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-8dmg4"]
Jul 27 19:18:56.730604 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.730575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndkjs\" (UniqueName: \"kubernetes.io/projected/867cd078-fd5f-49cd-b210-5d3e29127e73-kube-api-access-ndkjs\") pod \"klusterlet-addon-workmgr-65bdb645bc-tj6px\" (UID: \"867cd078-fd5f-49cd-b210-5d3e29127e73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"
Jul 27 19:18:56.730730 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.730622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/867cd078-fd5f-49cd-b210-5d3e29127e73-klusterlet-config\") pod \"klusterlet-addon-workmgr-65bdb645bc-tj6px\" (UID: \"867cd078-fd5f-49cd-b210-5d3e29127e73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"
Jul 27 19:18:56.730730 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.730651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/867cd078-fd5f-49cd-b210-5d3e29127e73-tmp\") pod \"klusterlet-addon-workmgr-65bdb645bc-tj6px\" (UID: \"867cd078-fd5f-49cd-b210-5d3e29127e73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"
Jul 27 19:18:56.731095 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.731073    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c8j6s"]
Jul 27 19:18:56.731229 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.731203    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-8dmg4"
Jul 27 19:18:56.732996 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.732976    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c8j6s"
Jul 27 19:18:56.735107 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.735050    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 19:18:56.735569 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.735398    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lwdf5\""
Jul 27 19:18:56.735569 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.735478    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 27 19:18:56.735569 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.735402    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ffs9j\""
Jul 27 19:18:56.736378 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.736358    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68b5695799-lm7c7"]
Jul 27 19:18:56.736927 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.736913    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 19:18:56.738772 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.738756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.741386 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.741348    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 19:18:56.741480 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.741386    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 19:18:56.742838 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.742608    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 19:18:56.742838 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.742620    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bhwfx\""
Jul 27 19:18:56.742838 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.742611    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 19:18:56.742838 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.742724    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 19:18:56.750524 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.750506    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 19:18:56.757352 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.757332    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-8dmg4"]
Jul 27 19:18:56.759821 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.759802    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c8j6s"]
Jul 27 19:18:56.760894 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.760874    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68b5695799-lm7c7"]
Jul 27 19:18:56.831661 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.831626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvmdn\" (UniqueName: \"kubernetes.io/projected/70705846-cc85-405d-b05a-ada57468d4ad-kube-api-access-mvmdn\") pod \"downloads-7c97dc4555-8dmg4\" (UID: \"70705846-cc85-405d-b05a-ada57468d4ad\") " pod="openshift-console/downloads-7c97dc4555-8dmg4"
Jul 27 19:18:56.831815 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.831681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck67g\" (UniqueName: \"kubernetes.io/projected/0588b6c6-5465-462c-a02f-5c72896bb9db-kube-api-access-ck67g\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.831815 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.831711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-console-config\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.831815 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.831744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-oauth-serving-cert\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.831815 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.831796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/867cd078-fd5f-49cd-b210-5d3e29127e73-klusterlet-config\") pod \"klusterlet-addon-workmgr-65bdb645bc-tj6px\" (UID: \"867cd078-fd5f-49cd-b210-5d3e29127e73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"
Jul 27 19:18:56.831954 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.831873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-trusted-ca-bundle\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.831954 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.831911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/867cd078-fd5f-49cd-b210-5d3e29127e73-tmp\") pod \"klusterlet-addon-workmgr-65bdb645bc-tj6px\" (UID: \"867cd078-fd5f-49cd-b210-5d3e29127e73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"
Jul 27 19:18:56.832026 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.831951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-service-ca\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.832026 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.831986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0588b6c6-5465-462c-a02f-5c72896bb9db-console-serving-cert\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.832026 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.832016    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0588b6c6-5465-462c-a02f-5c72896bb9db-console-oauth-config\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.832147 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.832049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e04b8dd0-d698-46e7-b992-9cb211e8ff1c-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-c8j6s\" (UID: \"e04b8dd0-d698-46e7-b992-9cb211e8ff1c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c8j6s"
Jul 27 19:18:56.832147 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.832095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndkjs\" (UniqueName: \"kubernetes.io/projected/867cd078-fd5f-49cd-b210-5d3e29127e73-kube-api-access-ndkjs\") pod \"klusterlet-addon-workmgr-65bdb645bc-tj6px\" (UID: \"867cd078-fd5f-49cd-b210-5d3e29127e73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"
Jul 27 19:18:56.832304 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.832287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/867cd078-fd5f-49cd-b210-5d3e29127e73-tmp\") pod \"klusterlet-addon-workmgr-65bdb645bc-tj6px\" (UID: \"867cd078-fd5f-49cd-b210-5d3e29127e73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"
Jul 27 19:18:56.834594 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.834574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/867cd078-fd5f-49cd-b210-5d3e29127e73-klusterlet-config\") pod \"klusterlet-addon-workmgr-65bdb645bc-tj6px\" (UID: \"867cd078-fd5f-49cd-b210-5d3e29127e73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"
Jul 27 19:18:56.844926 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.844903    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fcd5949f6-b67bh"]
Jul 27 19:18:56.846872 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.846852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:56.858703 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.858679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndkjs\" (UniqueName: \"kubernetes.io/projected/867cd078-fd5f-49cd-b210-5d3e29127e73-kube-api-access-ndkjs\") pod \"klusterlet-addon-workmgr-65bdb645bc-tj6px\" (UID: \"867cd078-fd5f-49cd-b210-5d3e29127e73\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"
Jul 27 19:18:56.875417 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.875354    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fcd5949f6-b67bh"]
Jul 27 19:18:56.932554 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.932517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-trusted-ca-bundle\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.932725 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.932561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2864c883-0507-46b5-bfb1-8dc3451ef61c-trusted-ca\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:56.932725 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.932681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2864c883-0507-46b5-bfb1-8dc3451ef61c-installation-pull-secrets\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:56.932840 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.932735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-service-ca\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.932840 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.932767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs8ss\" (UniqueName: \"kubernetes.io/projected/2864c883-0507-46b5-bfb1-8dc3451ef61c-kube-api-access-bs8ss\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:56.932931 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.932843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2864c883-0507-46b5-bfb1-8dc3451ef61c-image-registry-private-configuration\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:56.932931 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.932887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2864c883-0507-46b5-bfb1-8dc3451ef61c-registry-certificates\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:56.932931 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.932921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0588b6c6-5465-462c-a02f-5c72896bb9db-console-serving-cert\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.933082 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.932961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0588b6c6-5465-462c-a02f-5c72896bb9db-console-oauth-config\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.933082 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.933009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e04b8dd0-d698-46e7-b992-9cb211e8ff1c-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-c8j6s\" (UID: \"e04b8dd0-d698-46e7-b992-9cb211e8ff1c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c8j6s"
Jul 27 19:18:56.933082 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.933070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2864c883-0507-46b5-bfb1-8dc3451ef61c-bound-sa-token\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:56.933284 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.933106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvmdn\" (UniqueName: \"kubernetes.io/projected/70705846-cc85-405d-b05a-ada57468d4ad-kube-api-access-mvmdn\") pod \"downloads-7c97dc4555-8dmg4\" (UID: \"70705846-cc85-405d-b05a-ada57468d4ad\") " pod="openshift-console/downloads-7c97dc4555-8dmg4"
Jul 27 19:18:56.933284 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.933154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck67g\" (UniqueName: \"kubernetes.io/projected/0588b6c6-5465-462c-a02f-5c72896bb9db-kube-api-access-ck67g\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.933284 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.933181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-console-config\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.933284 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.933217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2864c883-0507-46b5-bfb1-8dc3451ef61c-registry-tls\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:56.933456 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.933287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-oauth-serving-cert\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.933456 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.933325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2864c883-0507-46b5-bfb1-8dc3451ef61c-ca-trust-extracted\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:56.933789 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.933747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-service-ca\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.934475 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.934450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-console-config\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.934620 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.934597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-oauth-serving-cert\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.934950 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.934703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"
Jul 27 19:18:56.934950 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.934789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-trusted-ca-bundle\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.936149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.936121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0588b6c6-5465-462c-a02f-5c72896bb9db-console-oauth-config\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.936264 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.936225    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e04b8dd0-d698-46e7-b992-9cb211e8ff1c-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-c8j6s\" (UID: \"e04b8dd0-d698-46e7-b992-9cb211e8ff1c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c8j6s"
Jul 27 19:18:56.936565 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.936544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0588b6c6-5465-462c-a02f-5c72896bb9db-console-serving-cert\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:56.944803 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.944755    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvmdn\" (UniqueName: \"kubernetes.io/projected/70705846-cc85-405d-b05a-ada57468d4ad-kube-api-access-mvmdn\") pod \"downloads-7c97dc4555-8dmg4\" (UID: \"70705846-cc85-405d-b05a-ada57468d4ad\") " pod="openshift-console/downloads-7c97dc4555-8dmg4"
Jul 27 19:18:56.947161 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:56.947143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck67g\" (UniqueName: \"kubernetes.io/projected/0588b6c6-5465-462c-a02f-5c72896bb9db-kube-api-access-ck67g\") pod \"console-68b5695799-lm7c7\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") " pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:57.034174 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.034137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs8ss\" (UniqueName: \"kubernetes.io/projected/2864c883-0507-46b5-bfb1-8dc3451ef61c-kube-api-access-bs8ss\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.034174 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.034179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2864c883-0507-46b5-bfb1-8dc3451ef61c-image-registry-private-configuration\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.034433 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.034202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2864c883-0507-46b5-bfb1-8dc3451ef61c-registry-certificates\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.034433 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.034295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2864c883-0507-46b5-bfb1-8dc3451ef61c-bound-sa-token\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.034433 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.034353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2864c883-0507-46b5-bfb1-8dc3451ef61c-registry-tls\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.034433 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.034386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2864c883-0507-46b5-bfb1-8dc3451ef61c-ca-trust-extracted\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.034645 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.034436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2864c883-0507-46b5-bfb1-8dc3451ef61c-trusted-ca\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.034645 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.034466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2864c883-0507-46b5-bfb1-8dc3451ef61c-installation-pull-secrets\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.039345 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.035766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2864c883-0507-46b5-bfb1-8dc3451ef61c-ca-trust-extracted\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.039345 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.036258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2864c883-0507-46b5-bfb1-8dc3451ef61c-registry-certificates\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.039345 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.036525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2864c883-0507-46b5-bfb1-8dc3451ef61c-trusted-ca\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.039345 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.037733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2864c883-0507-46b5-bfb1-8dc3451ef61c-installation-pull-secrets\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.039345 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.038174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2864c883-0507-46b5-bfb1-8dc3451ef61c-registry-tls\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.040600 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.040097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2864c883-0507-46b5-bfb1-8dc3451ef61c-image-registry-private-configuration\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.042841 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.042818    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-8dmg4"
Jul 27 19:18:57.050501 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.050480    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c8j6s"
Jul 27 19:18:57.055814 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.055792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs8ss\" (UniqueName: \"kubernetes.io/projected/2864c883-0507-46b5-bfb1-8dc3451ef61c-kube-api-access-bs8ss\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.055948 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.055928    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:18:57.064119 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.064094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2864c883-0507-46b5-bfb1-8dc3451ef61c-bound-sa-token\") pod \"image-registry-fcd5949f6-b67bh\" (UID: \"2864c883-0507-46b5-bfb1-8dc3451ef61c\") " pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:57.156718 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:57.156623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:18:59.660887 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:59.660837    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c8j6s"]
Jul 27 19:18:59.664830 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:18:59.664377    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04b8dd0_d698_46e7_b992_9cb211e8ff1c.slice/crio-a1e49a4f207f35ba3c78565df1e81c37a31d3fcaf14011dae9dcdcba2974719d WatchSource:0}: Error finding container a1e49a4f207f35ba3c78565df1e81c37a31d3fcaf14011dae9dcdcba2974719d: Status 404 returned error can't find the container with id a1e49a4f207f35ba3c78565df1e81c37a31d3fcaf14011dae9dcdcba2974719d
Jul 27 19:18:59.713497 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:59.713423    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68b5695799-lm7c7"]
Jul 27 19:18:59.716392 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:18:59.716368    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0588b6c6_5465_462c_a02f_5c72896bb9db.slice/crio-7a95b03a9ddfbab1473269b583b0001dbe7e141089b987106657780cb2353ec4 WatchSource:0}: Error finding container 7a95b03a9ddfbab1473269b583b0001dbe7e141089b987106657780cb2353ec4: Status 404 returned error can't find the container with id 7a95b03a9ddfbab1473269b583b0001dbe7e141089b987106657780cb2353ec4
Jul 27 19:18:59.880168 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:59.880113    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"]
Jul 27 19:18:59.883800 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:18:59.883774    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867cd078_fd5f_49cd_b210_5d3e29127e73.slice/crio-35977f1e5d2079d0bc0d5ad8bb420d943aeefec077a9208fdc9a6be4aecdc391 WatchSource:0}: Error finding container 35977f1e5d2079d0bc0d5ad8bb420d943aeefec077a9208fdc9a6be4aecdc391: Status 404 returned error can't find the container with id 35977f1e5d2079d0bc0d5ad8bb420d943aeefec077a9208fdc9a6be4aecdc391
Jul 27 19:18:59.883888 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:59.883811    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-8dmg4"]
Jul 27 19:18:59.886537 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:18:59.886515    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70705846_cc85_405d_b05a_ada57468d4ad.slice/crio-112d6e8aff8df0045d75e79734d2874c3100063beb0d4ba8c0fa509384518d15 WatchSource:0}: Error finding container 112d6e8aff8df0045d75e79734d2874c3100063beb0d4ba8c0fa509384518d15: Status 404 returned error can't find the container with id 112d6e8aff8df0045d75e79734d2874c3100063beb0d4ba8c0fa509384518d15
Jul 27 19:18:59.893133 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:18:59.893091    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fcd5949f6-b67bh"]
Jul 27 19:18:59.896717 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:18:59.896691    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2864c883_0507_46b5_bfb1_8dc3451ef61c.slice/crio-5c1228bf1deb811b102304b3388b5320a8fd7587408479b69d3dfc690c0dfd21 WatchSource:0}: Error finding container 5c1228bf1deb811b102304b3388b5320a8fd7587408479b69d3dfc690c0dfd21: Status 404 returned error can't find the container with id 5c1228bf1deb811b102304b3388b5320a8fd7587408479b69d3dfc690c0dfd21
Jul 27 19:19:00.365235 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:00.359374    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lbsxn"
Jul 27 19:19:00.399863 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:00.399758    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b5695799-lm7c7" event={"ID":"0588b6c6-5465-462c-a02f-5c72896bb9db","Type":"ContainerStarted","Data":"7a95b03a9ddfbab1473269b583b0001dbe7e141089b987106657780cb2353ec4"}
Jul 27 19:19:00.402719 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:00.402640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9bx4h" event={"ID":"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4","Type":"ContainerStarted","Data":"507dfd4f0d37615c1a772e921218cbb5e139910610038ab1a5eb566112635448"}
Jul 27 19:19:00.406038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:00.405979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px" event={"ID":"867cd078-fd5f-49cd-b210-5d3e29127e73","Type":"ContainerStarted","Data":"35977f1e5d2079d0bc0d5ad8bb420d943aeefec077a9208fdc9a6be4aecdc391"}
Jul 27 19:19:00.412452 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:00.412376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c8j6s" event={"ID":"e04b8dd0-d698-46e7-b992-9cb211e8ff1c","Type":"ContainerStarted","Data":"a1e49a4f207f35ba3c78565df1e81c37a31d3fcaf14011dae9dcdcba2974719d"}
Jul 27 19:19:00.415104 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:00.415027    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-8dmg4" event={"ID":"70705846-cc85-405d-b05a-ada57468d4ad","Type":"ContainerStarted","Data":"112d6e8aff8df0045d75e79734d2874c3100063beb0d4ba8c0fa509384518d15"}
Jul 27 19:19:00.418178 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:00.417337    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fcd5949f6-b67bh" event={"ID":"2864c883-0507-46b5-bfb1-8dc3451ef61c","Type":"ContainerStarted","Data":"478b33eb042c9918a2d82f15ed5884c0e79c45c177bef6c75c8eb0d615a155f3"}
Jul 27 19:19:00.418178 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:00.417373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fcd5949f6-b67bh" event={"ID":"2864c883-0507-46b5-bfb1-8dc3451ef61c","Type":"ContainerStarted","Data":"5c1228bf1deb811b102304b3388b5320a8fd7587408479b69d3dfc690c0dfd21"}
Jul 27 19:19:00.418178 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:00.418125    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:19:00.439799 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:00.438628    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fcd5949f6-b67bh" podStartSLOduration=4.438610269 podStartE2EDuration="4.438610269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:00.437919507 +0000 UTC m=+82.953624012" watchObservedRunningTime="2026-07-27 19:19:00.438610269 +0000 UTC m=+82.954314772"
Jul 27 19:19:02.435871 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:02.435835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c8j6s" event={"ID":"e04b8dd0-d698-46e7-b992-9cb211e8ff1c","Type":"ContainerStarted","Data":"0952fb3c56de685ff1d823e9c92d9389ad2bc1d505f1fdf8693cf8ebb1ca65a3"}
Jul 27 19:19:02.436234 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:02.436080    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c8j6s"
Jul 27 19:19:02.442426 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:02.442401    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c8j6s"
Jul 27 19:19:02.459115 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:02.458981    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c8j6s" podStartSLOduration=4.663469915 podStartE2EDuration="6.458960356s" totalImagesPullingTime="1.795490441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:02.457427931 +0000 UTC m=+84.973132437" watchObservedRunningTime="2026-07-27 19:19:02.458960356 +0000 UTC m=+84.974664856"
Jul 27 19:19:03.290138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.290104    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-xnv97"]
Jul 27 19:19:03.295182 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.295158    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:03.297857 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.297579    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 27 19:19:03.298194 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.298173    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 19:19:03.299234 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.298637    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 27 19:19:03.299234 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.298790    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 19:19:03.299234 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.298836    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wtk99\""
Jul 27 19:19:03.299234 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.299049    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 19:19:03.315037 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.315009    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-xnv97"]
Jul 27 19:19:03.393995 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.393952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpcpj\" (UniqueName: \"kubernetes.io/projected/109f3c2b-5122-44fe-8d17-63e0ffc51857-kube-api-access-bpcpj\") pod \"prometheus-operator-84d69f794d-xnv97\" (UID: \"109f3c2b-5122-44fe-8d17-63e0ffc51857\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:03.394178 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.394011    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/109f3c2b-5122-44fe-8d17-63e0ffc51857-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-xnv97\" (UID: \"109f3c2b-5122-44fe-8d17-63e0ffc51857\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:03.394178 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.394064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/109f3c2b-5122-44fe-8d17-63e0ffc51857-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-xnv97\" (UID: \"109f3c2b-5122-44fe-8d17-63e0ffc51857\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:03.394178 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.394093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/109f3c2b-5122-44fe-8d17-63e0ffc51857-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-xnv97\" (UID: \"109f3c2b-5122-44fe-8d17-63e0ffc51857\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:03.495785 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.494698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpcpj\" (UniqueName: \"kubernetes.io/projected/109f3c2b-5122-44fe-8d17-63e0ffc51857-kube-api-access-bpcpj\") pod \"prometheus-operator-84d69f794d-xnv97\" (UID: \"109f3c2b-5122-44fe-8d17-63e0ffc51857\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:03.495785 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.494752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/109f3c2b-5122-44fe-8d17-63e0ffc51857-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-xnv97\" (UID: \"109f3c2b-5122-44fe-8d17-63e0ffc51857\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:03.495785 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.494820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/109f3c2b-5122-44fe-8d17-63e0ffc51857-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-xnv97\" (UID: \"109f3c2b-5122-44fe-8d17-63e0ffc51857\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:03.495785 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.494849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/109f3c2b-5122-44fe-8d17-63e0ffc51857-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-xnv97\" (UID: \"109f3c2b-5122-44fe-8d17-63e0ffc51857\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:03.495785 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:19:03.495398    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 27 19:19:03.495785 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:19:03.495465    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/109f3c2b-5122-44fe-8d17-63e0ffc51857-prometheus-operator-tls podName:109f3c2b-5122-44fe-8d17-63e0ffc51857 nodeName:}" failed. No retries permitted until 2026-07-27 19:19:03.995443136 +0000 UTC m=+86.511147620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/109f3c2b-5122-44fe-8d17-63e0ffc51857-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-xnv97" (UID: "109f3c2b-5122-44fe-8d17-63e0ffc51857") : secret "prometheus-operator-tls" not found
Jul 27 19:19:03.496528 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.496324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/109f3c2b-5122-44fe-8d17-63e0ffc51857-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-xnv97\" (UID: \"109f3c2b-5122-44fe-8d17-63e0ffc51857\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:03.500903 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.500877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/109f3c2b-5122-44fe-8d17-63e0ffc51857-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-xnv97\" (UID: \"109f3c2b-5122-44fe-8d17-63e0ffc51857\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:03.516700 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.516656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpcpj\" (UniqueName: \"kubernetes.io/projected/109f3c2b-5122-44fe-8d17-63e0ffc51857-kube-api-access-bpcpj\") pod \"prometheus-operator-84d69f794d-xnv97\" (UID: \"109f3c2b-5122-44fe-8d17-63e0ffc51857\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:03.998738 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:03.998696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/109f3c2b-5122-44fe-8d17-63e0ffc51857-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-xnv97\" (UID: \"109f3c2b-5122-44fe-8d17-63e0ffc51857\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:04.003723 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:04.003692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/109f3c2b-5122-44fe-8d17-63e0ffc51857-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-xnv97\" (UID: \"109f3c2b-5122-44fe-8d17-63e0ffc51857\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:04.208612 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:04.208573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97"
Jul 27 19:19:06.702430 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:06.702399    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:19:08.564032 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:08.563936    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-xnv97"]
Jul 27 19:19:08.570467 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:19:08.570442    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109f3c2b_5122_44fe_8d17_63e0ffc51857.slice/crio-6e0d78ed1fe34999d36ef332e94a45ee588d57ad6723c48603c549bda5937709 WatchSource:0}: Error finding container 6e0d78ed1fe34999d36ef332e94a45ee588d57ad6723c48603c549bda5937709: Status 404 returned error can't find the container with id 6e0d78ed1fe34999d36ef332e94a45ee588d57ad6723c48603c549bda5937709
Jul 27 19:19:09.469672 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:09.469504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9bx4h" event={"ID":"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4","Type":"ContainerStarted","Data":"d26a9fe889145e8586e19b1e40ba1da84f081792cf7a868e3f0f844fb24cdd00"}
Jul 27 19:19:09.471471 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:09.471442    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px" event={"ID":"867cd078-fd5f-49cd-b210-5d3e29127e73","Type":"ContainerStarted","Data":"4f9fb6c0a05982b1bf86c99d67d868d5dbc5dcf63d942fe92c909f6674f5a857"}
Jul 27 19:19:09.472869 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:09.471908    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"
Jul 27 19:19:09.473886 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:09.473863    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px"
Jul 27 19:19:09.474541 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:09.474514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b5695799-lm7c7" event={"ID":"0588b6c6-5465-462c-a02f-5c72896bb9db","Type":"ContainerStarted","Data":"e668ee8f8cb9a6a71408344ec5c736c1115e26b924d857eaa8e2ddd2afbcec13"}
Jul 27 19:19:09.478442 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:09.478383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97" event={"ID":"109f3c2b-5122-44fe-8d17-63e0ffc51857","Type":"ContainerStarted","Data":"6e0d78ed1fe34999d36ef332e94a45ee588d57ad6723c48603c549bda5937709"}
Jul 27 19:19:09.498795 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:09.498525    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65bdb645bc-tj6px" podStartSLOduration=5.000771203 podStartE2EDuration="13.498504568s" totalImagesPullingTime="8.497733365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:09.497076811 +0000 UTC m=+92.012781316" watchObservedRunningTime="2026-07-27 19:19:09.498504568 +0000 UTC m=+92.014209074"
Jul 27 19:19:09.518609 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:09.518556    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68b5695799-lm7c7" podStartSLOduration=4.853002258 podStartE2EDuration="13.518538756s" totalImagesPullingTime="8.665536498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:09.51734284 +0000 UTC m=+92.033047347" watchObservedRunningTime="2026-07-27 19:19:09.518538756 +0000 UTC m=+92.034243259"
Jul 27 19:19:10.485192 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:10.484782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97" event={"ID":"109f3c2b-5122-44fe-8d17-63e0ffc51857","Type":"ContainerStarted","Data":"65236e6c6d78d6c9d64c23a72c335d0169ca4e077326b12822f8e54eaa5ba9a1"}
Jul 27 19:19:10.485192 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:10.484843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97" event={"ID":"109f3c2b-5122-44fe-8d17-63e0ffc51857","Type":"ContainerStarted","Data":"2dbdb071b015bf2978656935947d8691fa14f8645d6bf27fe60e12d0aa699b82"}
Jul 27 19:19:10.506480 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:10.506226    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-xnv97" podStartSLOduration=6.003047619 podStartE2EDuration="7.506205249s" totalImagesPullingTime="1.50315763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:10.504112812 +0000 UTC m=+93.019817318" watchObservedRunningTime="2026-07-27 19:19:10.506205249 +0000 UTC m=+93.021909754"
Jul 27 19:19:12.767657 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.767615    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wh48w"]
Jul 27 19:19:12.774693 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.774664    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.777452 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.777428    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 19:19:12.777574 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.777438    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 19:19:12.777881 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.777861    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 19:19:12.778071 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.778036    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4mgvm\""
Jul 27 19:19:12.878399 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.878365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77479c54-bff3-4d49-a067-26c7be1d304c-sys\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.878576 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.878408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-tls\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.878576 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.878476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-textfile\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.878576 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.878527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.878576 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.878559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77479c54-bff3-4d49-a067-26c7be1d304c-metrics-client-ca\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.878777 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.878585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtmt8\" (UniqueName: \"kubernetes.io/projected/77479c54-bff3-4d49-a067-26c7be1d304c-kube-api-access-mtmt8\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.878777 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.878646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.878777 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.878691    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77479c54-bff3-4d49-a067-26c7be1d304c-root\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.878777 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.878719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-wtmp\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.980787 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.980124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.980787 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.980171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77479c54-bff3-4d49-a067-26c7be1d304c-metrics-client-ca\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.980787 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.980199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtmt8\" (UniqueName: \"kubernetes.io/projected/77479c54-bff3-4d49-a067-26c7be1d304c-kube-api-access-mtmt8\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.980787 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.980262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.980787 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.980301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77479c54-bff3-4d49-a067-26c7be1d304c-root\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.980787 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.980328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-wtmp\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.980787 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.980372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77479c54-bff3-4d49-a067-26c7be1d304c-sys\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.980787 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.980393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-tls\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.980787 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.980414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-textfile\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.981415 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.980890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77479c54-bff3-4d49-a067-26c7be1d304c-metrics-client-ca\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.981415 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.981117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-textfile\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.981415 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.981409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-wtmp\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.981561 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.981452    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77479c54-bff3-4d49-a067-26c7be1d304c-sys\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.981561 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:19:12.981531    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 19:19:12.981658 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:19:12.981585    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-tls podName:77479c54-bff3-4d49-a067-26c7be1d304c nodeName:}" failed. No retries permitted until 2026-07-27 19:19:13.4815674 +0000 UTC m=+95.997271882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-tls") pod "node-exporter-wh48w" (UID: "77479c54-bff3-4d49-a067-26c7be1d304c") : secret "node-exporter-tls" not found
Jul 27 19:19:12.981880 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.981848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77479c54-bff3-4d49-a067-26c7be1d304c-root\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.981880 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.981848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.984746 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.984704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:12.992705 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:12.992659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtmt8\" (UniqueName: \"kubernetes.io/projected/77479c54-bff3-4d49-a067-26c7be1d304c-kube-api-access-mtmt8\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:13.484938 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:13.484907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-tls\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:13.487595 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:13.487548    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77479c54-bff3-4d49-a067-26c7be1d304c-node-exporter-tls\") pod \"node-exporter-wh48w\" (UID: \"77479c54-bff3-4d49-a067-26c7be1d304c\") " pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:13.687207 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:13.687176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wh48w"
Jul 27 19:19:17.056513 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.056482    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:19:17.057044 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.056895    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:19:17.062494 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.062468    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:19:17.105176 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.105145    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"]
Jul 27 19:19:17.320601 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.320514    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"]
Jul 27 19:19:17.320775 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.320653    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.323381 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.323178    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4odjq2t3sjok2\""
Jul 27 19:19:17.323381 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.323211    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 19:19:17.323381 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.323224    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 27 19:19:17.323381 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.323236    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 27 19:19:17.323381 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.323274    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-w7mz6\""
Jul 27 19:19:17.323381 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.323182    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 27 19:19:17.422882 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.422839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eff50f1b-297b-451f-9f18-8a76669dd4d4-metrics-server-audit-profiles\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.423066 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.422896    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eff50f1b-297b-451f-9f18-8a76669dd4d4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.423066 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.422925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eff50f1b-297b-451f-9f18-8a76669dd4d4-client-ca-bundle\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.423066 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.423040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eff50f1b-297b-451f-9f18-8a76669dd4d4-secret-metrics-server-client-certs\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.423208 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.423077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eff50f1b-297b-451f-9f18-8a76669dd4d4-secret-metrics-server-tls\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.423208 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.423105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnfcq\" (UniqueName: \"kubernetes.io/projected/eff50f1b-297b-451f-9f18-8a76669dd4d4-kube-api-access-fnfcq\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.423208 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.423124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eff50f1b-297b-451f-9f18-8a76669dd4d4-audit-log\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.517781 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.517752    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:19:17.520854 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.520828    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q"]
Jul 27 19:19:17.523645 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.523621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eff50f1b-297b-451f-9f18-8a76669dd4d4-secret-metrics-server-client-certs\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.523757 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.523654    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eff50f1b-297b-451f-9f18-8a76669dd4d4-secret-metrics-server-tls\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.523757 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.523685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnfcq\" (UniqueName: \"kubernetes.io/projected/eff50f1b-297b-451f-9f18-8a76669dd4d4-kube-api-access-fnfcq\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.523757 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.523713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eff50f1b-297b-451f-9f18-8a76669dd4d4-audit-log\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.523757 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.523753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eff50f1b-297b-451f-9f18-8a76669dd4d4-metrics-server-audit-profiles\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.523950 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.523778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eff50f1b-297b-451f-9f18-8a76669dd4d4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.523950 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.523796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eff50f1b-297b-451f-9f18-8a76669dd4d4-client-ca-bundle\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.524398 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.524351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eff50f1b-297b-451f-9f18-8a76669dd4d4-audit-log\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.524967 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.524920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eff50f1b-297b-451f-9f18-8a76669dd4d4-metrics-server-audit-profiles\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.525716 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.525673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eff50f1b-297b-451f-9f18-8a76669dd4d4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.527486 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.526867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eff50f1b-297b-451f-9f18-8a76669dd4d4-secret-metrics-server-client-certs\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.527486 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.526896    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eff50f1b-297b-451f-9f18-8a76669dd4d4-client-ca-bundle\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.528232 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.528074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eff50f1b-297b-451f-9f18-8a76669dd4d4-secret-metrics-server-tls\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.543525 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.543443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnfcq\" (UniqueName: \"kubernetes.io/projected/eff50f1b-297b-451f-9f18-8a76669dd4d4-kube-api-access-fnfcq\") pod \"metrics-server-7cc967c9d4-7sqn2\" (UID: \"eff50f1b-297b-451f-9f18-8a76669dd4d4\") " pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.557331 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.557270    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q"]
Jul 27 19:19:17.557483 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.557406    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q"
Jul 27 19:19:17.559840 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.559780    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 27 19:19:17.560299 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.560279    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-kmjbq\""
Jul 27 19:19:17.633149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.633048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:17.726133 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.726088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2486k\" (UniqueName: \"kubernetes.io/projected/dd83cc2e-4ca8-418c-8b3a-e3de88c57df7-kube-api-access-2486k\") pod \"monitoring-plugin-68b7fc9574-snj9q\" (UID: \"dd83cc2e-4ca8-418c-8b3a-e3de88c57df7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q"
Jul 27 19:19:17.726133 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.726134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd83cc2e-4ca8-418c-8b3a-e3de88c57df7-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-snj9q\" (UID: \"dd83cc2e-4ca8-418c-8b3a-e3de88c57df7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q"
Jul 27 19:19:17.827097 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.827054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2486k\" (UniqueName: \"kubernetes.io/projected/dd83cc2e-4ca8-418c-8b3a-e3de88c57df7-kube-api-access-2486k\") pod \"monitoring-plugin-68b7fc9574-snj9q\" (UID: \"dd83cc2e-4ca8-418c-8b3a-e3de88c57df7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q"
Jul 27 19:19:17.827309 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.827111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd83cc2e-4ca8-418c-8b3a-e3de88c57df7-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-snj9q\" (UID: \"dd83cc2e-4ca8-418c-8b3a-e3de88c57df7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q"
Jul 27 19:19:17.829802 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.829776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd83cc2e-4ca8-418c-8b3a-e3de88c57df7-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-snj9q\" (UID: \"dd83cc2e-4ca8-418c-8b3a-e3de88c57df7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q"
Jul 27 19:19:17.847874 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.847834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2486k\" (UniqueName: \"kubernetes.io/projected/dd83cc2e-4ca8-418c-8b3a-e3de88c57df7-kube-api-access-2486k\") pod \"monitoring-plugin-68b7fc9574-snj9q\" (UID: \"dd83cc2e-4ca8-418c-8b3a-e3de88c57df7\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q"
Jul 27 19:19:17.870742 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:17.870706    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q"
Jul 27 19:19:20.404900 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:19:20.404793    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77479c54_bff3_4d49_a067_26c7be1d304c.slice/crio-d5041308fc4681759c70c4e41b5b66b5751c508e0ea5c8a2dc8971034f7ef774 WatchSource:0}: Error finding container d5041308fc4681759c70c4e41b5b66b5751c508e0ea5c8a2dc8971034f7ef774: Status 404 returned error can't find the container with id d5041308fc4681759c70c4e41b5b66b5751c508e0ea5c8a2dc8971034f7ef774
Jul 27 19:19:20.527981 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:20.527897    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wh48w" event={"ID":"77479c54-bff3-4d49-a067-26c7be1d304c","Type":"ContainerStarted","Data":"d5041308fc4681759c70c4e41b5b66b5751c508e0ea5c8a2dc8971034f7ef774"}
Jul 27 19:19:20.546301 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:20.546266    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"]
Jul 27 19:19:20.549975 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:19:20.549942    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff50f1b_297b_451f_9f18_8a76669dd4d4.slice/crio-b94306df07c95c47b4c71d2b89fa8b9edb9627a7fa729173f0d7251b0b191050 WatchSource:0}: Error finding container b94306df07c95c47b4c71d2b89fa8b9edb9627a7fa729173f0d7251b0b191050: Status 404 returned error can't find the container with id b94306df07c95c47b4c71d2b89fa8b9edb9627a7fa729173f0d7251b0b191050
Jul 27 19:19:20.564439 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:20.564416    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q"]
Jul 27 19:19:20.567140 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:19:20.567113    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd83cc2e_4ca8_418c_8b3a_e3de88c57df7.slice/crio-4bf930a725efd080efa809fe5c8e98e7686f8972239032dbd3a991b2fa442679 WatchSource:0}: Error finding container 4bf930a725efd080efa809fe5c8e98e7686f8972239032dbd3a991b2fa442679: Status 404 returned error can't find the container with id 4bf930a725efd080efa809fe5c8e98e7686f8972239032dbd3a991b2fa442679
Jul 27 19:19:21.360175 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:21.359946    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7vhjf"
Jul 27 19:19:21.538282 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:21.538198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2" event={"ID":"eff50f1b-297b-451f-9f18-8a76669dd4d4","Type":"ContainerStarted","Data":"b94306df07c95c47b4c71d2b89fa8b9edb9627a7fa729173f0d7251b0b191050"}
Jul 27 19:19:21.542706 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:21.542186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9bx4h" event={"ID":"3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4","Type":"ContainerStarted","Data":"b5a3c95d31f21574073674a549d24117024190775f42ad745fc79e4706b96c93"}
Jul 27 19:19:21.546197 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:21.546175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q" event={"ID":"dd83cc2e-4ca8-418c-8b3a-e3de88c57df7","Type":"ContainerStarted","Data":"4bf930a725efd080efa809fe5c8e98e7686f8972239032dbd3a991b2fa442679"}
Jul 27 19:19:21.549500 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:21.549480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-8dmg4" event={"ID":"70705846-cc85-405d-b05a-ada57468d4ad","Type":"ContainerStarted","Data":"00a931e03ff5303b7278b8c12af49c6edd6f19c7b31485c76e831afb8c72a6c2"}
Jul 27 19:19:21.550263 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:21.550178    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-8dmg4"
Jul 27 19:19:21.563011 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:21.562978    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-8dmg4"
Jul 27 19:19:21.569190 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:21.569139    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9bx4h" podStartSLOduration=34.603625248 podStartE2EDuration="1m0.569122265s" totalImagesPullingTime="25.965497017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:18:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:21.568135249 +0000 UTC m=+104.083839759" watchObservedRunningTime="2026-07-27 19:19:21.569122265 +0000 UTC m=+104.084826775"
Jul 27 19:19:21.590370 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:21.589103    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-8dmg4" podStartSLOduration=5.004262864 podStartE2EDuration="25.589083744s" totalImagesPullingTime="20.58482088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:21.587286194 +0000 UTC m=+104.102990700" watchObservedRunningTime="2026-07-27 19:19:21.589083744 +0000 UTC m=+104.104788250"
Jul 27 19:19:21.715489 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:21.715412    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr" podUID="be513394-5a1a-435b-87c2-348762f04fe6" containerName="registry" containerID="cri-o://6c791d009c18a0e8b9239e87199d7c4c3b5313f301c57a11f4110eb20af0db81" gracePeriod=30
Jul 27 19:19:22.442874 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:22.442790    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fcd5949f6-b67bh"
Jul 27 19:19:22.560036 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:22.558623    2580 generic.go:356] "Generic (PLEG): container finished" podID="be513394-5a1a-435b-87c2-348762f04fe6" containerID="6c791d009c18a0e8b9239e87199d7c4c3b5313f301c57a11f4110eb20af0db81" exitCode=0
Jul 27 19:19:22.560036 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:22.559424    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr" event={"ID":"be513394-5a1a-435b-87c2-348762f04fe6","Type":"ContainerDied","Data":"6c791d009c18a0e8b9239e87199d7c4c3b5313f301c57a11f4110eb20af0db81"}
Jul 27 19:19:22.929576 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:22.929546    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:19:23.078877 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.078842    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be513394-5a1a-435b-87c2-348762f04fe6-image-registry-private-configuration\") pod \"be513394-5a1a-435b-87c2-348762f04fe6\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") "
Jul 27 19:19:23.078877 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.078888    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be513394-5a1a-435b-87c2-348762f04fe6-registry-certificates\") pod \"be513394-5a1a-435b-87c2-348762f04fe6\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") "
Jul 27 19:19:23.079108 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.078925    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vr6v\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-kube-api-access-9vr6v\") pod \"be513394-5a1a-435b-87c2-348762f04fe6\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") "
Jul 27 19:19:23.079108 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.078961    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls\") pod \"be513394-5a1a-435b-87c2-348762f04fe6\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") "
Jul 27 19:19:23.079108 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.079008    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be513394-5a1a-435b-87c2-348762f04fe6-installation-pull-secrets\") pod \"be513394-5a1a-435b-87c2-348762f04fe6\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") "
Jul 27 19:19:23.079320 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.079289    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-bound-sa-token\") pod \"be513394-5a1a-435b-87c2-348762f04fe6\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") "
Jul 27 19:19:23.079396 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.079328    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be513394-5a1a-435b-87c2-348762f04fe6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "be513394-5a1a-435b-87c2-348762f04fe6" (UID: "be513394-5a1a-435b-87c2-348762f04fe6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:19:23.079396 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.079369    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be513394-5a1a-435b-87c2-348762f04fe6-trusted-ca\") pod \"be513394-5a1a-435b-87c2-348762f04fe6\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") "
Jul 27 19:19:23.079497 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.079399    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be513394-5a1a-435b-87c2-348762f04fe6-ca-trust-extracted\") pod \"be513394-5a1a-435b-87c2-348762f04fe6\" (UID: \"be513394-5a1a-435b-87c2-348762f04fe6\") "
Jul 27 19:19:23.079713 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.079686    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be513394-5a1a-435b-87c2-348762f04fe6-registry-certificates\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:23.080321 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.080266    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be513394-5a1a-435b-87c2-348762f04fe6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "be513394-5a1a-435b-87c2-348762f04fe6" (UID: "be513394-5a1a-435b-87c2-348762f04fe6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:19:23.082088 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.082046    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-kube-api-access-9vr6v" (OuterVolumeSpecName: "kube-api-access-9vr6v") pod "be513394-5a1a-435b-87c2-348762f04fe6" (UID: "be513394-5a1a-435b-87c2-348762f04fe6"). InnerVolumeSpecName "kube-api-access-9vr6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:19:23.082182 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.082103    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be513394-5a1a-435b-87c2-348762f04fe6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "be513394-5a1a-435b-87c2-348762f04fe6" (UID: "be513394-5a1a-435b-87c2-348762f04fe6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:19:23.082624 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.082308    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "be513394-5a1a-435b-87c2-348762f04fe6" (UID: "be513394-5a1a-435b-87c2-348762f04fe6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:19:23.082624 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.082564    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be513394-5a1a-435b-87c2-348762f04fe6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "be513394-5a1a-435b-87c2-348762f04fe6" (UID: "be513394-5a1a-435b-87c2-348762f04fe6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:19:23.083449 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.083425    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "be513394-5a1a-435b-87c2-348762f04fe6" (UID: "be513394-5a1a-435b-87c2-348762f04fe6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:19:23.091823 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.091794    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be513394-5a1a-435b-87c2-348762f04fe6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "be513394-5a1a-435b-87c2-348762f04fe6" (UID: "be513394-5a1a-435b-87c2-348762f04fe6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:19:23.181121 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.181050    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be513394-5a1a-435b-87c2-348762f04fe6-installation-pull-secrets\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:23.181347 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.181310    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-bound-sa-token\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:23.181347 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.181343    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be513394-5a1a-435b-87c2-348762f04fe6-trusted-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:23.181347 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.181357    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be513394-5a1a-435b-87c2-348762f04fe6-ca-trust-extracted\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:23.181627 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.181372    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be513394-5a1a-435b-87c2-348762f04fe6-image-registry-private-configuration\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:23.181627 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.181389    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vr6v\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-kube-api-access-9vr6v\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:23.181627 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.181408    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be513394-5a1a-435b-87c2-348762f04fe6-registry-tls\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:23.564183 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.564152    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"
Jul 27 19:19:23.564183 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.564179    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9b8fc8fd5-kbqcr" event={"ID":"be513394-5a1a-435b-87c2-348762f04fe6","Type":"ContainerDied","Data":"93b1433cea2c22fc653f81b4591900af41e21bece8bee7063dec6378048acf50"}
Jul 27 19:19:23.564740 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.564295    2580 scope.go:117] "RemoveContainer" containerID="6c791d009c18a0e8b9239e87199d7c4c3b5313f301c57a11f4110eb20af0db81"
Jul 27 19:19:23.588228 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.588200    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"]
Jul 27 19:19:23.592355 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:23.592329    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-9b8fc8fd5-kbqcr"]
Jul 27 19:19:24.086906 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:24.086874    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be513394-5a1a-435b-87c2-348762f04fe6" path="/var/lib/kubelet/pods/be513394-5a1a-435b-87c2-348762f04fe6/volumes"
Jul 27 19:19:24.570134 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:24.570095    2580 generic.go:356] "Generic (PLEG): container finished" podID="77479c54-bff3-4d49-a067-26c7be1d304c" containerID="bc3fbf99bea53de7b8789003704bd381702b81344543f2894755d2f17e78c232" exitCode=0
Jul 27 19:19:24.570632 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:24.570196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wh48w" event={"ID":"77479c54-bff3-4d49-a067-26c7be1d304c","Type":"ContainerDied","Data":"bc3fbf99bea53de7b8789003704bd381702b81344543f2894755d2f17e78c232"}
Jul 27 19:19:24.572347 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:24.572322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2" event={"ID":"eff50f1b-297b-451f-9f18-8a76669dd4d4","Type":"ContainerStarted","Data":"e0739f6a172675c8cb18812ad4a5800ef216acad94d7d9aadcbb8680ed23d82b"}
Jul 27 19:19:24.575118 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:24.575024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q" event={"ID":"dd83cc2e-4ca8-418c-8b3a-e3de88c57df7","Type":"ContainerStarted","Data":"f1a3d3271e13f7c34a94f9e6f6c58c6ea75fec71d5eb1c5d7f7df978af75b24c"}
Jul 27 19:19:24.575319 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:24.575262    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q"
Jul 27 19:19:24.581454 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:24.581434    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q"
Jul 27 19:19:24.611614 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:24.611566    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2" podStartSLOduration=4.171791947 podStartE2EDuration="7.611552678s" totalImagesPullingTime="3.439760731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:24.610152768 +0000 UTC m=+107.125857275" watchObservedRunningTime="2026-07-27 19:19:24.611552678 +0000 UTC m=+107.127257244"
Jul 27 19:19:24.626379 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:24.626340    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-snj9q" podStartSLOduration=4.512145884 podStartE2EDuration="7.626326994s" totalImagesPullingTime="3.11418111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:24.625050875 +0000 UTC m=+107.140755379" watchObservedRunningTime="2026-07-27 19:19:24.626326994 +0000 UTC m=+107.142031498"
Jul 27 19:19:25.582398 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:25.582357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wh48w" event={"ID":"77479c54-bff3-4d49-a067-26c7be1d304c","Type":"ContainerStarted","Data":"1231933e33f8e844f2d45077bdc6b9680a932110dc4519e1bae14343804f229f"}
Jul 27 19:19:25.582764 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:25.582422    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wh48w" event={"ID":"77479c54-bff3-4d49-a067-26c7be1d304c","Type":"ContainerStarted","Data":"8eb11fc4807ab17958bf763eeb7b678f9b32ba36e80525ea025447c9f36c406b"}
Jul 27 19:19:25.623499 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:25.623438    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wh48w" podStartSLOduration=10.228214303 podStartE2EDuration="13.623422465s" totalImagesPullingTime="3.272413704s" totalInitContainerRuntime="122.794458ms" isStatefulPod=true podCreationTimestamp="2026-07-27 19:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:25.622811122 +0000 UTC m=+108.138515629" watchObservedRunningTime="2026-07-27 19:19:25.623422465 +0000 UTC m=+108.139126969"
Jul 27 19:19:28.387820 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:28.387787    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68b5695799-lm7c7"]
Jul 27 19:19:37.633946 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:37.633915    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:37.634558 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:37.633989    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:53.409665 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.409603    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68b5695799-lm7c7" podUID="0588b6c6-5465-462c-a02f-5c72896bb9db" containerName="console" containerID="cri-o://e668ee8f8cb9a6a71408344ec5c736c1115e26b924d857eaa8e2ddd2afbcec13" gracePeriod=15
Jul 27 19:19:53.649004 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.648978    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:19:53.680118 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.680043    2580 generic.go:356] "Generic (PLEG): container finished" podID="0588b6c6-5465-462c-a02f-5c72896bb9db" containerID="e668ee8f8cb9a6a71408344ec5c736c1115e26b924d857eaa8e2ddd2afbcec13" exitCode=0
Jul 27 19:19:53.680118 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.680103    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b5695799-lm7c7"
Jul 27 19:19:53.680345 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.680134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b5695799-lm7c7" event={"ID":"0588b6c6-5465-462c-a02f-5c72896bb9db","Type":"ContainerDied","Data":"e668ee8f8cb9a6a71408344ec5c736c1115e26b924d857eaa8e2ddd2afbcec13"}
Jul 27 19:19:53.680345 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.680173    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b5695799-lm7c7" event={"ID":"0588b6c6-5465-462c-a02f-5c72896bb9db","Type":"ContainerDied","Data":"7a95b03a9ddfbab1473269b583b0001dbe7e141089b987106657780cb2353ec4"}
Jul 27 19:19:53.680345 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.680192    2580 scope.go:117] "RemoveContainer" containerID="e668ee8f8cb9a6a71408344ec5c736c1115e26b924d857eaa8e2ddd2afbcec13"
Jul 27 19:19:53.687843 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.687827    2580 scope.go:117] "RemoveContainer" containerID="e668ee8f8cb9a6a71408344ec5c736c1115e26b924d857eaa8e2ddd2afbcec13"
Jul 27 19:19:53.688117 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:19:53.688097    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e668ee8f8cb9a6a71408344ec5c736c1115e26b924d857eaa8e2ddd2afbcec13\": container with ID starting with e668ee8f8cb9a6a71408344ec5c736c1115e26b924d857eaa8e2ddd2afbcec13 not found: ID does not exist" containerID="e668ee8f8cb9a6a71408344ec5c736c1115e26b924d857eaa8e2ddd2afbcec13"
Jul 27 19:19:53.688180 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.688126    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e668ee8f8cb9a6a71408344ec5c736c1115e26b924d857eaa8e2ddd2afbcec13"} err="failed to get container status \"e668ee8f8cb9a6a71408344ec5c736c1115e26b924d857eaa8e2ddd2afbcec13\": rpc error: code = NotFound desc = could not find container \"e668ee8f8cb9a6a71408344ec5c736c1115e26b924d857eaa8e2ddd2afbcec13\": container with ID starting with e668ee8f8cb9a6a71408344ec5c736c1115e26b924d857eaa8e2ddd2afbcec13 not found: ID does not exist"
Jul 27 19:19:53.716851 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.716828    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-oauth-serving-cert\") pod \"0588b6c6-5465-462c-a02f-5c72896bb9db\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") "
Jul 27 19:19:53.716930 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.716856    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0588b6c6-5465-462c-a02f-5c72896bb9db-console-serving-cert\") pod \"0588b6c6-5465-462c-a02f-5c72896bb9db\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") "
Jul 27 19:19:53.716930 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.716874    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-service-ca\") pod \"0588b6c6-5465-462c-a02f-5c72896bb9db\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") "
Jul 27 19:19:53.717091 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.716948    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-trusted-ca-bundle\") pod \"0588b6c6-5465-462c-a02f-5c72896bb9db\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") "
Jul 27 19:19:53.717091 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.716988    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0588b6c6-5465-462c-a02f-5c72896bb9db-console-oauth-config\") pod \"0588b6c6-5465-462c-a02f-5c72896bb9db\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") "
Jul 27 19:19:53.717091 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.717019    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-console-config\") pod \"0588b6c6-5465-462c-a02f-5c72896bb9db\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") "
Jul 27 19:19:53.717091 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.717050    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck67g\" (UniqueName: \"kubernetes.io/projected/0588b6c6-5465-462c-a02f-5c72896bb9db-kube-api-access-ck67g\") pod \"0588b6c6-5465-462c-a02f-5c72896bb9db\" (UID: \"0588b6c6-5465-462c-a02f-5c72896bb9db\") "
Jul 27 19:19:53.717315 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.717159    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0588b6c6-5465-462c-a02f-5c72896bb9db" (UID: "0588b6c6-5465-462c-a02f-5c72896bb9db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:19:53.717387 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.717368    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-oauth-serving-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:53.717440 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.717388    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-service-ca" (OuterVolumeSpecName: "service-ca") pod "0588b6c6-5465-462c-a02f-5c72896bb9db" (UID: "0588b6c6-5465-462c-a02f-5c72896bb9db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:19:53.717440 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.717427    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0588b6c6-5465-462c-a02f-5c72896bb9db" (UID: "0588b6c6-5465-462c-a02f-5c72896bb9db"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:19:53.717516 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.717441    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-console-config" (OuterVolumeSpecName: "console-config") pod "0588b6c6-5465-462c-a02f-5c72896bb9db" (UID: "0588b6c6-5465-462c-a02f-5c72896bb9db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:19:53.718998 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.718975    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0588b6c6-5465-462c-a02f-5c72896bb9db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0588b6c6-5465-462c-a02f-5c72896bb9db" (UID: "0588b6c6-5465-462c-a02f-5c72896bb9db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:19:53.719086 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.719025    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0588b6c6-5465-462c-a02f-5c72896bb9db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0588b6c6-5465-462c-a02f-5c72896bb9db" (UID: "0588b6c6-5465-462c-a02f-5c72896bb9db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:19:53.719318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.719298    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0588b6c6-5465-462c-a02f-5c72896bb9db-kube-api-access-ck67g" (OuterVolumeSpecName: "kube-api-access-ck67g") pod "0588b6c6-5465-462c-a02f-5c72896bb9db" (UID: "0588b6c6-5465-462c-a02f-5c72896bb9db"). InnerVolumeSpecName "kube-api-access-ck67g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:19:53.818320 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.818288    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck67g\" (UniqueName: \"kubernetes.io/projected/0588b6c6-5465-462c-a02f-5c72896bb9db-kube-api-access-ck67g\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:53.818320 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.818314    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0588b6c6-5465-462c-a02f-5c72896bb9db-console-serving-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:53.818320 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.818327    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:53.818519 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.818339    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-trusted-ca-bundle\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:53.818519 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.818351    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0588b6c6-5465-462c-a02f-5c72896bb9db-console-oauth-config\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:53.818519 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:53.818362    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0588b6c6-5465-462c-a02f-5c72896bb9db-console-config\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:19:54.002731 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:54.002705    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68b5695799-lm7c7"]
Jul 27 19:19:54.004761 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:54.004741    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68b5695799-lm7c7"]
Jul 27 19:19:54.087455 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:54.087427    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0588b6c6-5465-462c-a02f-5c72896bb9db" path="/var/lib/kubelet/pods/0588b6c6-5465-462c-a02f-5c72896bb9db/volumes"
Jul 27 19:19:57.638530 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:57.638499    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:19:57.642316 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:19:57.642296    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7cc967c9d4-7sqn2"
Jul 27 19:20:49.878833 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.878797    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6868c98757-9kql8"]
Jul 27 19:20:49.879224 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.879035    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be513394-5a1a-435b-87c2-348762f04fe6" containerName="registry"
Jul 27 19:20:49.879224 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.879047    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="be513394-5a1a-435b-87c2-348762f04fe6" containerName="registry"
Jul 27 19:20:49.879224 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.879062    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0588b6c6-5465-462c-a02f-5c72896bb9db" containerName="console"
Jul 27 19:20:49.879224 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.879067    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0588b6c6-5465-462c-a02f-5c72896bb9db" containerName="console"
Jul 27 19:20:49.879224 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.879106    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0588b6c6-5465-462c-a02f-5c72896bb9db" containerName="console"
Jul 27 19:20:49.879224 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.879114    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="be513394-5a1a-435b-87c2-348762f04fe6" containerName="registry"
Jul 27 19:20:49.881552 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.881536    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:49.886046 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.886020    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bhwfx\""
Jul 27 19:20:49.886165 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.886020    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 19:20:49.886165 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.886022    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 19:20:49.886307 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.886022    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 19:20:49.886307 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.886192    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 19:20:49.886430 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.886417    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 19:20:49.892765 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.892748    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 19:20:49.917487 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.917459    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6868c98757-9kql8"]
Jul 27 19:20:49.935460 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.935441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5548a379-4118-4f51-8460-ea30bd54f951-console-serving-cert\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:49.935563 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.935470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-trusted-ca-bundle\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:49.935563 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.935503    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-console-config\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:49.935563 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.935546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-service-ca\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:49.935692 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.935576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5548a379-4118-4f51-8460-ea30bd54f951-console-oauth-config\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:49.935692 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.935615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxngw\" (UniqueName: \"kubernetes.io/projected/5548a379-4118-4f51-8460-ea30bd54f951-kube-api-access-rxngw\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:49.935692 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:49.935636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-oauth-serving-cert\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.036150 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.036121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-console-config\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.036150 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.036150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-service-ca\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.036381 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.036171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5548a379-4118-4f51-8460-ea30bd54f951-console-oauth-config\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.036381 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.036298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxngw\" (UniqueName: \"kubernetes.io/projected/5548a379-4118-4f51-8460-ea30bd54f951-kube-api-access-rxngw\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.036381 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.036332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-oauth-serving-cert\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.036532 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.036470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5548a379-4118-4f51-8460-ea30bd54f951-console-serving-cert\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.036532 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.036500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-trusted-ca-bundle\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.036902 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.036877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-service-ca\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.037086 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.037063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-oauth-serving-cert\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.037160 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.037102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-console-config\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.037413 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.037394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-trusted-ca-bundle\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.038614 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.038586    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5548a379-4118-4f51-8460-ea30bd54f951-console-oauth-config\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.038710 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.038700    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5548a379-4118-4f51-8460-ea30bd54f951-console-serving-cert\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.047421 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.047399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxngw\" (UniqueName: \"kubernetes.io/projected/5548a379-4118-4f51-8460-ea30bd54f951-kube-api-access-rxngw\") pod \"console-6868c98757-9kql8\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") " pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.191234 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.191167    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:20:50.317879 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.317857    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6868c98757-9kql8"]
Jul 27 19:20:50.320273 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:20:50.320252    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5548a379_4118_4f51_8460_ea30bd54f951.slice/crio-0c4906eca3039bd0504c414ea57c7b6aeefae083c5f33d4fdaea1b26172ff179 WatchSource:0}: Error finding container 0c4906eca3039bd0504c414ea57c7b6aeefae083c5f33d4fdaea1b26172ff179: Status 404 returned error can't find the container with id 0c4906eca3039bd0504c414ea57c7b6aeefae083c5f33d4fdaea1b26172ff179
Jul 27 19:20:50.880847 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.880809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6868c98757-9kql8" event={"ID":"5548a379-4118-4f51-8460-ea30bd54f951","Type":"ContainerStarted","Data":"55e8474dc21a496b6e333525e0aee248dceccf41a47ad82c96a20b6018251070"}
Jul 27 19:20:50.881219 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.880868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6868c98757-9kql8" event={"ID":"5548a379-4118-4f51-8460-ea30bd54f951","Type":"ContainerStarted","Data":"0c4906eca3039bd0504c414ea57c7b6aeefae083c5f33d4fdaea1b26172ff179"}
Jul 27 19:20:50.903190 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:20:50.903147    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6868c98757-9kql8" podStartSLOduration=1.9031324619999999 podStartE2EDuration="1.903132462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:20:50.90129879 +0000 UTC m=+193.417003294" watchObservedRunningTime="2026-07-27 19:20:50.903132462 +0000 UTC m=+193.418836965"
Jul 27 19:21:00.191815 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:00.191776    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:21:00.192319 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:00.191845    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:21:00.197316 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:00.197293    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:21:00.917358 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:00.917332    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:21:56.977995 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:56.977958    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"]
Jul 27 19:21:56.980488 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:56.980472    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"
Jul 27 19:21:56.983035 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:56.982992    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 19:21:56.983250 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:56.983228    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q5wxr\""
Jul 27 19:21:56.983957 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:56.983935    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 19:21:56.991501 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:56.991482    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"]
Jul 27 19:21:57.033167 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:57.033141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79bd5cad-9713-4180-9a05-9be0ecf50218-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh\" (UID: \"79bd5cad-9713-4180-9a05-9be0ecf50218\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"
Jul 27 19:21:57.033284 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:57.033178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79bd5cad-9713-4180-9a05-9be0ecf50218-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh\" (UID: \"79bd5cad-9713-4180-9a05-9be0ecf50218\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"
Jul 27 19:21:57.033284 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:57.033210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn6dr\" (UniqueName: \"kubernetes.io/projected/79bd5cad-9713-4180-9a05-9be0ecf50218-kube-api-access-nn6dr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh\" (UID: \"79bd5cad-9713-4180-9a05-9be0ecf50218\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"
Jul 27 19:21:57.134009 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:57.133979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79bd5cad-9713-4180-9a05-9be0ecf50218-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh\" (UID: \"79bd5cad-9713-4180-9a05-9be0ecf50218\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"
Jul 27 19:21:57.134009 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:57.134015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79bd5cad-9713-4180-9a05-9be0ecf50218-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh\" (UID: \"79bd5cad-9713-4180-9a05-9be0ecf50218\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"
Jul 27 19:21:57.134211 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:57.134042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn6dr\" (UniqueName: \"kubernetes.io/projected/79bd5cad-9713-4180-9a05-9be0ecf50218-kube-api-access-nn6dr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh\" (UID: \"79bd5cad-9713-4180-9a05-9be0ecf50218\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"
Jul 27 19:21:57.134400 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:57.134381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79bd5cad-9713-4180-9a05-9be0ecf50218-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh\" (UID: \"79bd5cad-9713-4180-9a05-9be0ecf50218\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"
Jul 27 19:21:57.134450 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:57.134431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79bd5cad-9713-4180-9a05-9be0ecf50218-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh\" (UID: \"79bd5cad-9713-4180-9a05-9be0ecf50218\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"
Jul 27 19:21:57.142143 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:57.142121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn6dr\" (UniqueName: \"kubernetes.io/projected/79bd5cad-9713-4180-9a05-9be0ecf50218-kube-api-access-nn6dr\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh\" (UID: \"79bd5cad-9713-4180-9a05-9be0ecf50218\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"
Jul 27 19:21:57.288991 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:57.288968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"
Jul 27 19:21:57.415767 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:57.415734    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"]
Jul 27 19:21:57.418900 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:21:57.418873    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bd5cad_9713_4180_9a05_9be0ecf50218.slice/crio-8cda7e92661b63989385c29179a546aa5872180f11a49b4ddfef827d1d51a6ce WatchSource:0}: Error finding container 8cda7e92661b63989385c29179a546aa5872180f11a49b4ddfef827d1d51a6ce: Status 404 returned error can't find the container with id 8cda7e92661b63989385c29179a546aa5872180f11a49b4ddfef827d1d51a6ce
Jul 27 19:21:58.107620 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:21:58.107578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh" event={"ID":"79bd5cad-9713-4180-9a05-9be0ecf50218","Type":"ContainerStarted","Data":"8cda7e92661b63989385c29179a546aa5872180f11a49b4ddfef827d1d51a6ce"}
Jul 27 19:22:04.131634 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:04.131598    2580 generic.go:356] "Generic (PLEG): container finished" podID="79bd5cad-9713-4180-9a05-9be0ecf50218" containerID="03c0fb5e5f0579c80596b860a41eb37cc3a5e8c0fdd5c97153419ffe7929a073" exitCode=0
Jul 27 19:22:04.131979 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:04.131676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh" event={"ID":"79bd5cad-9713-4180-9a05-9be0ecf50218","Type":"ContainerDied","Data":"03c0fb5e5f0579c80596b860a41eb37cc3a5e8c0fdd5c97153419ffe7929a073"}
Jul 27 19:22:06.139818 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:06.139785    2580 generic.go:356] "Generic (PLEG): container finished" podID="79bd5cad-9713-4180-9a05-9be0ecf50218" containerID="b52f517facbc82693cf3574a11d644eb6f16c02e856c8fdd1e033f55b08ed8bc" exitCode=0
Jul 27 19:22:06.140157 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:06.139873    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh" event={"ID":"79bd5cad-9713-4180-9a05-9be0ecf50218","Type":"ContainerDied","Data":"b52f517facbc82693cf3574a11d644eb6f16c02e856c8fdd1e033f55b08ed8bc"}
Jul 27 19:22:14.172768 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:14.172732    2580 generic.go:356] "Generic (PLEG): container finished" podID="79bd5cad-9713-4180-9a05-9be0ecf50218" containerID="6f372b6fade8e34adcff8170b7fcd7737f05b1d1b96c4d3e1c823416aa5c604e" exitCode=0
Jul 27 19:22:14.173147 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:14.172819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh" event={"ID":"79bd5cad-9713-4180-9a05-9be0ecf50218","Type":"ContainerDied","Data":"6f372b6fade8e34adcff8170b7fcd7737f05b1d1b96c4d3e1c823416aa5c604e"}
Jul 27 19:22:15.306750 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:15.306725    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"
Jul 27 19:22:15.370950 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:15.370921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79bd5cad-9713-4180-9a05-9be0ecf50218-util\") pod \"79bd5cad-9713-4180-9a05-9be0ecf50218\" (UID: \"79bd5cad-9713-4180-9a05-9be0ecf50218\") "
Jul 27 19:22:15.370950 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:15.370955    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79bd5cad-9713-4180-9a05-9be0ecf50218-bundle\") pod \"79bd5cad-9713-4180-9a05-9be0ecf50218\" (UID: \"79bd5cad-9713-4180-9a05-9be0ecf50218\") "
Jul 27 19:22:15.371155 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:15.371018    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn6dr\" (UniqueName: \"kubernetes.io/projected/79bd5cad-9713-4180-9a05-9be0ecf50218-kube-api-access-nn6dr\") pod \"79bd5cad-9713-4180-9a05-9be0ecf50218\" (UID: \"79bd5cad-9713-4180-9a05-9be0ecf50218\") "
Jul 27 19:22:15.371687 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:15.371660    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bd5cad-9713-4180-9a05-9be0ecf50218-bundle" (OuterVolumeSpecName: "bundle") pod "79bd5cad-9713-4180-9a05-9be0ecf50218" (UID: "79bd5cad-9713-4180-9a05-9be0ecf50218"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:22:15.373141 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:15.373113    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bd5cad-9713-4180-9a05-9be0ecf50218-kube-api-access-nn6dr" (OuterVolumeSpecName: "kube-api-access-nn6dr") pod "79bd5cad-9713-4180-9a05-9be0ecf50218" (UID: "79bd5cad-9713-4180-9a05-9be0ecf50218"). InnerVolumeSpecName "kube-api-access-nn6dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:22:15.376182 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:15.376158    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bd5cad-9713-4180-9a05-9be0ecf50218-util" (OuterVolumeSpecName: "util") pod "79bd5cad-9713-4180-9a05-9be0ecf50218" (UID: "79bd5cad-9713-4180-9a05-9be0ecf50218"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:22:15.471881 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:15.471804    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn6dr\" (UniqueName: \"kubernetes.io/projected/79bd5cad-9713-4180-9a05-9be0ecf50218-kube-api-access-nn6dr\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:22:15.471881 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:15.471829    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79bd5cad-9713-4180-9a05-9be0ecf50218-util\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:22:15.471881 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:15.471839    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79bd5cad-9713-4180-9a05-9be0ecf50218-bundle\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:22:16.183156 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:16.183125    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh" event={"ID":"79bd5cad-9713-4180-9a05-9be0ecf50218","Type":"ContainerDied","Data":"8cda7e92661b63989385c29179a546aa5872180f11a49b4ddfef827d1d51a6ce"}
Jul 27 19:22:16.183330 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:16.183172    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cda7e92661b63989385c29179a546aa5872180f11a49b4ddfef827d1d51a6ce"
Jul 27 19:22:16.183330 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:16.183135    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb7dh"
Jul 27 19:22:20.286548 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.286515    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xs5lx"]
Jul 27 19:22:20.286924 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.286773    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bd5cad-9713-4180-9a05-9be0ecf50218" containerName="extract"
Jul 27 19:22:20.286924 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.286784    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bd5cad-9713-4180-9a05-9be0ecf50218" containerName="extract"
Jul 27 19:22:20.286924 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.286799    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bd5cad-9713-4180-9a05-9be0ecf50218" containerName="pull"
Jul 27 19:22:20.286924 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.286808    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bd5cad-9713-4180-9a05-9be0ecf50218" containerName="pull"
Jul 27 19:22:20.286924 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.286821    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bd5cad-9713-4180-9a05-9be0ecf50218" containerName="util"
Jul 27 19:22:20.286924 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.286827    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bd5cad-9713-4180-9a05-9be0ecf50218" containerName="util"
Jul 27 19:22:20.286924 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.286892    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="79bd5cad-9713-4180-9a05-9be0ecf50218" containerName="extract"
Jul 27 19:22:20.325760 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.325729    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xs5lx"]
Jul 27 19:22:20.325934 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.325824    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xs5lx"
Jul 27 19:22:20.328229 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.328194    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 27 19:22:20.328403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.328254    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-q98n6\""
Jul 27 19:22:20.328403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.328259    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 27 19:22:20.410356 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.410314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbp5j\" (UniqueName: \"kubernetes.io/projected/62182245-9140-45d5-977d-a59e582616aa-kube-api-access-hbp5j\") pod \"cert-manager-operator-controller-manager-55c444bf86-xs5lx\" (UID: \"62182245-9140-45d5-977d-a59e582616aa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xs5lx"
Jul 27 19:22:20.410541 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.410401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62182245-9140-45d5-977d-a59e582616aa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xs5lx\" (UID: \"62182245-9140-45d5-977d-a59e582616aa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xs5lx"
Jul 27 19:22:20.511064 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.511023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62182245-9140-45d5-977d-a59e582616aa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xs5lx\" (UID: \"62182245-9140-45d5-977d-a59e582616aa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xs5lx"
Jul 27 19:22:20.511234 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.511076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbp5j\" (UniqueName: \"kubernetes.io/projected/62182245-9140-45d5-977d-a59e582616aa-kube-api-access-hbp5j\") pod \"cert-manager-operator-controller-manager-55c444bf86-xs5lx\" (UID: \"62182245-9140-45d5-977d-a59e582616aa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xs5lx"
Jul 27 19:22:20.511441 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.511421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62182245-9140-45d5-977d-a59e582616aa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xs5lx\" (UID: \"62182245-9140-45d5-977d-a59e582616aa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xs5lx"
Jul 27 19:22:20.525694 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.525657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbp5j\" (UniqueName: \"kubernetes.io/projected/62182245-9140-45d5-977d-a59e582616aa-kube-api-access-hbp5j\") pod \"cert-manager-operator-controller-manager-55c444bf86-xs5lx\" (UID: \"62182245-9140-45d5-977d-a59e582616aa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xs5lx"
Jul 27 19:22:20.634970 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.634875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xs5lx"
Jul 27 19:22:20.805663 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:20.805633    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xs5lx"]
Jul 27 19:22:20.809359 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:22:20.809328    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62182245_9140_45d5_977d_a59e582616aa.slice/crio-34a485706c2479a4b81b112b369d9c4326cfa681da0a270b366c34d29d2e16a7 WatchSource:0}: Error finding container 34a485706c2479a4b81b112b369d9c4326cfa681da0a270b366c34d29d2e16a7: Status 404 returned error can't find the container with id 34a485706c2479a4b81b112b369d9c4326cfa681da0a270b366c34d29d2e16a7
Jul 27 19:22:21.206718 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:21.206685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xs5lx" event={"ID":"62182245-9140-45d5-977d-a59e582616aa","Type":"ContainerStarted","Data":"34a485706c2479a4b81b112b369d9c4326cfa681da0a270b366c34d29d2e16a7"}
Jul 27 19:22:23.222527 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:23.222449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xs5lx" event={"ID":"62182245-9140-45d5-977d-a59e582616aa","Type":"ContainerStarted","Data":"9dedc68832444ab7cf48691bff9d32bfcec03f7de7ac81f54e789a4e8b2be9ef"}
Jul 27 19:22:23.273029 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:23.272982    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xs5lx" podStartSLOduration=1.228000113 podStartE2EDuration="3.272961366s" totalImagesPullingTime="2.044961253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:22:23.272394048 +0000 UTC m=+285.788098552" watchObservedRunningTime="2026-07-27 19:22:23.272961366 +0000 UTC m=+285.788665869"
Jul 27 19:22:24.598801 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.598771    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"]
Jul 27 19:22:24.620156 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.620127    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"]
Jul 27 19:22:24.620335 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.620272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"
Jul 27 19:22:24.622636 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.622611    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 19:22:24.622756 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.622611    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 19:22:24.623356 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.623340    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q5wxr\""
Jul 27 19:22:24.644662 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.644643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz\" (UID: \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"
Jul 27 19:22:24.644764 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.644679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz\" (UID: \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"
Jul 27 19:22:24.644764 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.644698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5kqd\" (UniqueName: \"kubernetes.io/projected/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-kube-api-access-x5kqd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz\" (UID: \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"
Jul 27 19:22:24.745816 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.745788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz\" (UID: \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"
Jul 27 19:22:24.745957 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.745830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz\" (UID: \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"
Jul 27 19:22:24.745957 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.745854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5kqd\" (UniqueName: \"kubernetes.io/projected/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-kube-api-access-x5kqd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz\" (UID: \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"
Jul 27 19:22:24.746274 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.746233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz\" (UID: \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"
Jul 27 19:22:24.746274 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.746263    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz\" (UID: \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"
Jul 27 19:22:24.754815 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.754790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5kqd\" (UniqueName: \"kubernetes.io/projected/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-kube-api-access-x5kqd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz\" (UID: \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"
Jul 27 19:22:24.929261 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:24.929177    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"
Jul 27 19:22:25.079365 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:25.079332    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"]
Jul 27 19:22:25.082347 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:22:25.082318    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae8e00f_43c7_4917_b640_8bd7d273c4bd.slice/crio-f9f3de3c4ea0f73349136f86fb9caf3c981cdf38c3512c5a00c83e845f02b5b2 WatchSource:0}: Error finding container f9f3de3c4ea0f73349136f86fb9caf3c981cdf38c3512c5a00c83e845f02b5b2: Status 404 returned error can't find the container with id f9f3de3c4ea0f73349136f86fb9caf3c981cdf38c3512c5a00c83e845f02b5b2
Jul 27 19:22:25.232134 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:25.232099    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz" event={"ID":"3ae8e00f-43c7-4917-b640-8bd7d273c4bd","Type":"ContainerStarted","Data":"da352754ee7967edcc5c64de82c31a34ae10aae6f8ad3503a96da641cbed6e32"}
Jul 27 19:22:25.232307 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:25.232157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz" event={"ID":"3ae8e00f-43c7-4917-b640-8bd7d273c4bd","Type":"ContainerStarted","Data":"f9f3de3c4ea0f73349136f86fb9caf3c981cdf38c3512c5a00c83e845f02b5b2"}
Jul 27 19:22:26.237568 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:26.237534    2580 generic.go:356] "Generic (PLEG): container finished" podID="3ae8e00f-43c7-4917-b640-8bd7d273c4bd" containerID="da352754ee7967edcc5c64de82c31a34ae10aae6f8ad3503a96da641cbed6e32" exitCode=0
Jul 27 19:22:26.237947 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:26.237615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz" event={"ID":"3ae8e00f-43c7-4917-b640-8bd7d273c4bd","Type":"ContainerDied","Data":"da352754ee7967edcc5c64de82c31a34ae10aae6f8ad3503a96da641cbed6e32"}
Jul 27 19:22:29.251008 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:29.250972    2580 generic.go:356] "Generic (PLEG): container finished" podID="3ae8e00f-43c7-4917-b640-8bd7d273c4bd" containerID="9b488ce22aef92909d64e2ee2da626e721eb200d1e87c1e5cd469d72b5ad8def" exitCode=0
Jul 27 19:22:29.251503 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:29.251058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz" event={"ID":"3ae8e00f-43c7-4917-b640-8bd7d273c4bd","Type":"ContainerDied","Data":"9b488ce22aef92909d64e2ee2da626e721eb200d1e87c1e5cd469d72b5ad8def"}
Jul 27 19:22:30.256289 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:30.256235    2580 generic.go:356] "Generic (PLEG): container finished" podID="3ae8e00f-43c7-4917-b640-8bd7d273c4bd" containerID="5c59a09017e2860bdb31a8f71cf542631c30457f18b76b273f2806dbfaf1df58" exitCode=0
Jul 27 19:22:30.256289 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:30.256280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz" event={"ID":"3ae8e00f-43c7-4917-b640-8bd7d273c4bd","Type":"ContainerDied","Data":"5c59a09017e2860bdb31a8f71cf542631c30457f18b76b273f2806dbfaf1df58"}
Jul 27 19:22:30.886885 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:30.886852    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8xmwp"]
Jul 27 19:22:30.889542 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:30.889526    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8xmwp"
Jul 27 19:22:30.891805 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:30.891788    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 27 19:22:30.892556 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:30.892531    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ckrzj\""
Jul 27 19:22:30.892556 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:30.892532    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 27 19:22:30.900048 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:30.900028    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8xmwp"]
Jul 27 19:22:30.995182 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:30.995143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b39ddee-f465-4f6b-8bec-0921f7771d74-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8xmwp\" (UID: \"6b39ddee-f465-4f6b-8bec-0921f7771d74\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xmwp"
Jul 27 19:22:30.995383 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:30.995198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g55d\" (UniqueName: \"kubernetes.io/projected/6b39ddee-f465-4f6b-8bec-0921f7771d74-kube-api-access-7g55d\") pod \"cert-manager-webhook-6486ffd997-8xmwp\" (UID: \"6b39ddee-f465-4f6b-8bec-0921f7771d74\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xmwp"
Jul 27 19:22:31.096636 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.096603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b39ddee-f465-4f6b-8bec-0921f7771d74-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8xmwp\" (UID: \"6b39ddee-f465-4f6b-8bec-0921f7771d74\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xmwp"
Jul 27 19:22:31.096821 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.096652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g55d\" (UniqueName: \"kubernetes.io/projected/6b39ddee-f465-4f6b-8bec-0921f7771d74-kube-api-access-7g55d\") pod \"cert-manager-webhook-6486ffd997-8xmwp\" (UID: \"6b39ddee-f465-4f6b-8bec-0921f7771d74\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xmwp"
Jul 27 19:22:31.105048 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.105023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g55d\" (UniqueName: \"kubernetes.io/projected/6b39ddee-f465-4f6b-8bec-0921f7771d74-kube-api-access-7g55d\") pod \"cert-manager-webhook-6486ffd997-8xmwp\" (UID: \"6b39ddee-f465-4f6b-8bec-0921f7771d74\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xmwp"
Jul 27 19:22:31.105421 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.105401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b39ddee-f465-4f6b-8bec-0921f7771d74-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8xmwp\" (UID: \"6b39ddee-f465-4f6b-8bec-0921f7771d74\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xmwp"
Jul 27 19:22:31.208637 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.208553    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8xmwp"
Jul 27 19:22:31.347606 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.347573    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8xmwp"]
Jul 27 19:22:31.350690 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:22:31.350655    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b39ddee_f465_4f6b_8bec_0921f7771d74.slice/crio-9b551b236d6aa205269630e3a630987d98cd455298a55a7892b6bb5e86c85ba9 WatchSource:0}: Error finding container 9b551b236d6aa205269630e3a630987d98cd455298a55a7892b6bb5e86c85ba9: Status 404 returned error can't find the container with id 9b551b236d6aa205269630e3a630987d98cd455298a55a7892b6bb5e86c85ba9
Jul 27 19:22:31.386224 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.386201    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"
Jul 27 19:22:31.501290 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.501201    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-bundle\") pod \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\" (UID: \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\") "
Jul 27 19:22:31.501290 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.501272    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5kqd\" (UniqueName: \"kubernetes.io/projected/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-kube-api-access-x5kqd\") pod \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\" (UID: \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\") "
Jul 27 19:22:31.501456 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.501301    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-util\") pod \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\" (UID: \"3ae8e00f-43c7-4917-b640-8bd7d273c4bd\") "
Jul 27 19:22:31.501639 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.501620    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-bundle" (OuterVolumeSpecName: "bundle") pod "3ae8e00f-43c7-4917-b640-8bd7d273c4bd" (UID: "3ae8e00f-43c7-4917-b640-8bd7d273c4bd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:22:31.503267 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.503234    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-kube-api-access-x5kqd" (OuterVolumeSpecName: "kube-api-access-x5kqd") pod "3ae8e00f-43c7-4917-b640-8bd7d273c4bd" (UID: "3ae8e00f-43c7-4917-b640-8bd7d273c4bd"). InnerVolumeSpecName "kube-api-access-x5kqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:22:31.506273 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.506253    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-util" (OuterVolumeSpecName: "util") pod "3ae8e00f-43c7-4917-b640-8bd7d273c4bd" (UID: "3ae8e00f-43c7-4917-b640-8bd7d273c4bd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:22:31.602394 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.602363    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-bundle\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:22:31.602394 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.602392    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5kqd\" (UniqueName: \"kubernetes.io/projected/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-kube-api-access-x5kqd\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:22:31.602394 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:31.602401    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ae8e00f-43c7-4917-b640-8bd7d273c4bd-util\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:22:32.268852 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:32.268815    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz"
Jul 27 19:22:32.268852 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:32.268834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6gzsz" event={"ID":"3ae8e00f-43c7-4917-b640-8bd7d273c4bd","Type":"ContainerDied","Data":"f9f3de3c4ea0f73349136f86fb9caf3c981cdf38c3512c5a00c83e845f02b5b2"}
Jul 27 19:22:32.269109 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:32.268889    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9f3de3c4ea0f73349136f86fb9caf3c981cdf38c3512c5a00c83e845f02b5b2"
Jul 27 19:22:32.270277 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:32.270233    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8xmwp" event={"ID":"6b39ddee-f465-4f6b-8bec-0921f7771d74","Type":"ContainerStarted","Data":"9b551b236d6aa205269630e3a630987d98cd455298a55a7892b6bb5e86c85ba9"}
Jul 27 19:22:34.282766 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:34.282734    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8xmwp" event={"ID":"6b39ddee-f465-4f6b-8bec-0921f7771d74","Type":"ContainerStarted","Data":"1d9ade4678452e8bcf15566f06ef4c5289a6a9acc338612594730df86eb392c9"}
Jul 27 19:22:34.283149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:34.282799    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8xmwp"
Jul 27 19:22:34.301094 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:34.301044    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-8xmwp" podStartSLOduration=1.8881291180000002 podStartE2EDuration="4.30102947s" totalImagesPullingTime="2.412900352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:22:34.29915104 +0000 UTC m=+296.814855544" watchObservedRunningTime="2026-07-27 19:22:34.30102947 +0000 UTC m=+296.816733974"
Jul 27 19:22:37.939489 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:37.939460    2580 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 19:22:39.040307 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.040275    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6w6gf"]
Jul 27 19:22:39.041669 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.040551    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ae8e00f-43c7-4917-b640-8bd7d273c4bd" containerName="pull"
Jul 27 19:22:39.041669 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.040561    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ae8e00f-43c7-4917-b640-8bd7d273c4bd" containerName="pull"
Jul 27 19:22:39.041669 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.040570    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ae8e00f-43c7-4917-b640-8bd7d273c4bd" containerName="extract"
Jul 27 19:22:39.041669 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.040575    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ae8e00f-43c7-4917-b640-8bd7d273c4bd" containerName="extract"
Jul 27 19:22:39.041669 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.040594    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ae8e00f-43c7-4917-b640-8bd7d273c4bd" containerName="util"
Jul 27 19:22:39.041669 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.040599    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ae8e00f-43c7-4917-b640-8bd7d273c4bd" containerName="util"
Jul 27 19:22:39.041669 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.040648    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ae8e00f-43c7-4917-b640-8bd7d273c4bd" containerName="extract"
Jul 27 19:22:39.042366 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.042351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6w6gf"
Jul 27 19:22:39.044772 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.044746    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 19:22:39.045522 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.045502    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-rsfls\""
Jul 27 19:22:39.045618 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.045524    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 19:22:39.052565 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.052537    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6w6gf"]
Jul 27 19:22:39.169876 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.169839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/145dde04-5be2-4ac6-b7d3-c67edd7aee45-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6w6gf\" (UID: \"145dde04-5be2-4ac6-b7d3-c67edd7aee45\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6w6gf"
Jul 27 19:22:39.170047 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.169901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwp4n\" (UniqueName: \"kubernetes.io/projected/145dde04-5be2-4ac6-b7d3-c67edd7aee45-kube-api-access-xwp4n\") pod \"openshift-lws-operator-fd8ccff4c-6w6gf\" (UID: \"145dde04-5be2-4ac6-b7d3-c67edd7aee45\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6w6gf"
Jul 27 19:22:39.270729 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.270697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwp4n\" (UniqueName: \"kubernetes.io/projected/145dde04-5be2-4ac6-b7d3-c67edd7aee45-kube-api-access-xwp4n\") pod \"openshift-lws-operator-fd8ccff4c-6w6gf\" (UID: \"145dde04-5be2-4ac6-b7d3-c67edd7aee45\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6w6gf"
Jul 27 19:22:39.270883 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.270766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/145dde04-5be2-4ac6-b7d3-c67edd7aee45-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6w6gf\" (UID: \"145dde04-5be2-4ac6-b7d3-c67edd7aee45\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6w6gf"
Jul 27 19:22:39.271145 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.271126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/145dde04-5be2-4ac6-b7d3-c67edd7aee45-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6w6gf\" (UID: \"145dde04-5be2-4ac6-b7d3-c67edd7aee45\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6w6gf"
Jul 27 19:22:39.279679 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.279659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwp4n\" (UniqueName: \"kubernetes.io/projected/145dde04-5be2-4ac6-b7d3-c67edd7aee45-kube-api-access-xwp4n\") pod \"openshift-lws-operator-fd8ccff4c-6w6gf\" (UID: \"145dde04-5be2-4ac6-b7d3-c67edd7aee45\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6w6gf"
Jul 27 19:22:39.351368 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.351292    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6w6gf"
Jul 27 19:22:39.475397 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.475368    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6w6gf"]
Jul 27 19:22:39.478558 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:22:39.478532    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod145dde04_5be2_4ac6_b7d3_c67edd7aee45.slice/crio-143b4cd2b11643ec0d1e692c1d04d9f716961e42a14d209d6a1a0b6bbb6b5f11 WatchSource:0}: Error finding container 143b4cd2b11643ec0d1e692c1d04d9f716961e42a14d209d6a1a0b6bbb6b5f11: Status 404 returned error can't find the container with id 143b4cd2b11643ec0d1e692c1d04d9f716961e42a14d209d6a1a0b6bbb6b5f11
Jul 27 19:22:39.480572 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:39.480556    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:22:40.288921 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:40.288888    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8xmwp"
Jul 27 19:22:40.308721 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:40.308694    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6w6gf" event={"ID":"145dde04-5be2-4ac6-b7d3-c67edd7aee45","Type":"ContainerStarted","Data":"143b4cd2b11643ec0d1e692c1d04d9f716961e42a14d209d6a1a0b6bbb6b5f11"}
Jul 27 19:22:41.314539 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:41.314489    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6w6gf" event={"ID":"145dde04-5be2-4ac6-b7d3-c67edd7aee45","Type":"ContainerStarted","Data":"0b6158283ec33c7a582a25405f32b06695fa548b2b10056acddfec9fcad9f1a2"}
Jul 27 19:22:41.331330 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:41.331281    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6w6gf" podStartSLOduration=0.711928911 podStartE2EDuration="2.331265286s" totalImagesPullingTime="1.619336375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:22:41.330228264 +0000 UTC m=+303.845932768" watchObservedRunningTime="2026-07-27 19:22:41.331265286 +0000 UTC m=+303.846969790"
Jul 27 19:22:44.059339 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.059302    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"]
Jul 27 19:22:44.061985 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.061970    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"
Jul 27 19:22:44.064412 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.064389    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 19:22:44.064509 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.064459    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 19:22:44.064509 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.064471    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q5wxr\""
Jul 27 19:22:44.072443 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.072422    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"]
Jul 27 19:22:44.209505 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.209476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8m5\" (UniqueName: \"kubernetes.io/projected/31977289-6b01-48e0-8d5b-50d33f98b549-kube-api-access-6c8m5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n\" (UID: \"31977289-6b01-48e0-8d5b-50d33f98b549\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"
Jul 27 19:22:44.209660 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.209534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31977289-6b01-48e0-8d5b-50d33f98b549-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n\" (UID: \"31977289-6b01-48e0-8d5b-50d33f98b549\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"
Jul 27 19:22:44.209660 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.209595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31977289-6b01-48e0-8d5b-50d33f98b549-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n\" (UID: \"31977289-6b01-48e0-8d5b-50d33f98b549\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"
Jul 27 19:22:44.310883 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.310806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31977289-6b01-48e0-8d5b-50d33f98b549-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n\" (UID: \"31977289-6b01-48e0-8d5b-50d33f98b549\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"
Jul 27 19:22:44.310883 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.310843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31977289-6b01-48e0-8d5b-50d33f98b549-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n\" (UID: \"31977289-6b01-48e0-8d5b-50d33f98b549\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"
Jul 27 19:22:44.311053 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.310886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8m5\" (UniqueName: \"kubernetes.io/projected/31977289-6b01-48e0-8d5b-50d33f98b549-kube-api-access-6c8m5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n\" (UID: \"31977289-6b01-48e0-8d5b-50d33f98b549\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"
Jul 27 19:22:44.311186 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.311169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31977289-6b01-48e0-8d5b-50d33f98b549-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n\" (UID: \"31977289-6b01-48e0-8d5b-50d33f98b549\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"
Jul 27 19:22:44.311228 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.311192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31977289-6b01-48e0-8d5b-50d33f98b549-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n\" (UID: \"31977289-6b01-48e0-8d5b-50d33f98b549\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"
Jul 27 19:22:44.319183 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.319159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8m5\" (UniqueName: \"kubernetes.io/projected/31977289-6b01-48e0-8d5b-50d33f98b549-kube-api-access-6c8m5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n\" (UID: \"31977289-6b01-48e0-8d5b-50d33f98b549\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"
Jul 27 19:22:44.370373 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.370344    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"
Jul 27 19:22:44.496320 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:44.496297    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"]
Jul 27 19:22:44.498532 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:22:44.498508    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31977289_6b01_48e0_8d5b_50d33f98b549.slice/crio-6f06ba3a220b06e6ee2ea81c1eb113e30a28a64f248a47df5019c5845cd1f77b WatchSource:0}: Error finding container 6f06ba3a220b06e6ee2ea81c1eb113e30a28a64f248a47df5019c5845cd1f77b: Status 404 returned error can't find the container with id 6f06ba3a220b06e6ee2ea81c1eb113e30a28a64f248a47df5019c5845cd1f77b
Jul 27 19:22:45.334460 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:45.334428    2580 generic.go:356] "Generic (PLEG): container finished" podID="31977289-6b01-48e0-8d5b-50d33f98b549" containerID="ec5138c34b8c229d8b814274ca0b148f830dad3f6030a13c448872caf09823d0" exitCode=0
Jul 27 19:22:45.334783 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:45.334504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n" event={"ID":"31977289-6b01-48e0-8d5b-50d33f98b549","Type":"ContainerDied","Data":"ec5138c34b8c229d8b814274ca0b148f830dad3f6030a13c448872caf09823d0"}
Jul 27 19:22:45.334783 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:45.334546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n" event={"ID":"31977289-6b01-48e0-8d5b-50d33f98b549","Type":"ContainerStarted","Data":"6f06ba3a220b06e6ee2ea81c1eb113e30a28a64f248a47df5019c5845cd1f77b"}
Jul 27 19:22:46.340159 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:46.340129    2580 generic.go:356] "Generic (PLEG): container finished" podID="31977289-6b01-48e0-8d5b-50d33f98b549" containerID="cb1d327615a9a763f99a12d0e7ba7f98a24c516a58f561bf62787a043625ff54" exitCode=0
Jul 27 19:22:46.340558 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:46.340220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n" event={"ID":"31977289-6b01-48e0-8d5b-50d33f98b549","Type":"ContainerDied","Data":"cb1d327615a9a763f99a12d0e7ba7f98a24c516a58f561bf62787a043625ff54"}
Jul 27 19:22:47.346461 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:47.346427    2580 generic.go:356] "Generic (PLEG): container finished" podID="31977289-6b01-48e0-8d5b-50d33f98b549" containerID="2847d493ceb1a72652aac77b7c83853100e5c4ccb27899dbfd7b09c9023f2a43" exitCode=0
Jul 27 19:22:47.346840 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:47.346498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n" event={"ID":"31977289-6b01-48e0-8d5b-50d33f98b549","Type":"ContainerDied","Data":"2847d493ceb1a72652aac77b7c83853100e5c4ccb27899dbfd7b09c9023f2a43"}
Jul 27 19:22:48.471613 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:48.471586    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"
Jul 27 19:22:48.546978 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:48.546948    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c8m5\" (UniqueName: \"kubernetes.io/projected/31977289-6b01-48e0-8d5b-50d33f98b549-kube-api-access-6c8m5\") pod \"31977289-6b01-48e0-8d5b-50d33f98b549\" (UID: \"31977289-6b01-48e0-8d5b-50d33f98b549\") "
Jul 27 19:22:48.547140 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:48.547044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31977289-6b01-48e0-8d5b-50d33f98b549-bundle\") pod \"31977289-6b01-48e0-8d5b-50d33f98b549\" (UID: \"31977289-6b01-48e0-8d5b-50d33f98b549\") "
Jul 27 19:22:48.547140 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:48.547105    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31977289-6b01-48e0-8d5b-50d33f98b549-util\") pod \"31977289-6b01-48e0-8d5b-50d33f98b549\" (UID: \"31977289-6b01-48e0-8d5b-50d33f98b549\") "
Jul 27 19:22:48.547752 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:48.547723    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31977289-6b01-48e0-8d5b-50d33f98b549-bundle" (OuterVolumeSpecName: "bundle") pod "31977289-6b01-48e0-8d5b-50d33f98b549" (UID: "31977289-6b01-48e0-8d5b-50d33f98b549"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:22:48.549195 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:48.549165    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31977289-6b01-48e0-8d5b-50d33f98b549-kube-api-access-6c8m5" (OuterVolumeSpecName: "kube-api-access-6c8m5") pod "31977289-6b01-48e0-8d5b-50d33f98b549" (UID: "31977289-6b01-48e0-8d5b-50d33f98b549"). InnerVolumeSpecName "kube-api-access-6c8m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:22:48.552686 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:48.552659    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31977289-6b01-48e0-8d5b-50d33f98b549-util" (OuterVolumeSpecName: "util") pod "31977289-6b01-48e0-8d5b-50d33f98b549" (UID: "31977289-6b01-48e0-8d5b-50d33f98b549"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:22:48.648613 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:48.648547    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31977289-6b01-48e0-8d5b-50d33f98b549-bundle\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:22:48.648613 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:48.648570    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31977289-6b01-48e0-8d5b-50d33f98b549-util\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:22:48.648613 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:48.648579    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c8m5\" (UniqueName: \"kubernetes.io/projected/31977289-6b01-48e0-8d5b-50d33f98b549-kube-api-access-6c8m5\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:22:49.356774 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:49.356736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n" event={"ID":"31977289-6b01-48e0-8d5b-50d33f98b549","Type":"ContainerDied","Data":"6f06ba3a220b06e6ee2ea81c1eb113e30a28a64f248a47df5019c5845cd1f77b"}
Jul 27 19:22:49.356774 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:49.356780    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f06ba3a220b06e6ee2ea81c1eb113e30a28a64f248a47df5019c5845cd1f77b"
Jul 27 19:22:49.356973 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:49.356743    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5q47n"
Jul 27 19:22:57.070094 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.070063    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"]
Jul 27 19:22:57.070477 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.070374    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31977289-6b01-48e0-8d5b-50d33f98b549" containerName="util"
Jul 27 19:22:57.070477 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.070387    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31977289-6b01-48e0-8d5b-50d33f98b549" containerName="util"
Jul 27 19:22:57.070477 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.070394    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31977289-6b01-48e0-8d5b-50d33f98b549" containerName="extract"
Jul 27 19:22:57.070477 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.070399    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31977289-6b01-48e0-8d5b-50d33f98b549" containerName="extract"
Jul 27 19:22:57.070477 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.070407    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31977289-6b01-48e0-8d5b-50d33f98b549" containerName="pull"
Jul 27 19:22:57.070477 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.070412    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31977289-6b01-48e0-8d5b-50d33f98b549" containerName="pull"
Jul 27 19:22:57.070477 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.070465    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="31977289-6b01-48e0-8d5b-50d33f98b549" containerName="extract"
Jul 27 19:22:57.074044 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.074027    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.076855 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.076838    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 19:22:57.077336 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.077317    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 19:22:57.077435 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.077424    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 19:22:57.077516 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.077472    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 19:22:57.077572 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.077539    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-92f7h\""
Jul 27 19:22:57.110874 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.110848    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"]
Jul 27 19:22:57.216754 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.216719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b08034f2-0acf-44fc-8dae-42db64288bab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.216914 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.216787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcgcn\" (UniqueName: \"kubernetes.io/projected/b08034f2-0acf-44fc-8dae-42db64288bab-kube-api-access-bcgcn\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.216914 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.216838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b08034f2-0acf-44fc-8dae-42db64288bab-tmp\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.216914 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.216880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b08034f2-0acf-44fc-8dae-42db64288bab-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.216914 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.216900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b08034f2-0acf-44fc-8dae-42db64288bab-manifests\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.317656 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.317615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcgcn\" (UniqueName: \"kubernetes.io/projected/b08034f2-0acf-44fc-8dae-42db64288bab-kube-api-access-bcgcn\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.317656 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.317659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b08034f2-0acf-44fc-8dae-42db64288bab-tmp\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.317881 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.317778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b08034f2-0acf-44fc-8dae-42db64288bab-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.317881 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.317813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b08034f2-0acf-44fc-8dae-42db64288bab-manifests\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.317881 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.317850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b08034f2-0acf-44fc-8dae-42db64288bab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.318290 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.318238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b08034f2-0acf-44fc-8dae-42db64288bab-manifests\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.319820 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.319797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b08034f2-0acf-44fc-8dae-42db64288bab-tmp\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.320206 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.320158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b08034f2-0acf-44fc-8dae-42db64288bab-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.320344 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.320323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b08034f2-0acf-44fc-8dae-42db64288bab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.326313 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.326291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcgcn\" (UniqueName: \"kubernetes.io/projected/b08034f2-0acf-44fc-8dae-42db64288bab-kube-api-access-bcgcn\") pod \"opendatahub-operator-controller-manager-7d79566f86-jwj8m\" (UID: \"b08034f2-0acf-44fc-8dae-42db64288bab\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.383924 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.383899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:22:57.518279 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:57.518219    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"]
Jul 27 19:22:57.523079 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:22:57.523052    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08034f2_0acf_44fc_8dae_42db64288bab.slice/crio-65d6fbe641efd16e16d2cbda16ccd2bbeaf27b224938dd8f132aac211415a60e WatchSource:0}: Error finding container 65d6fbe641efd16e16d2cbda16ccd2bbeaf27b224938dd8f132aac211415a60e: Status 404 returned error can't find the container with id 65d6fbe641efd16e16d2cbda16ccd2bbeaf27b224938dd8f132aac211415a60e
Jul 27 19:22:58.397851 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:22:58.397812    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m" event={"ID":"b08034f2-0acf-44fc-8dae-42db64288bab","Type":"ContainerStarted","Data":"65d6fbe641efd16e16d2cbda16ccd2bbeaf27b224938dd8f132aac211415a60e"}
Jul 27 19:23:01.417468 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:01.417427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m" event={"ID":"b08034f2-0acf-44fc-8dae-42db64288bab","Type":"ContainerStarted","Data":"3e1bd1b6a8fca8bb4834da0eb02858b69bf3f135953e988e6becd10b379b8a5a"}
Jul 27 19:23:02.422880 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:02.422845    2580 generic.go:356] "Generic (PLEG): container finished" podID="b08034f2-0acf-44fc-8dae-42db64288bab" containerID="3e1bd1b6a8fca8bb4834da0eb02858b69bf3f135953e988e6becd10b379b8a5a" exitCode=0
Jul 27 19:23:02.423268 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:02.422923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m" event={"ID":"b08034f2-0acf-44fc-8dae-42db64288bab","Type":"ContainerDied","Data":"3e1bd1b6a8fca8bb4834da0eb02858b69bf3f135953e988e6becd10b379b8a5a"}
Jul 27 19:23:03.429454 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:03.429413    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m" event={"ID":"b08034f2-0acf-44fc-8dae-42db64288bab","Type":"ContainerStarted","Data":"1701ba3269121e94f0723ec7c0d42fbebe29901dee6bb1532fa533b60a493b09"}
Jul 27 19:23:03.429831 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:03.429532    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:23:03.453548 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:03.453508    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m" podStartSLOduration=2.162177359 podStartE2EDuration="6.453494192s" totalImagesPullingTime="3.253067173s" totalInitContainerRuntime="1.03824966s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:22:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:03.451607864 +0000 UTC m=+325.967312369" watchObservedRunningTime="2026-07-27 19:23:03.453494192 +0000 UTC m=+325.969198696"
Jul 27 19:23:14.435579 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:14.435547    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-jwj8m"
Jul 27 19:23:17.290174 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.290141    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"]
Jul 27 19:23:17.296719 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.296703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:17.315874 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.315847    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 19:23:17.316316 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.316291    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 19:23:17.316416 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.316297    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 19:23:17.316416 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.316347    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-g4v6z\""
Jul 27 19:23:17.343447 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.343421    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"]
Jul 27 19:23:17.383485 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.383446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhkf7\" (UniqueName: \"kubernetes.io/projected/ab1c6f30-49e8-4c0a-bacf-53263f7458fc-kube-api-access-fhkf7\") pod \"lws-controller-manager-6fd758c6bf-cjt5j\" (UID: \"ab1c6f30-49e8-4c0a-bacf-53263f7458fc\") " pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:17.383620 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.383557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab1c6f30-49e8-4c0a-bacf-53263f7458fc-cert\") pod \"lws-controller-manager-6fd758c6bf-cjt5j\" (UID: \"ab1c6f30-49e8-4c0a-bacf-53263f7458fc\") " pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:17.383683 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.383618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab1c6f30-49e8-4c0a-bacf-53263f7458fc-metrics-cert\") pod \"lws-controller-manager-6fd758c6bf-cjt5j\" (UID: \"ab1c6f30-49e8-4c0a-bacf-53263f7458fc\") " pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:17.383683 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.383655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ab1c6f30-49e8-4c0a-bacf-53263f7458fc-manager-config\") pod \"lws-controller-manager-6fd758c6bf-cjt5j\" (UID: \"ab1c6f30-49e8-4c0a-bacf-53263f7458fc\") " pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:17.412654 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.412626    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-949fd4987-nmvwf"]
Jul 27 19:23:17.414816 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.414798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-949fd4987-nmvwf"
Jul 27 19:23:17.427945 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.427929    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 19:23:17.428998 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.428984    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 19:23:17.430222 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.430210    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 19:23:17.441525 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.441504    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mrm97\""
Jul 27 19:23:17.441603 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.441526    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 19:23:17.467462 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.467441    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-949fd4987-nmvwf"]
Jul 27 19:23:17.484100 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.484073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ab1c6f30-49e8-4c0a-bacf-53263f7458fc-manager-config\") pod \"lws-controller-manager-6fd758c6bf-cjt5j\" (UID: \"ab1c6f30-49e8-4c0a-bacf-53263f7458fc\") " pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:17.484195 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.484104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b8hm\" (UniqueName: \"kubernetes.io/projected/0b40ea9c-865c-4631-afe4-6d48cfa7ffd6-kube-api-access-4b8hm\") pod \"kube-auth-proxy-949fd4987-nmvwf\" (UID: \"0b40ea9c-865c-4631-afe4-6d48cfa7ffd6\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nmvwf"
Jul 27 19:23:17.484195 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.484133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b40ea9c-865c-4631-afe4-6d48cfa7ffd6-tmp\") pod \"kube-auth-proxy-949fd4987-nmvwf\" (UID: \"0b40ea9c-865c-4631-afe4-6d48cfa7ffd6\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nmvwf"
Jul 27 19:23:17.484195 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.484159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhkf7\" (UniqueName: \"kubernetes.io/projected/ab1c6f30-49e8-4c0a-bacf-53263f7458fc-kube-api-access-fhkf7\") pod \"lws-controller-manager-6fd758c6bf-cjt5j\" (UID: \"ab1c6f30-49e8-4c0a-bacf-53263f7458fc\") " pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:17.484377 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.484299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b40ea9c-865c-4631-afe4-6d48cfa7ffd6-tls-certs\") pod \"kube-auth-proxy-949fd4987-nmvwf\" (UID: \"0b40ea9c-865c-4631-afe4-6d48cfa7ffd6\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nmvwf"
Jul 27 19:23:17.484377 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.484333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab1c6f30-49e8-4c0a-bacf-53263f7458fc-cert\") pod \"lws-controller-manager-6fd758c6bf-cjt5j\" (UID: \"ab1c6f30-49e8-4c0a-bacf-53263f7458fc\") " pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:17.484470 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.484388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab1c6f30-49e8-4c0a-bacf-53263f7458fc-metrics-cert\") pod \"lws-controller-manager-6fd758c6bf-cjt5j\" (UID: \"ab1c6f30-49e8-4c0a-bacf-53263f7458fc\") " pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:17.484920 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.484895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ab1c6f30-49e8-4c0a-bacf-53263f7458fc-manager-config\") pod \"lws-controller-manager-6fd758c6bf-cjt5j\" (UID: \"ab1c6f30-49e8-4c0a-bacf-53263f7458fc\") " pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:17.486813 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.486795    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ab1c6f30-49e8-4c0a-bacf-53263f7458fc-metrics-cert\") pod \"lws-controller-manager-6fd758c6bf-cjt5j\" (UID: \"ab1c6f30-49e8-4c0a-bacf-53263f7458fc\") " pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:17.486919 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.486815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab1c6f30-49e8-4c0a-bacf-53263f7458fc-cert\") pod \"lws-controller-manager-6fd758c6bf-cjt5j\" (UID: \"ab1c6f30-49e8-4c0a-bacf-53263f7458fc\") " pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:17.505274 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.505257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhkf7\" (UniqueName: \"kubernetes.io/projected/ab1c6f30-49e8-4c0a-bacf-53263f7458fc-kube-api-access-fhkf7\") pod \"lws-controller-manager-6fd758c6bf-cjt5j\" (UID: \"ab1c6f30-49e8-4c0a-bacf-53263f7458fc\") " pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:17.584821 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.584747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b40ea9c-865c-4631-afe4-6d48cfa7ffd6-tmp\") pod \"kube-auth-proxy-949fd4987-nmvwf\" (UID: \"0b40ea9c-865c-4631-afe4-6d48cfa7ffd6\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nmvwf"
Jul 27 19:23:17.584821 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.584794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b40ea9c-865c-4631-afe4-6d48cfa7ffd6-tls-certs\") pod \"kube-auth-proxy-949fd4987-nmvwf\" (UID: \"0b40ea9c-865c-4631-afe4-6d48cfa7ffd6\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nmvwf"
Jul 27 19:23:17.585019 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.584843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8hm\" (UniqueName: \"kubernetes.io/projected/0b40ea9c-865c-4631-afe4-6d48cfa7ffd6-kube-api-access-4b8hm\") pod \"kube-auth-proxy-949fd4987-nmvwf\" (UID: \"0b40ea9c-865c-4631-afe4-6d48cfa7ffd6\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nmvwf"
Jul 27 19:23:17.587014 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.586991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b40ea9c-865c-4631-afe4-6d48cfa7ffd6-tmp\") pod \"kube-auth-proxy-949fd4987-nmvwf\" (UID: \"0b40ea9c-865c-4631-afe4-6d48cfa7ffd6\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nmvwf"
Jul 27 19:23:17.587187 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.587169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b40ea9c-865c-4631-afe4-6d48cfa7ffd6-tls-certs\") pod \"kube-auth-proxy-949fd4987-nmvwf\" (UID: \"0b40ea9c-865c-4631-afe4-6d48cfa7ffd6\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nmvwf"
Jul 27 19:23:17.595388 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.595368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8hm\" (UniqueName: \"kubernetes.io/projected/0b40ea9c-865c-4631-afe4-6d48cfa7ffd6-kube-api-access-4b8hm\") pod \"kube-auth-proxy-949fd4987-nmvwf\" (UID: \"0b40ea9c-865c-4631-afe4-6d48cfa7ffd6\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nmvwf"
Jul 27 19:23:17.606170 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.606152    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:17.723981 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.723951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-949fd4987-nmvwf"
Jul 27 19:23:17.748318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.748296    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"]
Jul 27 19:23:17.750629 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:23:17.750603    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1c6f30_49e8_4c0a_bacf_53263f7458fc.slice/crio-6332d969c27d577079be80ab4e88bbf67306246b59d5656582b8991d937bcb3f WatchSource:0}: Error finding container 6332d969c27d577079be80ab4e88bbf67306246b59d5656582b8991d937bcb3f: Status 404 returned error can't find the container with id 6332d969c27d577079be80ab4e88bbf67306246b59d5656582b8991d937bcb3f
Jul 27 19:23:17.862997 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:17.862971    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-949fd4987-nmvwf"]
Jul 27 19:23:17.863922 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:23:17.863895    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b40ea9c_865c_4631_afe4_6d48cfa7ffd6.slice/crio-31cf6fccf481d4bf53897b0863160459b362a437d92b708b6b694ef286ae567b WatchSource:0}: Error finding container 31cf6fccf481d4bf53897b0863160459b362a437d92b708b6b694ef286ae567b: Status 404 returned error can't find the container with id 31cf6fccf481d4bf53897b0863160459b362a437d92b708b6b694ef286ae567b
Jul 27 19:23:18.499689 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:18.499651    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j" event={"ID":"ab1c6f30-49e8-4c0a-bacf-53263f7458fc","Type":"ContainerStarted","Data":"6332d969c27d577079be80ab4e88bbf67306246b59d5656582b8991d937bcb3f"}
Jul 27 19:23:18.502232 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:18.502100    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-949fd4987-nmvwf" event={"ID":"0b40ea9c-865c-4631-afe4-6d48cfa7ffd6","Type":"ContainerStarted","Data":"31cf6fccf481d4bf53897b0863160459b362a437d92b708b6b694ef286ae567b"}
Jul 27 19:23:19.508639 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:19.508604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j" event={"ID":"ab1c6f30-49e8-4c0a-bacf-53263f7458fc","Type":"ContainerStarted","Data":"a85791eb2dbf0234c5dc1cb98101da91a81ff159684a0b1067cfee1a420064b6"}
Jul 27 19:23:19.509058 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:19.508721    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:23:19.525470 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:19.525418    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j" podStartSLOduration=1.271563504 podStartE2EDuration="2.52540194s" totalImagesPullingTime="1.253838436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:19.524195002 +0000 UTC m=+342.039899508" watchObservedRunningTime="2026-07-27 19:23:19.52540194 +0000 UTC m=+342.041106517"
Jul 27 19:23:20.515706 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:20.515604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-949fd4987-nmvwf" event={"ID":"0b40ea9c-865c-4631-afe4-6d48cfa7ffd6","Type":"ContainerStarted","Data":"3a2a10269b65afa0ae674c12d1c5c9aed62b592c3e2bf36dde01697ad1867dc1"}
Jul 27 19:23:20.534190 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:20.534140    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-949fd4987-nmvwf" podStartSLOduration=1.248516237 podStartE2EDuration="3.534125232s" totalImagesPullingTime="2.285608995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:20.532627943 +0000 UTC m=+343.048332448" watchObservedRunningTime="2026-07-27 19:23:20.534125232 +0000 UTC m=+343.049829764"
Jul 27 19:23:30.517972 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:23:30.517893    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6fd758c6bf-cjt5j"
Jul 27 19:24:56.230603 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.230573    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"]
Jul 27 19:24:56.233765 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.233748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"
Jul 27 19:24:56.236208 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.236187    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 19:24:56.236332 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.236184    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 19:24:56.237007 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.236984    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6w68x\""
Jul 27 19:24:56.242271 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.242251    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"]
Jul 27 19:24:56.365547 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.365520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ed7bb6d-0419-4b71-bac9-833132273e22-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7\" (UID: \"1ed7bb6d-0419-4b71-bac9-833132273e22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"
Jul 27 19:24:56.365695 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.365561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ed7bb6d-0419-4b71-bac9-833132273e22-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7\" (UID: \"1ed7bb6d-0419-4b71-bac9-833132273e22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"
Jul 27 19:24:56.365695 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.365633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8p86\" (UniqueName: \"kubernetes.io/projected/1ed7bb6d-0419-4b71-bac9-833132273e22-kube-api-access-f8p86\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7\" (UID: \"1ed7bb6d-0419-4b71-bac9-833132273e22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"
Jul 27 19:24:56.466960 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.466933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ed7bb6d-0419-4b71-bac9-833132273e22-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7\" (UID: \"1ed7bb6d-0419-4b71-bac9-833132273e22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"
Jul 27 19:24:56.467089 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.466965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ed7bb6d-0419-4b71-bac9-833132273e22-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7\" (UID: \"1ed7bb6d-0419-4b71-bac9-833132273e22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"
Jul 27 19:24:56.467089 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.466994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8p86\" (UniqueName: \"kubernetes.io/projected/1ed7bb6d-0419-4b71-bac9-833132273e22-kube-api-access-f8p86\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7\" (UID: \"1ed7bb6d-0419-4b71-bac9-833132273e22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"
Jul 27 19:24:56.467326 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.467304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ed7bb6d-0419-4b71-bac9-833132273e22-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7\" (UID: \"1ed7bb6d-0419-4b71-bac9-833132273e22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"
Jul 27 19:24:56.467405 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.467350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ed7bb6d-0419-4b71-bac9-833132273e22-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7\" (UID: \"1ed7bb6d-0419-4b71-bac9-833132273e22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"
Jul 27 19:24:56.484627 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.484574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8p86\" (UniqueName: \"kubernetes.io/projected/1ed7bb6d-0419-4b71-bac9-833132273e22-kube-api-access-f8p86\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7\" (UID: \"1ed7bb6d-0419-4b71-bac9-833132273e22\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"
Jul 27 19:24:56.494312 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.494285    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"]
Jul 27 19:24:56.497513 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.497499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"
Jul 27 19:24:56.516703 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.516680    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"]
Jul 27 19:24:56.543911 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.543890    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"
Jul 27 19:24:56.665633 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.665608    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"]
Jul 27 19:24:56.667861 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:24:56.667832    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed7bb6d_0419_4b71_bac9_833132273e22.slice/crio-fdb31496b2ba2db5281f241299d4b6624474c987799f996acf203bc899662305 WatchSource:0}: Error finding container fdb31496b2ba2db5281f241299d4b6624474c987799f996acf203bc899662305: Status 404 returned error can't find the container with id fdb31496b2ba2db5281f241299d4b6624474c987799f996acf203bc899662305
Jul 27 19:24:56.668137 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.668118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a2c055d-801f-4365-b9b8-487b48ef669b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m\" (UID: \"2a2c055d-801f-4365-b9b8-487b48ef669b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"
Jul 27 19:24:56.668190 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.668146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a2c055d-801f-4365-b9b8-487b48ef669b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m\" (UID: \"2a2c055d-801f-4365-b9b8-487b48ef669b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"
Jul 27 19:24:56.668190 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.668182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k569q\" (UniqueName: \"kubernetes.io/projected/2a2c055d-801f-4365-b9b8-487b48ef669b-kube-api-access-k569q\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m\" (UID: \"2a2c055d-801f-4365-b9b8-487b48ef669b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"
Jul 27 19:24:56.769301 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.769273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a2c055d-801f-4365-b9b8-487b48ef669b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m\" (UID: \"2a2c055d-801f-4365-b9b8-487b48ef669b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"
Jul 27 19:24:56.769440 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.769313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a2c055d-801f-4365-b9b8-487b48ef669b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m\" (UID: \"2a2c055d-801f-4365-b9b8-487b48ef669b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"
Jul 27 19:24:56.769440 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.769355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k569q\" (UniqueName: \"kubernetes.io/projected/2a2c055d-801f-4365-b9b8-487b48ef669b-kube-api-access-k569q\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m\" (UID: \"2a2c055d-801f-4365-b9b8-487b48ef669b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"
Jul 27 19:24:56.769731 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.769662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a2c055d-801f-4365-b9b8-487b48ef669b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m\" (UID: \"2a2c055d-801f-4365-b9b8-487b48ef669b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"
Jul 27 19:24:56.769731 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.769694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a2c055d-801f-4365-b9b8-487b48ef669b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m\" (UID: \"2a2c055d-801f-4365-b9b8-487b48ef669b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"
Jul 27 19:24:56.778179 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.778159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k569q\" (UniqueName: \"kubernetes.io/projected/2a2c055d-801f-4365-b9b8-487b48ef669b-kube-api-access-k569q\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m\" (UID: \"2a2c055d-801f-4365-b9b8-487b48ef669b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"
Jul 27 19:24:56.806588 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.806571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"
Jul 27 19:24:56.886740 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.886709    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"]
Jul 27 19:24:56.890934 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.890909    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"
Jul 27 19:24:56.901463 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.901437    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"]
Jul 27 19:24:56.929149 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.929092    2580 generic.go:356] "Generic (PLEG): container finished" podID="1ed7bb6d-0419-4b71-bac9-833132273e22" containerID="b63a20c87d3ad114a81c19bb23f3b888fa22394dd505f778aab5d783d02b512d" exitCode=0
Jul 27 19:24:56.929318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.929206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7" event={"ID":"1ed7bb6d-0419-4b71-bac9-833132273e22","Type":"ContainerDied","Data":"b63a20c87d3ad114a81c19bb23f3b888fa22394dd505f778aab5d783d02b512d"}
Jul 27 19:24:56.929318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.929263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7" event={"ID":"1ed7bb6d-0419-4b71-bac9-833132273e22","Type":"ContainerStarted","Data":"fdb31496b2ba2db5281f241299d4b6624474c987799f996acf203bc899662305"}
Jul 27 19:24:56.931321 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:56.931302    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"]
Jul 27 19:24:56.933546 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:24:56.933523    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2c055d_801f_4365_b9b8_487b48ef669b.slice/crio-4a99ed0b6a0c80758733e756345678c1da2c09aeb522fb1dd8bd76a6ec69175e WatchSource:0}: Error finding container 4a99ed0b6a0c80758733e756345678c1da2c09aeb522fb1dd8bd76a6ec69175e: Status 404 returned error can't find the container with id 4a99ed0b6a0c80758733e756345678c1da2c09aeb522fb1dd8bd76a6ec69175e
Jul 27 19:24:57.071872 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.071849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8b28b09-f556-4607-88de-8e687b695110-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm\" (UID: \"c8b28b09-f556-4607-88de-8e687b695110\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"
Jul 27 19:24:57.071998 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.071883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh7tj\" (UniqueName: \"kubernetes.io/projected/c8b28b09-f556-4607-88de-8e687b695110-kube-api-access-lh7tj\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm\" (UID: \"c8b28b09-f556-4607-88de-8e687b695110\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"
Jul 27 19:24:57.071998 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.071960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8b28b09-f556-4607-88de-8e687b695110-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm\" (UID: \"c8b28b09-f556-4607-88de-8e687b695110\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"
Jul 27 19:24:57.172630 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.172603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8b28b09-f556-4607-88de-8e687b695110-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm\" (UID: \"c8b28b09-f556-4607-88de-8e687b695110\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"
Jul 27 19:24:57.172788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.172669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8b28b09-f556-4607-88de-8e687b695110-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm\" (UID: \"c8b28b09-f556-4607-88de-8e687b695110\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"
Jul 27 19:24:57.172788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.172703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh7tj\" (UniqueName: \"kubernetes.io/projected/c8b28b09-f556-4607-88de-8e687b695110-kube-api-access-lh7tj\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm\" (UID: \"c8b28b09-f556-4607-88de-8e687b695110\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"
Jul 27 19:24:57.172988 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.172968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8b28b09-f556-4607-88de-8e687b695110-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm\" (UID: \"c8b28b09-f556-4607-88de-8e687b695110\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"
Jul 27 19:24:57.173046 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.172998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8b28b09-f556-4607-88de-8e687b695110-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm\" (UID: \"c8b28b09-f556-4607-88de-8e687b695110\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"
Jul 27 19:24:57.181012 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.180990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh7tj\" (UniqueName: \"kubernetes.io/projected/c8b28b09-f556-4607-88de-8e687b695110-kube-api-access-lh7tj\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm\" (UID: \"c8b28b09-f556-4607-88de-8e687b695110\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"
Jul 27 19:24:57.202950 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.202923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"
Jul 27 19:24:57.300047 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.299976    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"]
Jul 27 19:24:57.303753 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.303726    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"
Jul 27 19:24:57.314141 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.314119    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"]
Jul 27 19:24:57.328564 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.328545    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"]
Jul 27 19:24:57.330639 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:24:57.330621    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b28b09_f556_4607_88de_8e687b695110.slice/crio-a9aba19b4536af85c5bcddd3a445e977c3d7db04de9479a76f526bd2625efefd WatchSource:0}: Error finding container a9aba19b4536af85c5bcddd3a445e977c3d7db04de9479a76f526bd2625efefd: Status 404 returned error can't find the container with id a9aba19b4536af85c5bcddd3a445e977c3d7db04de9479a76f526bd2625efefd
Jul 27 19:24:57.474653 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.474626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2k5\" (UniqueName: \"kubernetes.io/projected/f47d2dd3-c40a-48ae-b792-986c76171d77-kube-api-access-5p2k5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw\" (UID: \"f47d2dd3-c40a-48ae-b792-986c76171d77\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"
Jul 27 19:24:57.474787 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.474673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47d2dd3-c40a-48ae-b792-986c76171d77-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw\" (UID: \"f47d2dd3-c40a-48ae-b792-986c76171d77\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"
Jul 27 19:24:57.474833 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.474786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47d2dd3-c40a-48ae-b792-986c76171d77-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw\" (UID: \"f47d2dd3-c40a-48ae-b792-986c76171d77\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"
Jul 27 19:24:57.575959 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.575921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47d2dd3-c40a-48ae-b792-986c76171d77-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw\" (UID: \"f47d2dd3-c40a-48ae-b792-986c76171d77\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"
Jul 27 19:24:57.576122 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.575987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2k5\" (UniqueName: \"kubernetes.io/projected/f47d2dd3-c40a-48ae-b792-986c76171d77-kube-api-access-5p2k5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw\" (UID: \"f47d2dd3-c40a-48ae-b792-986c76171d77\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"
Jul 27 19:24:57.576122 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.576027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47d2dd3-c40a-48ae-b792-986c76171d77-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw\" (UID: \"f47d2dd3-c40a-48ae-b792-986c76171d77\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"
Jul 27 19:24:57.576343 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.576327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47d2dd3-c40a-48ae-b792-986c76171d77-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw\" (UID: \"f47d2dd3-c40a-48ae-b792-986c76171d77\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"
Jul 27 19:24:57.576381 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.576362    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47d2dd3-c40a-48ae-b792-986c76171d77-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw\" (UID: \"f47d2dd3-c40a-48ae-b792-986c76171d77\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"
Jul 27 19:24:57.585366 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.585348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2k5\" (UniqueName: \"kubernetes.io/projected/f47d2dd3-c40a-48ae-b792-986c76171d77-kube-api-access-5p2k5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw\" (UID: \"f47d2dd3-c40a-48ae-b792-986c76171d77\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"
Jul 27 19:24:57.613339 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.613321    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"
Jul 27 19:24:57.766294 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.766269    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"]
Jul 27 19:24:57.767940 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:24:57.767914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47d2dd3_c40a_48ae_b792_986c76171d77.slice/crio-77635bbdf9ac1b1df84b88087000a860daafce4b17e9ec749bc5666870940bce WatchSource:0}: Error finding container 77635bbdf9ac1b1df84b88087000a860daafce4b17e9ec749bc5666870940bce: Status 404 returned error can't find the container with id 77635bbdf9ac1b1df84b88087000a860daafce4b17e9ec749bc5666870940bce
Jul 27 19:24:57.934567 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.934539    2580 generic.go:356] "Generic (PLEG): container finished" podID="2a2c055d-801f-4365-b9b8-487b48ef669b" containerID="c6f465251055788a05ba4533d0d62f062335f22ab2fea20fb512fe90fe178dff" exitCode=0
Jul 27 19:24:57.934707 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.934626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m" event={"ID":"2a2c055d-801f-4365-b9b8-487b48ef669b","Type":"ContainerDied","Data":"c6f465251055788a05ba4533d0d62f062335f22ab2fea20fb512fe90fe178dff"}
Jul 27 19:24:57.934707 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.934683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m" event={"ID":"2a2c055d-801f-4365-b9b8-487b48ef669b","Type":"ContainerStarted","Data":"4a99ed0b6a0c80758733e756345678c1da2c09aeb522fb1dd8bd76a6ec69175e"}
Jul 27 19:24:57.936106 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.936084    2580 generic.go:356] "Generic (PLEG): container finished" podID="f47d2dd3-c40a-48ae-b792-986c76171d77" containerID="23bc0aee28426958c9b7a2ec70b492e16de754d024104464edb2947d2860f1e6" exitCode=0
Jul 27 19:24:57.936202 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.936131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw" event={"ID":"f47d2dd3-c40a-48ae-b792-986c76171d77","Type":"ContainerDied","Data":"23bc0aee28426958c9b7a2ec70b492e16de754d024104464edb2947d2860f1e6"}
Jul 27 19:24:57.936202 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.936165    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw" event={"ID":"f47d2dd3-c40a-48ae-b792-986c76171d77","Type":"ContainerStarted","Data":"77635bbdf9ac1b1df84b88087000a860daafce4b17e9ec749bc5666870940bce"}
Jul 27 19:24:57.937779 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.937755    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8b28b09-f556-4607-88de-8e687b695110" containerID="f8232b7a1baf377da07efb29a768ce1b32ff6a17b6bb72e7abd6baf0e54cac38" exitCode=0
Jul 27 19:24:57.937878 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.937812    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm" event={"ID":"c8b28b09-f556-4607-88de-8e687b695110","Type":"ContainerDied","Data":"f8232b7a1baf377da07efb29a768ce1b32ff6a17b6bb72e7abd6baf0e54cac38"}
Jul 27 19:24:57.937878 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:57.937856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm" event={"ID":"c8b28b09-f556-4607-88de-8e687b695110","Type":"ContainerStarted","Data":"a9aba19b4536af85c5bcddd3a445e977c3d7db04de9479a76f526bd2625efefd"}
Jul 27 19:24:58.944617 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:58.944591    2580 generic.go:356] "Generic (PLEG): container finished" podID="f47d2dd3-c40a-48ae-b792-986c76171d77" containerID="fc4c7ef6f9a0b271b59a146911d7c0fd251f3ffb4ba99b9602162091dd6f3d67" exitCode=0
Jul 27 19:24:58.944955 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:58.944665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw" event={"ID":"f47d2dd3-c40a-48ae-b792-986c76171d77","Type":"ContainerDied","Data":"fc4c7ef6f9a0b271b59a146911d7c0fd251f3ffb4ba99b9602162091dd6f3d67"}
Jul 27 19:24:58.946303 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:58.946278    2580 generic.go:356] "Generic (PLEG): container finished" podID="1ed7bb6d-0419-4b71-bac9-833132273e22" containerID="2f696106ecd3c61612de0d6dbea8fcb93564ac1ca2e5f720faebdf8e9f258630" exitCode=0
Jul 27 19:24:58.946402 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:58.946347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7" event={"ID":"1ed7bb6d-0419-4b71-bac9-833132273e22","Type":"ContainerDied","Data":"2f696106ecd3c61612de0d6dbea8fcb93564ac1ca2e5f720faebdf8e9f258630"}
Jul 27 19:24:58.948003 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:58.947982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm" event={"ID":"c8b28b09-f556-4607-88de-8e687b695110","Type":"ContainerStarted","Data":"9c0df08abcd0f69c9e0dc2907810fd9dfbbeb38f91b089010d994b7ba43b4404"}
Jul 27 19:24:59.955191 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:59.955153    2580 generic.go:356] "Generic (PLEG): container finished" podID="1ed7bb6d-0419-4b71-bac9-833132273e22" containerID="281a843dc4c14f094c053f4b261d94b025745138ccb4df1c4e0561dc98716c98" exitCode=0
Jul 27 19:24:59.955603 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:59.955235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7" event={"ID":"1ed7bb6d-0419-4b71-bac9-833132273e22","Type":"ContainerDied","Data":"281a843dc4c14f094c053f4b261d94b025745138ccb4df1c4e0561dc98716c98"}
Jul 27 19:24:59.956998 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:59.956978    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8b28b09-f556-4607-88de-8e687b695110" containerID="9c0df08abcd0f69c9e0dc2907810fd9dfbbeb38f91b089010d994b7ba43b4404" exitCode=0
Jul 27 19:24:59.956998 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:59.956996    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8b28b09-f556-4607-88de-8e687b695110" containerID="1bb2fdf55769b1db86f64cd6f00c3b50a08c7142b50baf6ad412fd12c9cb848c" exitCode=0
Jul 27 19:24:59.957142 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:59.957102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm" event={"ID":"c8b28b09-f556-4607-88de-8e687b695110","Type":"ContainerDied","Data":"9c0df08abcd0f69c9e0dc2907810fd9dfbbeb38f91b089010d994b7ba43b4404"}
Jul 27 19:24:59.957193 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:59.957142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm" event={"ID":"c8b28b09-f556-4607-88de-8e687b695110","Type":"ContainerDied","Data":"1bb2fdf55769b1db86f64cd6f00c3b50a08c7142b50baf6ad412fd12c9cb848c"}
Jul 27 19:24:59.958574 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:59.958558    2580 generic.go:356] "Generic (PLEG): container finished" podID="2a2c055d-801f-4365-b9b8-487b48ef669b" containerID="9ccc37bad966289c3e9671349a33c28f5391c7ae2bd7f038a0189f34d55ea859" exitCode=0
Jul 27 19:24:59.958658 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:59.958619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m" event={"ID":"2a2c055d-801f-4365-b9b8-487b48ef669b","Type":"ContainerDied","Data":"9ccc37bad966289c3e9671349a33c28f5391c7ae2bd7f038a0189f34d55ea859"}
Jul 27 19:24:59.960667 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:59.960650    2580 generic.go:356] "Generic (PLEG): container finished" podID="f47d2dd3-c40a-48ae-b792-986c76171d77" containerID="caf50deac6cde4a2e7e6b3c2181643e79bcd94610d6b59f44371820ed1a30731" exitCode=0
Jul 27 19:24:59.960749 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:24:59.960684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw" event={"ID":"f47d2dd3-c40a-48ae-b792-986c76171d77","Type":"ContainerDied","Data":"caf50deac6cde4a2e7e6b3c2181643e79bcd94610d6b59f44371820ed1a30731"}
Jul 27 19:25:00.967424 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:00.967390    2580 generic.go:356] "Generic (PLEG): container finished" podID="2a2c055d-801f-4365-b9b8-487b48ef669b" containerID="ad159cc2fc8e46c7f5d293af277fea36b7bdfbf5f3362b248b635efa5ec42f14" exitCode=0
Jul 27 19:25:00.967821 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:00.967466    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m" event={"ID":"2a2c055d-801f-4365-b9b8-487b48ef669b","Type":"ContainerDied","Data":"ad159cc2fc8e46c7f5d293af277fea36b7bdfbf5f3362b248b635efa5ec42f14"}
Jul 27 19:25:01.105445 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.105419    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"
Jul 27 19:25:01.143014 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.142987    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"
Jul 27 19:25:01.152038 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.152022    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"
Jul 27 19:25:01.208044 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.208017    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p2k5\" (UniqueName: \"kubernetes.io/projected/f47d2dd3-c40a-48ae-b792-986c76171d77-kube-api-access-5p2k5\") pod \"f47d2dd3-c40a-48ae-b792-986c76171d77\" (UID: \"f47d2dd3-c40a-48ae-b792-986c76171d77\") "
Jul 27 19:25:01.208194 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.208049    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47d2dd3-c40a-48ae-b792-986c76171d77-util\") pod \"f47d2dd3-c40a-48ae-b792-986c76171d77\" (UID: \"f47d2dd3-c40a-48ae-b792-986c76171d77\") "
Jul 27 19:25:01.208194 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.208082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47d2dd3-c40a-48ae-b792-986c76171d77-bundle\") pod \"f47d2dd3-c40a-48ae-b792-986c76171d77\" (UID: \"f47d2dd3-c40a-48ae-b792-986c76171d77\") "
Jul 27 19:25:01.208892 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.208861    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47d2dd3-c40a-48ae-b792-986c76171d77-bundle" (OuterVolumeSpecName: "bundle") pod "f47d2dd3-c40a-48ae-b792-986c76171d77" (UID: "f47d2dd3-c40a-48ae-b792-986c76171d77"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:25:01.210399 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.210378    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47d2dd3-c40a-48ae-b792-986c76171d77-kube-api-access-5p2k5" (OuterVolumeSpecName: "kube-api-access-5p2k5") pod "f47d2dd3-c40a-48ae-b792-986c76171d77" (UID: "f47d2dd3-c40a-48ae-b792-986c76171d77"). InnerVolumeSpecName "kube-api-access-5p2k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:25:01.214016 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.213994    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47d2dd3-c40a-48ae-b792-986c76171d77-util" (OuterVolumeSpecName: "util") pod "f47d2dd3-c40a-48ae-b792-986c76171d77" (UID: "f47d2dd3-c40a-48ae-b792-986c76171d77"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:25:01.309411 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.309386    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8p86\" (UniqueName: \"kubernetes.io/projected/1ed7bb6d-0419-4b71-bac9-833132273e22-kube-api-access-f8p86\") pod \"1ed7bb6d-0419-4b71-bac9-833132273e22\" (UID: \"1ed7bb6d-0419-4b71-bac9-833132273e22\") "
Jul 27 19:25:01.309551 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.309425    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ed7bb6d-0419-4b71-bac9-833132273e22-bundle\") pod \"1ed7bb6d-0419-4b71-bac9-833132273e22\" (UID: \"1ed7bb6d-0419-4b71-bac9-833132273e22\") "
Jul 27 19:25:01.309551 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.309454    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ed7bb6d-0419-4b71-bac9-833132273e22-util\") pod \"1ed7bb6d-0419-4b71-bac9-833132273e22\" (UID: \"1ed7bb6d-0419-4b71-bac9-833132273e22\") "
Jul 27 19:25:01.309551 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.309472    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh7tj\" (UniqueName: \"kubernetes.io/projected/c8b28b09-f556-4607-88de-8e687b695110-kube-api-access-lh7tj\") pod \"c8b28b09-f556-4607-88de-8e687b695110\" (UID: \"c8b28b09-f556-4607-88de-8e687b695110\") "
Jul 27 19:25:01.309551 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.309491    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8b28b09-f556-4607-88de-8e687b695110-bundle\") pod \"c8b28b09-f556-4607-88de-8e687b695110\" (UID: \"c8b28b09-f556-4607-88de-8e687b695110\") "
Jul 27 19:25:01.309761 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.309559    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8b28b09-f556-4607-88de-8e687b695110-util\") pod \"c8b28b09-f556-4607-88de-8e687b695110\" (UID: \"c8b28b09-f556-4607-88de-8e687b695110\") "
Jul 27 19:25:01.309813 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.309803    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p2k5\" (UniqueName: \"kubernetes.io/projected/f47d2dd3-c40a-48ae-b792-986c76171d77-kube-api-access-5p2k5\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:25:01.309878 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.309820    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47d2dd3-c40a-48ae-b792-986c76171d77-util\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:25:01.309878 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.309833    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47d2dd3-c40a-48ae-b792-986c76171d77-bundle\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:25:01.310069 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.310022    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b28b09-f556-4607-88de-8e687b695110-bundle" (OuterVolumeSpecName: "bundle") pod "c8b28b09-f556-4607-88de-8e687b695110" (UID: "c8b28b09-f556-4607-88de-8e687b695110"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:25:01.310069 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.310054    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed7bb6d-0419-4b71-bac9-833132273e22-bundle" (OuterVolumeSpecName: "bundle") pod "1ed7bb6d-0419-4b71-bac9-833132273e22" (UID: "1ed7bb6d-0419-4b71-bac9-833132273e22"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:25:01.311590 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.311558    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed7bb6d-0419-4b71-bac9-833132273e22-kube-api-access-f8p86" (OuterVolumeSpecName: "kube-api-access-f8p86") pod "1ed7bb6d-0419-4b71-bac9-833132273e22" (UID: "1ed7bb6d-0419-4b71-bac9-833132273e22"). InnerVolumeSpecName "kube-api-access-f8p86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:25:01.311675 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.311587    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b28b09-f556-4607-88de-8e687b695110-kube-api-access-lh7tj" (OuterVolumeSpecName: "kube-api-access-lh7tj") pod "c8b28b09-f556-4607-88de-8e687b695110" (UID: "c8b28b09-f556-4607-88de-8e687b695110"). InnerVolumeSpecName "kube-api-access-lh7tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:25:01.314348 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.314313    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed7bb6d-0419-4b71-bac9-833132273e22-util" (OuterVolumeSpecName: "util") pod "1ed7bb6d-0419-4b71-bac9-833132273e22" (UID: "1ed7bb6d-0419-4b71-bac9-833132273e22"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:25:01.314994 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.314977    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b28b09-f556-4607-88de-8e687b695110-util" (OuterVolumeSpecName: "util") pod "c8b28b09-f556-4607-88de-8e687b695110" (UID: "c8b28b09-f556-4607-88de-8e687b695110"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:25:01.411192 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.411164    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8b28b09-f556-4607-88de-8e687b695110-util\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:25:01.411192 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.411188    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8p86\" (UniqueName: \"kubernetes.io/projected/1ed7bb6d-0419-4b71-bac9-833132273e22-kube-api-access-f8p86\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:25:01.411192 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.411199    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ed7bb6d-0419-4b71-bac9-833132273e22-bundle\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:25:01.411397 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.411207    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ed7bb6d-0419-4b71-bac9-833132273e22-util\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:25:01.411397 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.411214    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh7tj\" (UniqueName: \"kubernetes.io/projected/c8b28b09-f556-4607-88de-8e687b695110-kube-api-access-lh7tj\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:25:01.411397 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.411222    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8b28b09-f556-4607-88de-8e687b695110-bundle\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:25:01.973939 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.973907    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw"
Jul 27 19:25:01.973939 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.973916    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw" event={"ID":"f47d2dd3-c40a-48ae-b792-986c76171d77","Type":"ContainerDied","Data":"77635bbdf9ac1b1df84b88087000a860daafce4b17e9ec749bc5666870940bce"}
Jul 27 19:25:01.974389 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.973963    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77635bbdf9ac1b1df84b88087000a860daafce4b17e9ec749bc5666870940bce"
Jul 27 19:25:01.975745 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.975722    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7" event={"ID":"1ed7bb6d-0419-4b71-bac9-833132273e22","Type":"ContainerDied","Data":"fdb31496b2ba2db5281f241299d4b6624474c987799f996acf203bc899662305"}
Jul 27 19:25:01.975858 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.975755    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdb31496b2ba2db5281f241299d4b6624474c987799f996acf203bc899662305"
Jul 27 19:25:01.975858 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.975729    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7"
Jul 27 19:25:01.977701 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.977686    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm"
Jul 27 19:25:01.977806 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.977727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm" event={"ID":"c8b28b09-f556-4607-88de-8e687b695110","Type":"ContainerDied","Data":"a9aba19b4536af85c5bcddd3a445e977c3d7db04de9479a76f526bd2625efefd"}
Jul 27 19:25:01.977806 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:01.977761    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9aba19b4536af85c5bcddd3a445e977c3d7db04de9479a76f526bd2625efefd"
Jul 27 19:25:02.103601 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:02.103580    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"
Jul 27 19:25:02.218650 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:02.218610    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a2c055d-801f-4365-b9b8-487b48ef669b-util\") pod \"2a2c055d-801f-4365-b9b8-487b48ef669b\" (UID: \"2a2c055d-801f-4365-b9b8-487b48ef669b\") "
Jul 27 19:25:02.218821 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:02.218679    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a2c055d-801f-4365-b9b8-487b48ef669b-bundle\") pod \"2a2c055d-801f-4365-b9b8-487b48ef669b\" (UID: \"2a2c055d-801f-4365-b9b8-487b48ef669b\") "
Jul 27 19:25:02.218821 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:02.218741    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k569q\" (UniqueName: \"kubernetes.io/projected/2a2c055d-801f-4365-b9b8-487b48ef669b-kube-api-access-k569q\") pod \"2a2c055d-801f-4365-b9b8-487b48ef669b\" (UID: \"2a2c055d-801f-4365-b9b8-487b48ef669b\") "
Jul 27 19:25:02.219193 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:02.219168    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2c055d-801f-4365-b9b8-487b48ef669b-bundle" (OuterVolumeSpecName: "bundle") pod "2a2c055d-801f-4365-b9b8-487b48ef669b" (UID: "2a2c055d-801f-4365-b9b8-487b48ef669b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:25:02.224305 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:02.221837    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2c055d-801f-4365-b9b8-487b48ef669b-kube-api-access-k569q" (OuterVolumeSpecName: "kube-api-access-k569q") pod "2a2c055d-801f-4365-b9b8-487b48ef669b" (UID: "2a2c055d-801f-4365-b9b8-487b48ef669b"). InnerVolumeSpecName "kube-api-access-k569q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:25:02.227079 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:02.227050    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2c055d-801f-4365-b9b8-487b48ef669b-util" (OuterVolumeSpecName: "util") pod "2a2c055d-801f-4365-b9b8-487b48ef669b" (UID: "2a2c055d-801f-4365-b9b8-487b48ef669b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:25:02.319999 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:02.319958    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k569q\" (UniqueName: \"kubernetes.io/projected/2a2c055d-801f-4365-b9b8-487b48ef669b-kube-api-access-k569q\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:25:02.319999 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:02.319995    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a2c055d-801f-4365-b9b8-487b48ef669b-util\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:25:02.320150 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:02.320010    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a2c055d-801f-4365-b9b8-487b48ef669b-bundle\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:25:02.984065 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:02.984037    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m"
Jul 27 19:25:02.984485 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:02.984041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m" event={"ID":"2a2c055d-801f-4365-b9b8-487b48ef669b","Type":"ContainerDied","Data":"4a99ed0b6a0c80758733e756345678c1da2c09aeb522fb1dd8bd76a6ec69175e"}
Jul 27 19:25:02.984485 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:02.984166    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a99ed0b6a0c80758733e756345678c1da2c09aeb522fb1dd8bd76a6ec69175e"
Jul 27 19:25:40.757533 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.757497    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sdjnc"]
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.757904    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8b28b09-f556-4607-88de-8e687b695110" containerName="util"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.757921    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8b28b09-f556-4607-88de-8e687b695110" containerName="util"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.757932    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47d2dd3-c40a-48ae-b792-986c76171d77" containerName="extract"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.757941    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47d2dd3-c40a-48ae-b792-986c76171d77" containerName="extract"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.757955    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2c055d-801f-4365-b9b8-487b48ef669b" containerName="pull"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.757963    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2c055d-801f-4365-b9b8-487b48ef669b" containerName="pull"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.757980    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ed7bb6d-0419-4b71-bac9-833132273e22" containerName="util"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.757988    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ed7bb6d-0419-4b71-bac9-833132273e22" containerName="util"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.757996    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2c055d-801f-4365-b9b8-487b48ef669b" containerName="extract"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758006    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2c055d-801f-4365-b9b8-487b48ef669b" containerName="extract"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758016    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2c055d-801f-4365-b9b8-487b48ef669b" containerName="util"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758024    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2c055d-801f-4365-b9b8-487b48ef669b" containerName="util"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758039    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47d2dd3-c40a-48ae-b792-986c76171d77" containerName="util"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758046    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47d2dd3-c40a-48ae-b792-986c76171d77" containerName="util"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758056    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ed7bb6d-0419-4b71-bac9-833132273e22" containerName="pull"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758063    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ed7bb6d-0419-4b71-bac9-833132273e22" containerName="pull"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758072    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47d2dd3-c40a-48ae-b792-986c76171d77" containerName="pull"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758080    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47d2dd3-c40a-48ae-b792-986c76171d77" containerName="pull"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758092    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8b28b09-f556-4607-88de-8e687b695110" containerName="extract"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758098    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8b28b09-f556-4607-88de-8e687b695110" containerName="extract"
Jul 27 19:25:40.758101 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758109    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8b28b09-f556-4607-88de-8e687b695110" containerName="pull"
Jul 27 19:25:40.759065 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758117    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8b28b09-f556-4607-88de-8e687b695110" containerName="pull"
Jul 27 19:25:40.759065 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758124    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ed7bb6d-0419-4b71-bac9-833132273e22" containerName="extract"
Jul 27 19:25:40.759065 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758131    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ed7bb6d-0419-4b71-bac9-833132273e22" containerName="extract"
Jul 27 19:25:40.759065 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758209    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ed7bb6d-0419-4b71-bac9-833132273e22" containerName="extract"
Jul 27 19:25:40.759065 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758221    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47d2dd3-c40a-48ae-b792-986c76171d77" containerName="extract"
Jul 27 19:25:40.759065 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758230    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8b28b09-f556-4607-88de-8e687b695110" containerName="extract"
Jul 27 19:25:40.759065 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.758262    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a2c055d-801f-4365-b9b8-487b48ef669b" containerName="extract"
Jul 27 19:25:40.763042 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.763022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sdjnc"
Jul 27 19:25:40.765443 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.765423    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 19:25:40.765596 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.765478    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tlnv2\""
Jul 27 19:25:40.765596 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.765429    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 19:25:40.782027 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.782002    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sdjnc"]
Jul 27 19:25:40.812707 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.812675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt4mt\" (UniqueName: \"kubernetes.io/projected/d715763c-b7ef-4584-ac16-c78e14462f37-kube-api-access-zt4mt\") pod \"limitador-operator-controller-manager-859d7b78bd-sdjnc\" (UID: \"d715763c-b7ef-4584-ac16-c78e14462f37\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sdjnc"
Jul 27 19:25:40.913357 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.913325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt4mt\" (UniqueName: \"kubernetes.io/projected/d715763c-b7ef-4584-ac16-c78e14462f37-kube-api-access-zt4mt\") pod \"limitador-operator-controller-manager-859d7b78bd-sdjnc\" (UID: \"d715763c-b7ef-4584-ac16-c78e14462f37\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sdjnc"
Jul 27 19:25:40.925984 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:40.925953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt4mt\" (UniqueName: \"kubernetes.io/projected/d715763c-b7ef-4584-ac16-c78e14462f37-kube-api-access-zt4mt\") pod \"limitador-operator-controller-manager-859d7b78bd-sdjnc\" (UID: \"d715763c-b7ef-4584-ac16-c78e14462f37\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sdjnc"
Jul 27 19:25:41.073013 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:41.072986    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sdjnc"
Jul 27 19:25:41.217531 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:41.217504    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sdjnc"]
Jul 27 19:25:41.219269 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:25:41.219212    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd715763c_b7ef_4584_ac16_c78e14462f37.slice/crio-f6f49b88e08e1c722fe427125567caeec82c0039cfbedc2409ef389ef29cd3dc WatchSource:0}: Error finding container f6f49b88e08e1c722fe427125567caeec82c0039cfbedc2409ef389ef29cd3dc: Status 404 returned error can't find the container with id f6f49b88e08e1c722fe427125567caeec82c0039cfbedc2409ef389ef29cd3dc
Jul 27 19:25:42.171386 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:42.171341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sdjnc" event={"ID":"d715763c-b7ef-4584-ac16-c78e14462f37","Type":"ContainerStarted","Data":"f6f49b88e08e1c722fe427125567caeec82c0039cfbedc2409ef389ef29cd3dc"}
Jul 27 19:25:43.177759 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:43.177672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sdjnc" event={"ID":"d715763c-b7ef-4584-ac16-c78e14462f37","Type":"ContainerStarted","Data":"45ce48cad0ff86e5366f15c15bb0864b769723e5f6b0b23bfc25aa0ad64b931f"}
Jul 27 19:25:43.178100 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:43.177787    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sdjnc"
Jul 27 19:25:43.201908 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:43.201863    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sdjnc" podStartSLOduration=1.561119749 podStartE2EDuration="3.201845937s" totalImagesPullingTime="1.640726188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:43.201813402 +0000 UTC m=+485.717517906" watchObservedRunningTime="2026-07-27 19:25:43.201845937 +0000 UTC m=+485.717550441"
Jul 27 19:25:54.183699 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:25:54.183666    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sdjnc"
Jul 27 19:26:53.274519 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.274483    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-jddwl"]
Jul 27 19:26:53.277746 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.277724    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jddwl"
Jul 27 19:26:53.280702 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.280679    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fzjq4\""
Jul 27 19:26:53.289189 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.289169    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jddwl"]
Jul 27 19:26:53.415705 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.415677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxvt\" (UniqueName: \"kubernetes.io/projected/1b327c79-75dc-4be9-967f-163a606b504e-kube-api-access-smxvt\") pod \"authorino-58cd955675-jddwl\" (UID: \"1b327c79-75dc-4be9-967f-163a606b504e\") " pod="kuadrant-system/authorino-58cd955675-jddwl"
Jul 27 19:26:53.516608 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.516578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxvt\" (UniqueName: \"kubernetes.io/projected/1b327c79-75dc-4be9-967f-163a606b504e-kube-api-access-smxvt\") pod \"authorino-58cd955675-jddwl\" (UID: \"1b327c79-75dc-4be9-967f-163a606b504e\") " pod="kuadrant-system/authorino-58cd955675-jddwl"
Jul 27 19:26:53.535079 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.535015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxvt\" (UniqueName: \"kubernetes.io/projected/1b327c79-75dc-4be9-967f-163a606b504e-kube-api-access-smxvt\") pod \"authorino-58cd955675-jddwl\" (UID: \"1b327c79-75dc-4be9-967f-163a606b504e\") " pod="kuadrant-system/authorino-58cd955675-jddwl"
Jul 27 19:26:53.586814 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.586782    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jddwl"
Jul 27 19:26:53.617079 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.617049    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-lzz67"]
Jul 27 19:26:53.620466 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.620443    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lzz67"
Jul 27 19:26:53.629670 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.629642    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lzz67"]
Jul 27 19:26:53.718658 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.718623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mldl2\" (UniqueName: \"kubernetes.io/projected/c15070dd-7d4d-43c8-af64-1f4782bd0578-kube-api-access-mldl2\") pod \"authorino-75d556bd77-lzz67\" (UID: \"c15070dd-7d4d-43c8-af64-1f4782bd0578\") " pod="kuadrant-system/authorino-75d556bd77-lzz67"
Jul 27 19:26:53.719818 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.719703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jddwl"]
Jul 27 19:26:53.722403 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:26:53.722376    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b327c79_75dc_4be9_967f_163a606b504e.slice/crio-bdcd83cebea498d3e72ee7046a9ecc42e9672ee6e93e2cd91bf5cfa577fa2388 WatchSource:0}: Error finding container bdcd83cebea498d3e72ee7046a9ecc42e9672ee6e93e2cd91bf5cfa577fa2388: Status 404 returned error can't find the container with id bdcd83cebea498d3e72ee7046a9ecc42e9672ee6e93e2cd91bf5cfa577fa2388
Jul 27 19:26:53.819389 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.819365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mldl2\" (UniqueName: \"kubernetes.io/projected/c15070dd-7d4d-43c8-af64-1f4782bd0578-kube-api-access-mldl2\") pod \"authorino-75d556bd77-lzz67\" (UID: \"c15070dd-7d4d-43c8-af64-1f4782bd0578\") " pod="kuadrant-system/authorino-75d556bd77-lzz67"
Jul 27 19:26:53.829033 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.829012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mldl2\" (UniqueName: \"kubernetes.io/projected/c15070dd-7d4d-43c8-af64-1f4782bd0578-kube-api-access-mldl2\") pod \"authorino-75d556bd77-lzz67\" (UID: \"c15070dd-7d4d-43c8-af64-1f4782bd0578\") " pod="kuadrant-system/authorino-75d556bd77-lzz67"
Jul 27 19:26:53.930132 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:53.930100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lzz67"
Jul 27 19:26:54.063598 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:54.063569    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lzz67"]
Jul 27 19:26:54.067491 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:26:54.067465    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc15070dd_7d4d_43c8_af64_1f4782bd0578.slice/crio-02fa8df72e325485400df44a485702765d0e0f67faa2d26159b654d79b3668d9 WatchSource:0}: Error finding container 02fa8df72e325485400df44a485702765d0e0f67faa2d26159b654d79b3668d9: Status 404 returned error can't find the container with id 02fa8df72e325485400df44a485702765d0e0f67faa2d26159b654d79b3668d9
Jul 27 19:26:54.500802 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:54.500724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lzz67" event={"ID":"c15070dd-7d4d-43c8-af64-1f4782bd0578","Type":"ContainerStarted","Data":"02fa8df72e325485400df44a485702765d0e0f67faa2d26159b654d79b3668d9"}
Jul 27 19:26:54.501890 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:54.501868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jddwl" event={"ID":"1b327c79-75dc-4be9-967f-163a606b504e","Type":"ContainerStarted","Data":"bdcd83cebea498d3e72ee7046a9ecc42e9672ee6e93e2cd91bf5cfa577fa2388"}
Jul 27 19:26:57.521590 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:57.521525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jddwl" event={"ID":"1b327c79-75dc-4be9-967f-163a606b504e","Type":"ContainerStarted","Data":"612676f00b49aa778ccac8e1e54774335016168ce96ecdb7261e6d2f9f14437b"}
Jul 27 19:26:57.523854 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:57.523809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lzz67" event={"ID":"c15070dd-7d4d-43c8-af64-1f4782bd0578","Type":"ContainerStarted","Data":"bf4d9bbb95b20075acfc9892bdde86b749d4c1a2553d0d385337c2d6444f81bc"}
Jul 27 19:26:57.553957 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:57.553907    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-jddwl" podStartSLOduration=0.94242691 podStartE2EDuration="4.553891828s" totalImagesPullingTime="3.611464918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:26:57.553326864 +0000 UTC m=+560.069031369" watchObservedRunningTime="2026-07-27 19:26:57.553891828 +0000 UTC m=+560.069596331"
Jul 27 19:26:57.582643 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:57.582586    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-lzz67" podStartSLOduration=1.302511283 podStartE2EDuration="4.582570979s" totalImagesPullingTime="3.280059696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:26:57.580017287 +0000 UTC m=+560.095721790" watchObservedRunningTime="2026-07-27 19:26:57.582570979 +0000 UTC m=+560.098275482"
Jul 27 19:26:57.618271 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:57.618163    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jddwl"]
Jul 27 19:26:59.533113 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:59.533075    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-jddwl" podUID="1b327c79-75dc-4be9-967f-163a606b504e" containerName="authorino" containerID="cri-o://612676f00b49aa778ccac8e1e54774335016168ce96ecdb7261e6d2f9f14437b" gracePeriod=30
Jul 27 19:26:59.776137 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:59.776112    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jddwl"
Jul 27 19:26:59.866382 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:59.866354    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smxvt\" (UniqueName: \"kubernetes.io/projected/1b327c79-75dc-4be9-967f-163a606b504e-kube-api-access-smxvt\") pod \"1b327c79-75dc-4be9-967f-163a606b504e\" (UID: \"1b327c79-75dc-4be9-967f-163a606b504e\") "
Jul 27 19:26:59.868380 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:59.868346    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b327c79-75dc-4be9-967f-163a606b504e-kube-api-access-smxvt" (OuterVolumeSpecName: "kube-api-access-smxvt") pod "1b327c79-75dc-4be9-967f-163a606b504e" (UID: "1b327c79-75dc-4be9-967f-163a606b504e"). InnerVolumeSpecName "kube-api-access-smxvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:26:59.967228 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:26:59.967195    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smxvt\" (UniqueName: \"kubernetes.io/projected/1b327c79-75dc-4be9-967f-163a606b504e-kube-api-access-smxvt\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:27:00.539095 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:00.539059    2580 generic.go:356] "Generic (PLEG): container finished" podID="1b327c79-75dc-4be9-967f-163a606b504e" containerID="612676f00b49aa778ccac8e1e54774335016168ce96ecdb7261e6d2f9f14437b" exitCode=0
Jul 27 19:27:00.539531 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:00.539102    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jddwl"
Jul 27 19:27:00.539531 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:00.539107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jddwl" event={"ID":"1b327c79-75dc-4be9-967f-163a606b504e","Type":"ContainerDied","Data":"612676f00b49aa778ccac8e1e54774335016168ce96ecdb7261e6d2f9f14437b"}
Jul 27 19:27:00.539531 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:00.539139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jddwl" event={"ID":"1b327c79-75dc-4be9-967f-163a606b504e","Type":"ContainerDied","Data":"bdcd83cebea498d3e72ee7046a9ecc42e9672ee6e93e2cd91bf5cfa577fa2388"}
Jul 27 19:27:00.539531 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:00.539155    2580 scope.go:117] "RemoveContainer" containerID="612676f00b49aa778ccac8e1e54774335016168ce96ecdb7261e6d2f9f14437b"
Jul 27 19:27:00.547577 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:00.547557    2580 scope.go:117] "RemoveContainer" containerID="612676f00b49aa778ccac8e1e54774335016168ce96ecdb7261e6d2f9f14437b"
Jul 27 19:27:00.547855 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:27:00.547831    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612676f00b49aa778ccac8e1e54774335016168ce96ecdb7261e6d2f9f14437b\": container with ID starting with 612676f00b49aa778ccac8e1e54774335016168ce96ecdb7261e6d2f9f14437b not found: ID does not exist" containerID="612676f00b49aa778ccac8e1e54774335016168ce96ecdb7261e6d2f9f14437b"
Jul 27 19:27:00.547945 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:00.547862    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612676f00b49aa778ccac8e1e54774335016168ce96ecdb7261e6d2f9f14437b"} err="failed to get container status \"612676f00b49aa778ccac8e1e54774335016168ce96ecdb7261e6d2f9f14437b\": rpc error: code = NotFound desc = could not find container \"612676f00b49aa778ccac8e1e54774335016168ce96ecdb7261e6d2f9f14437b\": container with ID starting with 612676f00b49aa778ccac8e1e54774335016168ce96ecdb7261e6d2f9f14437b not found: ID does not exist"
Jul 27 19:27:00.555916 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:00.555893    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jddwl"]
Jul 27 19:27:00.559414 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:00.559392    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-jddwl"]
Jul 27 19:27:02.086004 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:02.085969    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b327c79-75dc-4be9-967f-163a606b504e" path="/var/lib/kubelet/pods/1b327c79-75dc-4be9-967f-163a606b504e/volumes"
Jul 27 19:27:26.313761 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.313729    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sps46"]
Jul 27 19:27:26.314256 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.314052    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b327c79-75dc-4be9-967f-163a606b504e" containerName="authorino"
Jul 27 19:27:26.314256 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.314064    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b327c79-75dc-4be9-967f-163a606b504e" containerName="authorino"
Jul 27 19:27:26.314256 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.314122    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b327c79-75dc-4be9-967f-163a606b504e" containerName="authorino"
Jul 27 19:27:26.318637 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.318619    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sps46"
Jul 27 19:27:26.324064 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.324040    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sps46"]
Jul 27 19:27:26.377834 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.377796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcc5p\" (UniqueName: \"kubernetes.io/projected/5fbd41fc-5776-424a-9652-8738061641d1-kube-api-access-wcc5p\") pod \"authorino-5d4b67c446-sps46\" (UID: \"5fbd41fc-5776-424a-9652-8738061641d1\") " pod="kuadrant-system/authorino-5d4b67c446-sps46"
Jul 27 19:27:26.478609 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.478569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcc5p\" (UniqueName: \"kubernetes.io/projected/5fbd41fc-5776-424a-9652-8738061641d1-kube-api-access-wcc5p\") pod \"authorino-5d4b67c446-sps46\" (UID: \"5fbd41fc-5776-424a-9652-8738061641d1\") " pod="kuadrant-system/authorino-5d4b67c446-sps46"
Jul 27 19:27:26.488706 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.488680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcc5p\" (UniqueName: \"kubernetes.io/projected/5fbd41fc-5776-424a-9652-8738061641d1-kube-api-access-wcc5p\") pod \"authorino-5d4b67c446-sps46\" (UID: \"5fbd41fc-5776-424a-9652-8738061641d1\") " pod="kuadrant-system/authorino-5d4b67c446-sps46"
Jul 27 19:27:26.556778 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.556742    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sps46"]
Jul 27 19:27:26.556993 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.556981    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sps46"
Jul 27 19:27:26.579007 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.578977    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-695488d46b-wpmgb"]
Jul 27 19:27:26.582493 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.582477    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-695488d46b-wpmgb"
Jul 27 19:27:26.590904 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.590873    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-695488d46b-wpmgb"]
Jul 27 19:27:26.679766 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.679734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznj2\" (UniqueName: \"kubernetes.io/projected/0d569c2c-99d9-4570-95c5-5a309d98033c-kube-api-access-qznj2\") pod \"authorino-695488d46b-wpmgb\" (UID: \"0d569c2c-99d9-4570-95c5-5a309d98033c\") " pod="kuadrant-system/authorino-695488d46b-wpmgb"
Jul 27 19:27:26.694165 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.694142    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sps46"]
Jul 27 19:27:26.695765 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:27:26.695738    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fbd41fc_5776_424a_9652_8738061641d1.slice/crio-9e315888188bd530adcba4aa277e2529d87a76037533cee1b408ca9af429493e WatchSource:0}: Error finding container 9e315888188bd530adcba4aa277e2529d87a76037533cee1b408ca9af429493e: Status 404 returned error can't find the container with id 9e315888188bd530adcba4aa277e2529d87a76037533cee1b408ca9af429493e
Jul 27 19:27:26.780977 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.780944    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-695488d46b-wpmgb"]
Jul 27 19:27:26.781151 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.781101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznj2\" (UniqueName: \"kubernetes.io/projected/0d569c2c-99d9-4570-95c5-5a309d98033c-kube-api-access-qznj2\") pod \"authorino-695488d46b-wpmgb\" (UID: \"0d569c2c-99d9-4570-95c5-5a309d98033c\") " pod="kuadrant-system/authorino-695488d46b-wpmgb"
Jul 27 19:27:26.781200 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:27:26.781154    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qznj2], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-695488d46b-wpmgb" podUID="0d569c2c-99d9-4570-95c5-5a309d98033c"
Jul 27 19:27:26.797510 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.797477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznj2\" (UniqueName: \"kubernetes.io/projected/0d569c2c-99d9-4570-95c5-5a309d98033c-kube-api-access-qznj2\") pod \"authorino-695488d46b-wpmgb\" (UID: \"0d569c2c-99d9-4570-95c5-5a309d98033c\") " pod="kuadrant-system/authorino-695488d46b-wpmgb"
Jul 27 19:27:26.825184 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.825118    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6698cdf7c8-plg8k"]
Jul 27 19:27:26.828462 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.828446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6698cdf7c8-plg8k"
Jul 27 19:27:26.830952 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.830932    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 27 19:27:26.841009 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.839450    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6698cdf7c8-plg8k"]
Jul 27 19:27:26.882041 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.881988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a8634ff-bf39-4217-b391-e5c345767bb1-tls-cert\") pod \"authorino-6698cdf7c8-plg8k\" (UID: \"5a8634ff-bf39-4217-b391-e5c345767bb1\") " pod="kuadrant-system/authorino-6698cdf7c8-plg8k"
Jul 27 19:27:26.882213 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.882154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtd58\" (UniqueName: \"kubernetes.io/projected/5a8634ff-bf39-4217-b391-e5c345767bb1-kube-api-access-jtd58\") pod \"authorino-6698cdf7c8-plg8k\" (UID: \"5a8634ff-bf39-4217-b391-e5c345767bb1\") " pod="kuadrant-system/authorino-6698cdf7c8-plg8k"
Jul 27 19:27:26.983154 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.983115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtd58\" (UniqueName: \"kubernetes.io/projected/5a8634ff-bf39-4217-b391-e5c345767bb1-kube-api-access-jtd58\") pod \"authorino-6698cdf7c8-plg8k\" (UID: \"5a8634ff-bf39-4217-b391-e5c345767bb1\") " pod="kuadrant-system/authorino-6698cdf7c8-plg8k"
Jul 27 19:27:26.983154 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.983154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a8634ff-bf39-4217-b391-e5c345767bb1-tls-cert\") pod \"authorino-6698cdf7c8-plg8k\" (UID: \"5a8634ff-bf39-4217-b391-e5c345767bb1\") " pod="kuadrant-system/authorino-6698cdf7c8-plg8k"
Jul 27 19:27:26.985605 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.985579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a8634ff-bf39-4217-b391-e5c345767bb1-tls-cert\") pod \"authorino-6698cdf7c8-plg8k\" (UID: \"5a8634ff-bf39-4217-b391-e5c345767bb1\") " pod="kuadrant-system/authorino-6698cdf7c8-plg8k"
Jul 27 19:27:26.991747 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:26.991728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtd58\" (UniqueName: \"kubernetes.io/projected/5a8634ff-bf39-4217-b391-e5c345767bb1-kube-api-access-jtd58\") pod \"authorino-6698cdf7c8-plg8k\" (UID: \"5a8634ff-bf39-4217-b391-e5c345767bb1\") " pod="kuadrant-system/authorino-6698cdf7c8-plg8k"
Jul 27 19:27:27.137967 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.137943    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6698cdf7c8-plg8k"
Jul 27 19:27:27.283048 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.282977    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6698cdf7c8-plg8k"]
Jul 27 19:27:27.285080 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:27:27.285053    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a8634ff_bf39_4217_b391_e5c345767bb1.slice/crio-d9e95d32e447a9789a5b3794d2dcdd8a5ebefc08b9426d6b599fdcc996a1c140 WatchSource:0}: Error finding container d9e95d32e447a9789a5b3794d2dcdd8a5ebefc08b9426d6b599fdcc996a1c140: Status 404 returned error can't find the container with id d9e95d32e447a9789a5b3794d2dcdd8a5ebefc08b9426d6b599fdcc996a1c140
Jul 27 19:27:27.670400 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.670373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6698cdf7c8-plg8k" event={"ID":"5a8634ff-bf39-4217-b391-e5c345767bb1","Type":"ContainerStarted","Data":"d9e95d32e447a9789a5b3794d2dcdd8a5ebefc08b9426d6b599fdcc996a1c140"}
Jul 27 19:27:27.671745 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.671717    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sps46" event={"ID":"5fbd41fc-5776-424a-9652-8738061641d1","Type":"ContainerStarted","Data":"5445883815f2122b79b671af566b22735266e9bc42a1b0cc412979213d9f240a"}
Jul 27 19:27:27.671840 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.671764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sps46" event={"ID":"5fbd41fc-5776-424a-9652-8738061641d1","Type":"ContainerStarted","Data":"9e315888188bd530adcba4aa277e2529d87a76037533cee1b408ca9af429493e"}
Jul 27 19:27:27.671840 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.671734    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-695488d46b-wpmgb"
Jul 27 19:27:27.671840 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.671796    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-sps46" podUID="5fbd41fc-5776-424a-9652-8738061641d1" containerName="authorino" containerID="cri-o://5445883815f2122b79b671af566b22735266e9bc42a1b0cc412979213d9f240a" gracePeriod=30
Jul 27 19:27:27.704189 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.704144    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-sps46" podStartSLOduration=1.295600236 podStartE2EDuration="1.70412822s" totalImagesPullingTime="408.527984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:27:27.702425193 +0000 UTC m=+590.218129698" watchObservedRunningTime="2026-07-27 19:27:27.70412822 +0000 UTC m=+590.219832723"
Jul 27 19:27:27.762441 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.762412    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-695488d46b-wpmgb"
Jul 27 19:27:27.891286 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.891193    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qznj2\" (UniqueName: \"kubernetes.io/projected/0d569c2c-99d9-4570-95c5-5a309d98033c-kube-api-access-qznj2\") pod \"0d569c2c-99d9-4570-95c5-5a309d98033c\" (UID: \"0d569c2c-99d9-4570-95c5-5a309d98033c\") "
Jul 27 19:27:27.894605 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.894574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d569c2c-99d9-4570-95c5-5a309d98033c-kube-api-access-qznj2" (OuterVolumeSpecName: "kube-api-access-qznj2") pod "0d569c2c-99d9-4570-95c5-5a309d98033c" (UID: "0d569c2c-99d9-4570-95c5-5a309d98033c"). InnerVolumeSpecName "kube-api-access-qznj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:27:27.921167 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.921145    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sps46"
Jul 27 19:27:27.991765 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.991732    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcc5p\" (UniqueName: \"kubernetes.io/projected/5fbd41fc-5776-424a-9652-8738061641d1-kube-api-access-wcc5p\") pod \"5fbd41fc-5776-424a-9652-8738061641d1\" (UID: \"5fbd41fc-5776-424a-9652-8738061641d1\") "
Jul 27 19:27:27.991950 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.991937    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qznj2\" (UniqueName: \"kubernetes.io/projected/0d569c2c-99d9-4570-95c5-5a309d98033c-kube-api-access-qznj2\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:27:27.993775 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:27.993750    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbd41fc-5776-424a-9652-8738061641d1-kube-api-access-wcc5p" (OuterVolumeSpecName: "kube-api-access-wcc5p") pod "5fbd41fc-5776-424a-9652-8738061641d1" (UID: "5fbd41fc-5776-424a-9652-8738061641d1"). InnerVolumeSpecName "kube-api-access-wcc5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:27:28.092660 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.092633    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcc5p\" (UniqueName: \"kubernetes.io/projected/5fbd41fc-5776-424a-9652-8738061641d1-kube-api-access-wcc5p\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:27:28.678891 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.678860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6698cdf7c8-plg8k" event={"ID":"5a8634ff-bf39-4217-b391-e5c345767bb1","Type":"ContainerStarted","Data":"a7530ebd4b7f9e4a1ee5921959866796b6e3e0a3241a24fc3f56185590814ff0"}
Jul 27 19:27:28.680073 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.680051    2580 generic.go:356] "Generic (PLEG): container finished" podID="5fbd41fc-5776-424a-9652-8738061641d1" containerID="5445883815f2122b79b671af566b22735266e9bc42a1b0cc412979213d9f240a" exitCode=0
Jul 27 19:27:28.680137 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.680080    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sps46"
Jul 27 19:27:28.680137 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.680129    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sps46" event={"ID":"5fbd41fc-5776-424a-9652-8738061641d1","Type":"ContainerDied","Data":"5445883815f2122b79b671af566b22735266e9bc42a1b0cc412979213d9f240a"}
Jul 27 19:27:28.680207 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.680160    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-sps46" event={"ID":"5fbd41fc-5776-424a-9652-8738061641d1","Type":"ContainerDied","Data":"9e315888188bd530adcba4aa277e2529d87a76037533cee1b408ca9af429493e"}
Jul 27 19:27:28.680207 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.680179    2580 scope.go:117] "RemoveContainer" containerID="5445883815f2122b79b671af566b22735266e9bc42a1b0cc412979213d9f240a"
Jul 27 19:27:28.680297 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.680252    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-695488d46b-wpmgb"
Jul 27 19:27:28.688383 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.688363    2580 scope.go:117] "RemoveContainer" containerID="5445883815f2122b79b671af566b22735266e9bc42a1b0cc412979213d9f240a"
Jul 27 19:27:28.688635 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:27:28.688614    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5445883815f2122b79b671af566b22735266e9bc42a1b0cc412979213d9f240a\": container with ID starting with 5445883815f2122b79b671af566b22735266e9bc42a1b0cc412979213d9f240a not found: ID does not exist" containerID="5445883815f2122b79b671af566b22735266e9bc42a1b0cc412979213d9f240a"
Jul 27 19:27:28.688695 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.688643    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5445883815f2122b79b671af566b22735266e9bc42a1b0cc412979213d9f240a"} err="failed to get container status \"5445883815f2122b79b671af566b22735266e9bc42a1b0cc412979213d9f240a\": rpc error: code = NotFound desc = could not find container \"5445883815f2122b79b671af566b22735266e9bc42a1b0cc412979213d9f240a\": container with ID starting with 5445883815f2122b79b671af566b22735266e9bc42a1b0cc412979213d9f240a not found: ID does not exist"
Jul 27 19:27:28.695389 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.695350    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6698cdf7c8-plg8k" podStartSLOduration=2.340332931 podStartE2EDuration="2.695337614s" totalImagesPullingTime="355.004683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:27:28.694608504 +0000 UTC m=+591.210313007" watchObservedRunningTime="2026-07-27 19:27:28.695337614 +0000 UTC m=+591.211042132"
Jul 27 19:27:28.709933 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.709906    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sps46"]
Jul 27 19:27:28.713261 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.713219    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sps46"]
Jul 27 19:27:28.732271 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.732228    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lzz67"]
Jul 27 19:27:28.732483 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.732458    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-lzz67" podUID="c15070dd-7d4d-43c8-af64-1f4782bd0578" containerName="authorino" containerID="cri-o://bf4d9bbb95b20075acfc9892bdde86b749d4c1a2553d0d385337c2d6444f81bc" gracePeriod=30
Jul 27 19:27:28.739026 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.739004    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-695488d46b-wpmgb"]
Jul 27 19:27:28.752010 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.751952    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-695488d46b-wpmgb"]
Jul 27 19:27:28.980941 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:28.980919    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lzz67"
Jul 27 19:27:29.101614 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:29.101578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mldl2\" (UniqueName: \"kubernetes.io/projected/c15070dd-7d4d-43c8-af64-1f4782bd0578-kube-api-access-mldl2\") pod \"c15070dd-7d4d-43c8-af64-1f4782bd0578\" (UID: \"c15070dd-7d4d-43c8-af64-1f4782bd0578\") "
Jul 27 19:27:29.103644 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:29.103613    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15070dd-7d4d-43c8-af64-1f4782bd0578-kube-api-access-mldl2" (OuterVolumeSpecName: "kube-api-access-mldl2") pod "c15070dd-7d4d-43c8-af64-1f4782bd0578" (UID: "c15070dd-7d4d-43c8-af64-1f4782bd0578"). InnerVolumeSpecName "kube-api-access-mldl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:27:29.202877 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:29.202842    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mldl2\" (UniqueName: \"kubernetes.io/projected/c15070dd-7d4d-43c8-af64-1f4782bd0578-kube-api-access-mldl2\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:27:29.686585 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:29.686536    2580 generic.go:356] "Generic (PLEG): container finished" podID="c15070dd-7d4d-43c8-af64-1f4782bd0578" containerID="bf4d9bbb95b20075acfc9892bdde86b749d4c1a2553d0d385337c2d6444f81bc" exitCode=0
Jul 27 19:27:29.687037 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:29.686601    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-lzz67"
Jul 27 19:27:29.687037 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:29.686622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lzz67" event={"ID":"c15070dd-7d4d-43c8-af64-1f4782bd0578","Type":"ContainerDied","Data":"bf4d9bbb95b20075acfc9892bdde86b749d4c1a2553d0d385337c2d6444f81bc"}
Jul 27 19:27:29.687037 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:29.686672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-lzz67" event={"ID":"c15070dd-7d4d-43c8-af64-1f4782bd0578","Type":"ContainerDied","Data":"02fa8df72e325485400df44a485702765d0e0f67faa2d26159b654d79b3668d9"}
Jul 27 19:27:29.687037 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:29.686696    2580 scope.go:117] "RemoveContainer" containerID="bf4d9bbb95b20075acfc9892bdde86b749d4c1a2553d0d385337c2d6444f81bc"
Jul 27 19:27:29.695477 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:29.695461    2580 scope.go:117] "RemoveContainer" containerID="bf4d9bbb95b20075acfc9892bdde86b749d4c1a2553d0d385337c2d6444f81bc"
Jul 27 19:27:29.695723 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:27:29.695702    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4d9bbb95b20075acfc9892bdde86b749d4c1a2553d0d385337c2d6444f81bc\": container with ID starting with bf4d9bbb95b20075acfc9892bdde86b749d4c1a2553d0d385337c2d6444f81bc not found: ID does not exist" containerID="bf4d9bbb95b20075acfc9892bdde86b749d4c1a2553d0d385337c2d6444f81bc"
Jul 27 19:27:29.695793 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:29.695733    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4d9bbb95b20075acfc9892bdde86b749d4c1a2553d0d385337c2d6444f81bc"} err="failed to get container status \"bf4d9bbb95b20075acfc9892bdde86b749d4c1a2553d0d385337c2d6444f81bc\": rpc error: code = NotFound desc = could not find container \"bf4d9bbb95b20075acfc9892bdde86b749d4c1a2553d0d385337c2d6444f81bc\": container with ID starting with bf4d9bbb95b20075acfc9892bdde86b749d4c1a2553d0d385337c2d6444f81bc not found: ID does not exist"
Jul 27 19:27:29.707976 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:29.707946    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lzz67"]
Jul 27 19:27:29.712306 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:29.712284    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-lzz67"]
Jul 27 19:27:30.086040 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:30.085999    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d569c2c-99d9-4570-95c5-5a309d98033c" path="/var/lib/kubelet/pods/0d569c2c-99d9-4570-95c5-5a309d98033c/volumes"
Jul 27 19:27:30.086328 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:30.086310    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fbd41fc-5776-424a-9652-8738061641d1" path="/var/lib/kubelet/pods/5fbd41fc-5776-424a-9652-8738061641d1/volumes"
Jul 27 19:27:30.086642 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:27:30.086626    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c15070dd-7d4d-43c8-af64-1f4782bd0578" path="/var/lib/kubelet/pods/c15070dd-7d4d-43c8-af64-1f4782bd0578/volumes"
Jul 27 19:28:54.576549 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.576513    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"]
Jul 27 19:28:54.577811 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.577787    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fbd41fc-5776-424a-9652-8738061641d1" containerName="authorino"
Jul 27 19:28:54.577950 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.577939    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fbd41fc-5776-424a-9652-8738061641d1" containerName="authorino"
Jul 27 19:28:54.578079 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.578069    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c15070dd-7d4d-43c8-af64-1f4782bd0578" containerName="authorino"
Jul 27 19:28:54.578152 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.578144    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c15070dd-7d4d-43c8-af64-1f4782bd0578" containerName="authorino"
Jul 27 19:28:54.578403 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.578386    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c15070dd-7d4d-43c8-af64-1f4782bd0578" containerName="authorino"
Jul 27 19:28:54.578503 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.578494    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fbd41fc-5776-424a-9652-8738061641d1" containerName="authorino"
Jul 27 19:28:54.583140 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.583116    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.585571 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.585547    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 19:28:54.585680 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.585608    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 19:28:54.585680 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.585622    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zs4xt\""
Jul 27 19:28:54.586262 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.586227    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 27 19:28:54.590610 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.590258    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"]
Jul 27 19:28:54.619496 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.619469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199a9ce2-abc8-409b-93c0-5dd920214dde-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.619597 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.619549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199a9ce2-abc8-409b-93c0-5dd920214dde-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.619597 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.619587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199a9ce2-abc8-409b-93c0-5dd920214dde-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.619678 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.619606    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdrrl\" (UniqueName: \"kubernetes.io/projected/199a9ce2-abc8-409b-93c0-5dd920214dde-kube-api-access-rdrrl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.619678 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.619624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199a9ce2-abc8-409b-93c0-5dd920214dde-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.619678 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.619655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199a9ce2-abc8-409b-93c0-5dd920214dde-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.720277 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.720236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199a9ce2-abc8-409b-93c0-5dd920214dde-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.720376 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.720295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199a9ce2-abc8-409b-93c0-5dd920214dde-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.720376 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.720327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199a9ce2-abc8-409b-93c0-5dd920214dde-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.720376 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.720345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdrrl\" (UniqueName: \"kubernetes.io/projected/199a9ce2-abc8-409b-93c0-5dd920214dde-kube-api-access-rdrrl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.720376 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.720362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199a9ce2-abc8-409b-93c0-5dd920214dde-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.720538 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.720391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199a9ce2-abc8-409b-93c0-5dd920214dde-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.720703 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.720679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199a9ce2-abc8-409b-93c0-5dd920214dde-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.720755 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.720710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199a9ce2-abc8-409b-93c0-5dd920214dde-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.720793 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.720761    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199a9ce2-abc8-409b-93c0-5dd920214dde-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.722634 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.722608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199a9ce2-abc8-409b-93c0-5dd920214dde-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.722728 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.722713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199a9ce2-abc8-409b-93c0-5dd920214dde-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.732611 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.732588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdrrl\" (UniqueName: \"kubernetes.io/projected/199a9ce2-abc8-409b-93c0-5dd920214dde-kube-api-access-rdrrl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mqgcm\" (UID: \"199a9ce2-abc8-409b-93c0-5dd920214dde\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:54.895158 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:54.895082    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:28:55.021008 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:55.020978    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"]
Jul 27 19:28:55.022479 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:28:55.022450    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199a9ce2_abc8_409b_93c0_5dd920214dde.slice/crio-3f5edb53579c8ab8ddabcd44168f06c7cb3bb034edc614d70a7afd39d1cf2d93 WatchSource:0}: Error finding container 3f5edb53579c8ab8ddabcd44168f06c7cb3bb034edc614d70a7afd39d1cf2d93: Status 404 returned error can't find the container with id 3f5edb53579c8ab8ddabcd44168f06c7cb3bb034edc614d70a7afd39d1cf2d93
Jul 27 19:28:55.023825 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:55.023809    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:28:55.092925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:55.092896    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm" event={"ID":"199a9ce2-abc8-409b-93c0-5dd920214dde","Type":"ContainerStarted","Data":"3f5edb53579c8ab8ddabcd44168f06c7cb3bb034edc614d70a7afd39d1cf2d93"}
Jul 27 19:28:57.116601 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:57.116567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm" event={"ID":"199a9ce2-abc8-409b-93c0-5dd920214dde","Type":"ContainerStarted","Data":"67d54cd1ca32b7bb8b49b2fbc8b8d8b674920ce6eb780e7c56022340cafb046a"}
Jul 27 19:28:57.135068 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:28:57.135019    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm" podStartSLOduration=2.084170733 podStartE2EDuration="3.135003718s" totalImagesPullingTime="1.050832985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:28:57.133981495 +0000 UTC m=+679.649686001" watchObservedRunningTime="2026-07-27 19:28:57.135003718 +0000 UTC m=+679.650708221"
Jul 27 19:29:04.896194 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:04.896156    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:29:04.896618 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:04.896259    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:29:04.909044 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:04.909020    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:29:05.167200 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:05.167121    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mqgcm"
Jul 27 19:29:26.783503 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.783465    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"]
Jul 27 19:29:26.789685 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.789664    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.792015 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.791997    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 27 19:29:26.795437 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.795417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"]
Jul 27 19:29:26.881277 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.881222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ef021d-b933-4209-abb1-4cd5d10f634c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.881461 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.881287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ef021d-b933-4209-abb1-4cd5d10f634c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.881461 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.881308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgzc8\" (UniqueName: \"kubernetes.io/projected/f6ef021d-b933-4209-abb1-4cd5d10f634c-kube-api-access-kgzc8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.881461 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.881327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ef021d-b933-4209-abb1-4cd5d10f634c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.881461 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.881355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ef021d-b933-4209-abb1-4cd5d10f634c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.881602 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.881462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ef021d-b933-4209-abb1-4cd5d10f634c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.981946 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.981906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ef021d-b933-4209-abb1-4cd5d10f634c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.982089 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.981974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ef021d-b933-4209-abb1-4cd5d10f634c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.982089 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.982021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ef021d-b933-4209-abb1-4cd5d10f634c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.982089 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.982062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ef021d-b933-4209-abb1-4cd5d10f634c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.982233 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.982089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgzc8\" (UniqueName: \"kubernetes.io/projected/f6ef021d-b933-4209-abb1-4cd5d10f634c-kube-api-access-kgzc8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.982233 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.982117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ef021d-b933-4209-abb1-4cd5d10f634c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.982493 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.982460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ef021d-b933-4209-abb1-4cd5d10f634c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.982493 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.982477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ef021d-b933-4209-abb1-4cd5d10f634c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.982644 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.982559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ef021d-b933-4209-abb1-4cd5d10f634c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.984225 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.984209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ef021d-b933-4209-abb1-4cd5d10f634c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.984367 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.984349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ef021d-b933-4209-abb1-4cd5d10f634c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:26.990794 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:26.990772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgzc8\" (UniqueName: \"kubernetes.io/projected/f6ef021d-b933-4209-abb1-4cd5d10f634c-kube-api-access-kgzc8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd\" (UID: \"f6ef021d-b933-4209-abb1-4cd5d10f634c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:27.100513 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:27.100475    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:27.228854 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:27.228827    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"]
Jul 27 19:29:27.230364 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:29:27.230338    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ef021d_b933_4209_abb1_4cd5d10f634c.slice/crio-7024dacccaeb8d7556c69e8b536ba8b0065f1020279bd5264f5f0245d27a51f9 WatchSource:0}: Error finding container 7024dacccaeb8d7556c69e8b536ba8b0065f1020279bd5264f5f0245d27a51f9: Status 404 returned error can't find the container with id 7024dacccaeb8d7556c69e8b536ba8b0065f1020279bd5264f5f0245d27a51f9
Jul 27 19:29:27.260268 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:27.260217    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd" event={"ID":"f6ef021d-b933-4209-abb1-4cd5d10f634c","Type":"ContainerStarted","Data":"7024dacccaeb8d7556c69e8b536ba8b0065f1020279bd5264f5f0245d27a51f9"}
Jul 27 19:29:28.269653 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:28.269622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd" event={"ID":"f6ef021d-b933-4209-abb1-4cd5d10f634c","Type":"ContainerStarted","Data":"3744d4c87397d396182f37bfdb212d0bd66d0a34439dce91972d6284debdbc6f"}
Jul 27 19:29:28.288098 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:28.288051    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd" podStartSLOduration=2.091222445 podStartE2EDuration="2.288035943s" totalImagesPullingTime="196.813498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:29:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:29:28.286508336 +0000 UTC m=+710.802212839" watchObservedRunningTime="2026-07-27 19:29:28.288035943 +0000 UTC m=+710.803740446"
Jul 27 19:29:37.100798 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:37.100757    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:37.101187 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:37.100830    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:37.113639 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:37.113614    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:37.323862 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:37.323828    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd"
Jul 27 19:29:53.375394 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.375360    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"]
Jul 27 19:29:53.378586 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.378570    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.381012 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.380990    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 27 19:29:53.390255 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.390223    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"]
Jul 27 19:29:53.406691 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.406669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnwm5\" (UniqueName: \"kubernetes.io/projected/8e95eaa4-c410-4444-a5d3-a0cf342e9798-kube-api-access-rnwm5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.406812 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.406701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e95eaa4-c410-4444-a5d3-a0cf342e9798-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.406812 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.406735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e95eaa4-c410-4444-a5d3-a0cf342e9798-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.406913 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.406819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e95eaa4-c410-4444-a5d3-a0cf342e9798-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.406913 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.406862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e95eaa4-c410-4444-a5d3-a0cf342e9798-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.407002 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.406934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e95eaa4-c410-4444-a5d3-a0cf342e9798-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.507925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.507888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e95eaa4-c410-4444-a5d3-a0cf342e9798-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.507925 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.507930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e95eaa4-c410-4444-a5d3-a0cf342e9798-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.508169 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.507951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e95eaa4-c410-4444-a5d3-a0cf342e9798-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.508169 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.508065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e95eaa4-c410-4444-a5d3-a0cf342e9798-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.508169 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.508133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnwm5\" (UniqueName: \"kubernetes.io/projected/8e95eaa4-c410-4444-a5d3-a0cf342e9798-kube-api-access-rnwm5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.508169 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.508167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e95eaa4-c410-4444-a5d3-a0cf342e9798-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.508409 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.508327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e95eaa4-c410-4444-a5d3-a0cf342e9798-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.508448 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.508419    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e95eaa4-c410-4444-a5d3-a0cf342e9798-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.508506 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.508487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e95eaa4-c410-4444-a5d3-a0cf342e9798-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.510359 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.510331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e95eaa4-c410-4444-a5d3-a0cf342e9798-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.510548 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.510532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e95eaa4-c410-4444-a5d3-a0cf342e9798-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.519180 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.519157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnwm5\" (UniqueName: \"kubernetes.io/projected/8e95eaa4-c410-4444-a5d3-a0cf342e9798-kube-api-access-rnwm5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4v22l\" (UID: \"8e95eaa4-c410-4444-a5d3-a0cf342e9798\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.687124 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.687047    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:29:53.813251 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:53.813213    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"]
Jul 27 19:29:53.814708 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:29:53.814683    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e95eaa4_c410_4444_a5d3_a0cf342e9798.slice/crio-bb7f42abe26228d4fda683aa4ef2f367bb94c12a94d9315e1ed4c2a7172335e7 WatchSource:0}: Error finding container bb7f42abe26228d4fda683aa4ef2f367bb94c12a94d9315e1ed4c2a7172335e7: Status 404 returned error can't find the container with id bb7f42abe26228d4fda683aa4ef2f367bb94c12a94d9315e1ed4c2a7172335e7
Jul 27 19:29:54.394603 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:54.394565    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l" event={"ID":"8e95eaa4-c410-4444-a5d3-a0cf342e9798","Type":"ContainerStarted","Data":"52da4e8b17e29ff9d0dcb02e03fe2efe84d99bd561edeeaa0a29bca4358babd9"}
Jul 27 19:29:54.394958 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:54.394613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l" event={"ID":"8e95eaa4-c410-4444-a5d3-a0cf342e9798","Type":"ContainerStarted","Data":"bb7f42abe26228d4fda683aa4ef2f367bb94c12a94d9315e1ed4c2a7172335e7"}
Jul 27 19:29:54.415181 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:29:54.415136    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l" podStartSLOduration=1.202305194 podStartE2EDuration="1.415119912s" totalImagesPullingTime="212.814718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:29:54.413284007 +0000 UTC m=+736.928988512" watchObservedRunningTime="2026-07-27 19:29:54.415119912 +0000 UTC m=+736.930824414"
Jul 27 19:30:03.687417 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:03.687381    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:30:03.687903 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:03.687435    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:30:03.699897 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:03.699869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:30:04.459169 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:04.459141    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4v22l"
Jul 27 19:30:38.886046 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:38.886010    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-67884bbdc-4mxts"]
Jul 27 19:30:38.888756 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:38.888738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67884bbdc-4mxts"
Jul 27 19:30:38.899620 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:38.899594    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67884bbdc-4mxts"]
Jul 27 19:30:39.002138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:39.002099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq28x\" (UniqueName: \"kubernetes.io/projected/7ef2bdcb-9a9b-496d-8c5f-a05f4293947d-kube-api-access-gq28x\") pod \"authorino-67884bbdc-4mxts\" (UID: \"7ef2bdcb-9a9b-496d-8c5f-a05f4293947d\") " pod="kuadrant-system/authorino-67884bbdc-4mxts"
Jul 27 19:30:39.002138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:39.002146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ef2bdcb-9a9b-496d-8c5f-a05f4293947d-tls-cert\") pod \"authorino-67884bbdc-4mxts\" (UID: \"7ef2bdcb-9a9b-496d-8c5f-a05f4293947d\") " pod="kuadrant-system/authorino-67884bbdc-4mxts"
Jul 27 19:30:39.103428 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:39.103385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq28x\" (UniqueName: \"kubernetes.io/projected/7ef2bdcb-9a9b-496d-8c5f-a05f4293947d-kube-api-access-gq28x\") pod \"authorino-67884bbdc-4mxts\" (UID: \"7ef2bdcb-9a9b-496d-8c5f-a05f4293947d\") " pod="kuadrant-system/authorino-67884bbdc-4mxts"
Jul 27 19:30:39.103632 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:39.103449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ef2bdcb-9a9b-496d-8c5f-a05f4293947d-tls-cert\") pod \"authorino-67884bbdc-4mxts\" (UID: \"7ef2bdcb-9a9b-496d-8c5f-a05f4293947d\") " pod="kuadrant-system/authorino-67884bbdc-4mxts"
Jul 27 19:30:39.105900 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:39.105880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ef2bdcb-9a9b-496d-8c5f-a05f4293947d-tls-cert\") pod \"authorino-67884bbdc-4mxts\" (UID: \"7ef2bdcb-9a9b-496d-8c5f-a05f4293947d\") " pod="kuadrant-system/authorino-67884bbdc-4mxts"
Jul 27 19:30:39.112083 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:39.112061    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq28x\" (UniqueName: \"kubernetes.io/projected/7ef2bdcb-9a9b-496d-8c5f-a05f4293947d-kube-api-access-gq28x\") pod \"authorino-67884bbdc-4mxts\" (UID: \"7ef2bdcb-9a9b-496d-8c5f-a05f4293947d\") " pod="kuadrant-system/authorino-67884bbdc-4mxts"
Jul 27 19:30:39.197597 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:39.197518    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67884bbdc-4mxts"
Jul 27 19:30:39.334185 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:39.334157    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67884bbdc-4mxts"]
Jul 27 19:30:39.336071 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:30:39.336042    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ef2bdcb_9a9b_496d_8c5f_a05f4293947d.slice/crio-8a2638f6c4012eee112056c1725d2e6d827cca8d6bc826e92dc5c7bd48607b48 WatchSource:0}: Error finding container 8a2638f6c4012eee112056c1725d2e6d827cca8d6bc826e92dc5c7bd48607b48: Status 404 returned error can't find the container with id 8a2638f6c4012eee112056c1725d2e6d827cca8d6bc826e92dc5c7bd48607b48
Jul 27 19:30:39.614168 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:39.614135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67884bbdc-4mxts" event={"ID":"7ef2bdcb-9a9b-496d-8c5f-a05f4293947d","Type":"ContainerStarted","Data":"8a2638f6c4012eee112056c1725d2e6d827cca8d6bc826e92dc5c7bd48607b48"}
Jul 27 19:30:40.621718 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:40.621677    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67884bbdc-4mxts" event={"ID":"7ef2bdcb-9a9b-496d-8c5f-a05f4293947d","Type":"ContainerStarted","Data":"199572ce6c074eaaab60a1e75ba5c0df3b73b4b60e0ce0e1e8cdba4f0c402d63"}
Jul 27 19:30:40.643832 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:40.643764    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-67884bbdc-4mxts" podStartSLOduration=2.164638491 podStartE2EDuration="2.643741947s" totalImagesPullingTime="479.103456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:30:40.640934755 +0000 UTC m=+783.156639280" watchObservedRunningTime="2026-07-27 19:30:40.643741947 +0000 UTC m=+783.159446452"
Jul 27 19:30:40.681903 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:40.681856    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6698cdf7c8-plg8k"]
Jul 27 19:30:40.682102 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:40.682063    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6698cdf7c8-plg8k" podUID="5a8634ff-bf39-4217-b391-e5c345767bb1" containerName="authorino" containerID="cri-o://a7530ebd4b7f9e4a1ee5921959866796b6e3e0a3241a24fc3f56185590814ff0" gracePeriod=30
Jul 27 19:30:40.958811 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:40.958781    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6698cdf7c8-plg8k"
Jul 27 19:30:41.021652 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.021617    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a8634ff-bf39-4217-b391-e5c345767bb1-tls-cert\") pod \"5a8634ff-bf39-4217-b391-e5c345767bb1\" (UID: \"5a8634ff-bf39-4217-b391-e5c345767bb1\") "
Jul 27 19:30:41.021807 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.021697    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtd58\" (UniqueName: \"kubernetes.io/projected/5a8634ff-bf39-4217-b391-e5c345767bb1-kube-api-access-jtd58\") pod \"5a8634ff-bf39-4217-b391-e5c345767bb1\" (UID: \"5a8634ff-bf39-4217-b391-e5c345767bb1\") "
Jul 27 19:30:41.023729 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.023695    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8634ff-bf39-4217-b391-e5c345767bb1-kube-api-access-jtd58" (OuterVolumeSpecName: "kube-api-access-jtd58") pod "5a8634ff-bf39-4217-b391-e5c345767bb1" (UID: "5a8634ff-bf39-4217-b391-e5c345767bb1"). InnerVolumeSpecName "kube-api-access-jtd58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:30:41.032849 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.032819    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a8634ff-bf39-4217-b391-e5c345767bb1-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5a8634ff-bf39-4217-b391-e5c345767bb1" (UID: "5a8634ff-bf39-4217-b391-e5c345767bb1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:30:41.122993 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.122961    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a8634ff-bf39-4217-b391-e5c345767bb1-tls-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:30:41.122993 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.122988    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtd58\" (UniqueName: \"kubernetes.io/projected/5a8634ff-bf39-4217-b391-e5c345767bb1-kube-api-access-jtd58\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:30:41.628813 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.628777    2580 generic.go:356] "Generic (PLEG): container finished" podID="5a8634ff-bf39-4217-b391-e5c345767bb1" containerID="a7530ebd4b7f9e4a1ee5921959866796b6e3e0a3241a24fc3f56185590814ff0" exitCode=0
Jul 27 19:30:41.629211 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.628833    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6698cdf7c8-plg8k"
Jul 27 19:30:41.629211 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.628864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6698cdf7c8-plg8k" event={"ID":"5a8634ff-bf39-4217-b391-e5c345767bb1","Type":"ContainerDied","Data":"a7530ebd4b7f9e4a1ee5921959866796b6e3e0a3241a24fc3f56185590814ff0"}
Jul 27 19:30:41.629211 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.628911    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6698cdf7c8-plg8k" event={"ID":"5a8634ff-bf39-4217-b391-e5c345767bb1","Type":"ContainerDied","Data":"d9e95d32e447a9789a5b3794d2dcdd8a5ebefc08b9426d6b599fdcc996a1c140"}
Jul 27 19:30:41.629211 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.628944    2580 scope.go:117] "RemoveContainer" containerID="a7530ebd4b7f9e4a1ee5921959866796b6e3e0a3241a24fc3f56185590814ff0"
Jul 27 19:30:41.637498 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.637473    2580 scope.go:117] "RemoveContainer" containerID="a7530ebd4b7f9e4a1ee5921959866796b6e3e0a3241a24fc3f56185590814ff0"
Jul 27 19:30:41.637803 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:30:41.637770    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7530ebd4b7f9e4a1ee5921959866796b6e3e0a3241a24fc3f56185590814ff0\": container with ID starting with a7530ebd4b7f9e4a1ee5921959866796b6e3e0a3241a24fc3f56185590814ff0 not found: ID does not exist" containerID="a7530ebd4b7f9e4a1ee5921959866796b6e3e0a3241a24fc3f56185590814ff0"
Jul 27 19:30:41.637891 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.637799    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7530ebd4b7f9e4a1ee5921959866796b6e3e0a3241a24fc3f56185590814ff0"} err="failed to get container status \"a7530ebd4b7f9e4a1ee5921959866796b6e3e0a3241a24fc3f56185590814ff0\": rpc error: code = NotFound desc = could not find container \"a7530ebd4b7f9e4a1ee5921959866796b6e3e0a3241a24fc3f56185590814ff0\": container with ID starting with a7530ebd4b7f9e4a1ee5921959866796b6e3e0a3241a24fc3f56185590814ff0 not found: ID does not exist"
Jul 27 19:30:41.654652 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.654631    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6698cdf7c8-plg8k"]
Jul 27 19:30:41.660281 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:41.660260    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6698cdf7c8-plg8k"]
Jul 27 19:30:42.085891 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:30:42.085860    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a8634ff-bf39-4217-b391-e5c345767bb1" path="/var/lib/kubelet/pods/5a8634ff-bf39-4217-b391-e5c345767bb1/volumes"
Jul 27 19:36:51.083576 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:36:51.083537    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6868c98757-9kql8"]
Jul 27 19:37:16.102071 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.102002    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6868c98757-9kql8" podUID="5548a379-4118-4f51-8460-ea30bd54f951" containerName="console" containerID="cri-o://55e8474dc21a496b6e333525e0aee248dceccf41a47ad82c96a20b6018251070" gracePeriod=15
Jul 27 19:37:16.358639 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.358581    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:37:16.408138 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.408105    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-console-config\") pod \"5548a379-4118-4f51-8460-ea30bd54f951\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") "
Jul 27 19:37:16.408340 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.408171    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxngw\" (UniqueName: \"kubernetes.io/projected/5548a379-4118-4f51-8460-ea30bd54f951-kube-api-access-rxngw\") pod \"5548a379-4118-4f51-8460-ea30bd54f951\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") "
Jul 27 19:37:16.408340 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.408192    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5548a379-4118-4f51-8460-ea30bd54f951-console-oauth-config\") pod \"5548a379-4118-4f51-8460-ea30bd54f951\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") "
Jul 27 19:37:16.408340 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.408213    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-trusted-ca-bundle\") pod \"5548a379-4118-4f51-8460-ea30bd54f951\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") "
Jul 27 19:37:16.408340 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.408269    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5548a379-4118-4f51-8460-ea30bd54f951-console-serving-cert\") pod \"5548a379-4118-4f51-8460-ea30bd54f951\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") "
Jul 27 19:37:16.408340 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.408294    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-service-ca\") pod \"5548a379-4118-4f51-8460-ea30bd54f951\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") "
Jul 27 19:37:16.408340 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.408341    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-oauth-serving-cert\") pod \"5548a379-4118-4f51-8460-ea30bd54f951\" (UID: \"5548a379-4118-4f51-8460-ea30bd54f951\") "
Jul 27 19:37:16.408684 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.408647    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-console-config" (OuterVolumeSpecName: "console-config") pod "5548a379-4118-4f51-8460-ea30bd54f951" (UID: "5548a379-4118-4f51-8460-ea30bd54f951"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:37:16.408798 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.408685    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-service-ca" (OuterVolumeSpecName: "service-ca") pod "5548a379-4118-4f51-8460-ea30bd54f951" (UID: "5548a379-4118-4f51-8460-ea30bd54f951"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:37:16.408798 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.408750    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5548a379-4118-4f51-8460-ea30bd54f951" (UID: "5548a379-4118-4f51-8460-ea30bd54f951"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:37:16.408798 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.408784    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5548a379-4118-4f51-8460-ea30bd54f951" (UID: "5548a379-4118-4f51-8460-ea30bd54f951"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:37:16.410528 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.410498    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5548a379-4118-4f51-8460-ea30bd54f951-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5548a379-4118-4f51-8460-ea30bd54f951" (UID: "5548a379-4118-4f51-8460-ea30bd54f951"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:37:16.410528 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.410521    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5548a379-4118-4f51-8460-ea30bd54f951-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5548a379-4118-4f51-8460-ea30bd54f951" (UID: "5548a379-4118-4f51-8460-ea30bd54f951"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:37:16.410650 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.410530    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5548a379-4118-4f51-8460-ea30bd54f951-kube-api-access-rxngw" (OuterVolumeSpecName: "kube-api-access-rxngw") pod "5548a379-4118-4f51-8460-ea30bd54f951" (UID: "5548a379-4118-4f51-8460-ea30bd54f951"). InnerVolumeSpecName "kube-api-access-rxngw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:37:16.509256 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.509217    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5548a379-4118-4f51-8460-ea30bd54f951-console-serving-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:37:16.509421 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.509271    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:37:16.509421 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.509281    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-oauth-serving-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:37:16.509421 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.509290    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-console-config\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:37:16.509421 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.509298    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxngw\" (UniqueName: \"kubernetes.io/projected/5548a379-4118-4f51-8460-ea30bd54f951-kube-api-access-rxngw\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:37:16.509421 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.509307    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5548a379-4118-4f51-8460-ea30bd54f951-console-oauth-config\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:37:16.509421 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.509316    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5548a379-4118-4f51-8460-ea30bd54f951-trusted-ca-bundle\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:37:16.547073 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.547041    2580 generic.go:356] "Generic (PLEG): container finished" podID="5548a379-4118-4f51-8460-ea30bd54f951" containerID="55e8474dc21a496b6e333525e0aee248dceccf41a47ad82c96a20b6018251070" exitCode=0
Jul 27 19:37:16.547260 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.547100    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6868c98757-9kql8"
Jul 27 19:37:16.547260 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.547137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6868c98757-9kql8" event={"ID":"5548a379-4118-4f51-8460-ea30bd54f951","Type":"ContainerDied","Data":"55e8474dc21a496b6e333525e0aee248dceccf41a47ad82c96a20b6018251070"}
Jul 27 19:37:16.547260 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.547219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6868c98757-9kql8" event={"ID":"5548a379-4118-4f51-8460-ea30bd54f951","Type":"ContainerDied","Data":"0c4906eca3039bd0504c414ea57c7b6aeefae083c5f33d4fdaea1b26172ff179"}
Jul 27 19:37:16.547260 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.547259    2580 scope.go:117] "RemoveContainer" containerID="55e8474dc21a496b6e333525e0aee248dceccf41a47ad82c96a20b6018251070"
Jul 27 19:37:16.556323 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.556302    2580 scope.go:117] "RemoveContainer" containerID="55e8474dc21a496b6e333525e0aee248dceccf41a47ad82c96a20b6018251070"
Jul 27 19:37:16.556591 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:37:16.556574    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e8474dc21a496b6e333525e0aee248dceccf41a47ad82c96a20b6018251070\": container with ID starting with 55e8474dc21a496b6e333525e0aee248dceccf41a47ad82c96a20b6018251070 not found: ID does not exist" containerID="55e8474dc21a496b6e333525e0aee248dceccf41a47ad82c96a20b6018251070"
Jul 27 19:37:16.556643 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.556598    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e8474dc21a496b6e333525e0aee248dceccf41a47ad82c96a20b6018251070"} err="failed to get container status \"55e8474dc21a496b6e333525e0aee248dceccf41a47ad82c96a20b6018251070\": rpc error: code = NotFound desc = could not find container \"55e8474dc21a496b6e333525e0aee248dceccf41a47ad82c96a20b6018251070\": container with ID starting with 55e8474dc21a496b6e333525e0aee248dceccf41a47ad82c96a20b6018251070 not found: ID does not exist"
Jul 27 19:37:16.569007 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.568978    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6868c98757-9kql8"]
Jul 27 19:37:16.572844 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:16.572820    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6868c98757-9kql8"]
Jul 27 19:37:18.085706 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:37:18.085667    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5548a379-4118-4f51-8460-ea30bd54f951" path="/var/lib/kubelet/pods/5548a379-4118-4f51-8460-ea30bd54f951/volumes"
Jul 27 19:41:15.986056 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:15.986022    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"]
Jul 27 19:41:15.988398 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:15.986511    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5548a379-4118-4f51-8460-ea30bd54f951" containerName="console"
Jul 27 19:41:15.988398 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:15.986525    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5548a379-4118-4f51-8460-ea30bd54f951" containerName="console"
Jul 27 19:41:15.988398 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:15.986539    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8634ff-bf39-4217-b391-e5c345767bb1" containerName="authorino"
Jul 27 19:41:15.988398 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:15.986544    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8634ff-bf39-4217-b391-e5c345767bb1" containerName="authorino"
Jul 27 19:41:15.988398 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:15.986601    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5548a379-4118-4f51-8460-ea30bd54f951" containerName="console"
Jul 27 19:41:15.988398 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:15.986610    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8634ff-bf39-4217-b391-e5c345767bb1" containerName="authorino"
Jul 27 19:41:15.989174 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:15.989160    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:15.992591 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:15.992569    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6fceca16-kserve-self-signed-certs\""
Jul 27 19:41:16.013223 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.013201    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"]
Jul 27 19:41:16.118617 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.118578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkvfl\" (UniqueName: \"kubernetes.io/projected/368f2d84-c714-423a-b6bb-96183725997f-kube-api-access-lkvfl\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.118786 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.118635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-dshm\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.118786 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.118659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368f2d84-c714-423a-b6bb-96183725997f-tls-certs\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.118786 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.118712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-home\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.118786 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.118746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-model-cache\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.118941 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.118846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-tmp-dir\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.219547 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.219514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-home\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.219547 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.219551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-model-cache\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.219746 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.219607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-tmp-dir\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.219746 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.219648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkvfl\" (UniqueName: \"kubernetes.io/projected/368f2d84-c714-423a-b6bb-96183725997f-kube-api-access-lkvfl\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.219746 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.219682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-dshm\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.219746 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.219703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368f2d84-c714-423a-b6bb-96183725997f-tls-certs\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.220011 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.219986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-home\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.220137 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.220114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-model-cache\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.220216 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.220190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-tmp-dir\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.221895 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.221867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-dshm\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.222162 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.222138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368f2d84-c714-423a-b6bb-96183725997f-tls-certs\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.230647 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.230626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkvfl\" (UniqueName: \"kubernetes.io/projected/368f2d84-c714-423a-b6bb-96183725997f-kube-api-access-lkvfl\") pod \"e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") " pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.298545 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.298519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:16.457390 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.457364    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"]
Jul 27 19:41:16.459378 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:41:16.459352    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod368f2d84_c714_423a_b6bb_96183725997f.slice/crio-7941fbff021b5878c1349ad63dec79d9b094e70d150bf474d36511c0e76212b0 WatchSource:0}: Error finding container 7941fbff021b5878c1349ad63dec79d9b094e70d150bf474d36511c0e76212b0: Status 404 returned error can't find the container with id 7941fbff021b5878c1349ad63dec79d9b094e70d150bf474d36511c0e76212b0
Jul 27 19:41:16.732369 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.732281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h" event={"ID":"368f2d84-c714-423a-b6bb-96183725997f","Type":"ContainerStarted","Data":"a62cbbfd37df4a009406afce64829ae48246b566b72ecd34271f9ca511f9f090"}
Jul 27 19:41:16.732369 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.732325    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h" event={"ID":"368f2d84-c714-423a-b6bb-96183725997f","Type":"ContainerStarted","Data":"7941fbff021b5878c1349ad63dec79d9b094e70d150bf474d36511c0e76212b0"}
Jul 27 19:41:16.750492 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:16.750434    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h" podStartSLOduration=1.750417406 podStartE2EDuration="1.750417406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:16.748763324 +0000 UTC m=+1419.264467828" watchObservedRunningTime="2026-07-27 19:41:16.750417406 +0000 UTC m=+1419.266121909"
Jul 27 19:41:22.956873 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:22.956842    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"]
Jul 27 19:41:22.960288 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:22.960268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:22.963318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:22.963287    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6fceca16-kserve-self-signed-certs\""
Jul 27 19:41:22.976613 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:22.976584    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"]
Jul 27 19:41:23.077684 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.077648    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-dshm\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.077846 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.077783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-tmp-dir\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.077846 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.077826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-tls-certs\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.077929 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.077864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvjbw\" (UniqueName: \"kubernetes.io/projected/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-kube-api-access-fvjbw\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.077965 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.077925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-model-cache\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.078005 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.077984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-home\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.179031 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.178993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-dshm\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.179201 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.179058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-tmp-dir\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.179201 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.179082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-tls-certs\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.179201 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.179107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvjbw\" (UniqueName: \"kubernetes.io/projected/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-kube-api-access-fvjbw\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.179201 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.179126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-model-cache\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.179201 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.179147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-home\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.179550 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.179515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-home\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.179676 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.179574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-tmp-dir\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.179676 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.179604    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-model-cache\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.181405 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.181381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-dshm\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.181669 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.181646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-tls-certs\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.188286 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.188266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvjbw\" (UniqueName: \"kubernetes.io/projected/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-kube-api-access-fvjbw\") pod \"e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") " pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.270005 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.269930    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:23.409647 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.409622    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"]
Jul 27 19:41:23.411773 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:41:23.411744    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ee12f35_07fb_4f8e_bc65_b2b8d88c2563.slice/crio-1d91c498027bd9af79180914da1a975c5f1f9781d85025d8d7ebff8af09b032e WatchSource:0}: Error finding container 1d91c498027bd9af79180914da1a975c5f1f9781d85025d8d7ebff8af09b032e: Status 404 returned error can't find the container with id 1d91c498027bd9af79180914da1a975c5f1f9781d85025d8d7ebff8af09b032e
Jul 27 19:41:23.766641 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.766597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c" event={"ID":"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563","Type":"ContainerStarted","Data":"5793ada55bd6a58ca692d0373de6342de8446d1cd53988d75926ac8612baac90"}
Jul 27 19:41:23.766804 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.766651    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c" event={"ID":"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563","Type":"ContainerStarted","Data":"1d91c498027bd9af79180914da1a975c5f1f9781d85025d8d7ebff8af09b032e"}
Jul 27 19:41:23.785962 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:23.785908    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c" podStartSLOduration=1.785891297 podStartE2EDuration="1.785891297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:41:23.783887891 +0000 UTC m=+1426.299592396" watchObservedRunningTime="2026-07-27 19:41:23.785891297 +0000 UTC m=+1426.301595799"
Jul 27 19:41:26.299068 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:26.299028    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:26.299429 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:26.299088    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:26.311361 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:26.311337    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:26.792951 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:26.792925    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:33.270576 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:33.270494    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:33.270576 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:33.270546    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:33.283156 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:33.283131    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:33.826875 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:33.826842    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:41.203147 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:41.203113    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"]
Jul 27 19:41:43.069422 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:43.069386    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"]
Jul 27 19:41:56.221749 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.221698    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h" podUID="368f2d84-c714-423a-b6bb-96183725997f" containerName="main" containerID="cri-o://a62cbbfd37df4a009406afce64829ae48246b566b72ecd34271f9ca511f9f090" gracePeriod=45
Jul 27 19:41:56.489384 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.489357    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:56.547798 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.547763    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkvfl\" (UniqueName: \"kubernetes.io/projected/368f2d84-c714-423a-b6bb-96183725997f-kube-api-access-lkvfl\") pod \"368f2d84-c714-423a-b6bb-96183725997f\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") "
Jul 27 19:41:56.547974 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.547850    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368f2d84-c714-423a-b6bb-96183725997f-tls-certs\") pod \"368f2d84-c714-423a-b6bb-96183725997f\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") "
Jul 27 19:41:56.547974 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.547882    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-dshm\") pod \"368f2d84-c714-423a-b6bb-96183725997f\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") "
Jul 27 19:41:56.547974 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.547926    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-model-cache\") pod \"368f2d84-c714-423a-b6bb-96183725997f\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") "
Jul 27 19:41:56.548144 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.547996    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-tmp-dir\") pod \"368f2d84-c714-423a-b6bb-96183725997f\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") "
Jul 27 19:41:56.548144 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.548036    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-home\") pod \"368f2d84-c714-423a-b6bb-96183725997f\" (UID: \"368f2d84-c714-423a-b6bb-96183725997f\") "
Jul 27 19:41:56.548280 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.548185    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-model-cache" (OuterVolumeSpecName: "model-cache") pod "368f2d84-c714-423a-b6bb-96183725997f" (UID: "368f2d84-c714-423a-b6bb-96183725997f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:41:56.548340 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.548311    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:41:56.548407 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.548379    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "368f2d84-c714-423a-b6bb-96183725997f" (UID: "368f2d84-c714-423a-b6bb-96183725997f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:41:56.548463 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.548421    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-home" (OuterVolumeSpecName: "home") pod "368f2d84-c714-423a-b6bb-96183725997f" (UID: "368f2d84-c714-423a-b6bb-96183725997f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:41:56.550111 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.550079    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/368f2d84-c714-423a-b6bb-96183725997f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "368f2d84-c714-423a-b6bb-96183725997f" (UID: "368f2d84-c714-423a-b6bb-96183725997f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:41:56.550219 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.550144    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368f2d84-c714-423a-b6bb-96183725997f-kube-api-access-lkvfl" (OuterVolumeSpecName: "kube-api-access-lkvfl") pod "368f2d84-c714-423a-b6bb-96183725997f" (UID: "368f2d84-c714-423a-b6bb-96183725997f"). InnerVolumeSpecName "kube-api-access-lkvfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:41:56.550501 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.550483    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-dshm" (OuterVolumeSpecName: "dshm") pod "368f2d84-c714-423a-b6bb-96183725997f" (UID: "368f2d84-c714-423a-b6bb-96183725997f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:41:56.649496 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.649463    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368f2d84-c714-423a-b6bb-96183725997f-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:41:56.649496 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.649489    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:41:56.649689 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.649518    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:41:56.649689 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.649526    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368f2d84-c714-423a-b6bb-96183725997f-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:41:56.649689 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.649534    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkvfl\" (UniqueName: \"kubernetes.io/projected/368f2d84-c714-423a-b6bb-96183725997f-kube-api-access-lkvfl\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:41:56.939963 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.939923    2580 generic.go:356] "Generic (PLEG): container finished" podID="368f2d84-c714-423a-b6bb-96183725997f" containerID="a62cbbfd37df4a009406afce64829ae48246b566b72ecd34271f9ca511f9f090" exitCode=0
Jul 27 19:41:56.940124 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.939968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h" event={"ID":"368f2d84-c714-423a-b6bb-96183725997f","Type":"ContainerDied","Data":"a62cbbfd37df4a009406afce64829ae48246b566b72ecd34271f9ca511f9f090"}
Jul 27 19:41:56.940124 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.939977    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"
Jul 27 19:41:56.940124 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.939996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h" event={"ID":"368f2d84-c714-423a-b6bb-96183725997f","Type":"ContainerDied","Data":"7941fbff021b5878c1349ad63dec79d9b094e70d150bf474d36511c0e76212b0"}
Jul 27 19:41:56.940124 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.940015    2580 scope.go:117] "RemoveContainer" containerID="a62cbbfd37df4a009406afce64829ae48246b566b72ecd34271f9ca511f9f090"
Jul 27 19:41:56.950233 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.950197    2580 scope.go:117] "RemoveContainer" containerID="a62cbbfd37df4a009406afce64829ae48246b566b72ecd34271f9ca511f9f090"
Jul 27 19:41:56.950532 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:41:56.950508    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a62cbbfd37df4a009406afce64829ae48246b566b72ecd34271f9ca511f9f090\": container with ID starting with a62cbbfd37df4a009406afce64829ae48246b566b72ecd34271f9ca511f9f090 not found: ID does not exist" containerID="a62cbbfd37df4a009406afce64829ae48246b566b72ecd34271f9ca511f9f090"
Jul 27 19:41:56.950613 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.950543    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62cbbfd37df4a009406afce64829ae48246b566b72ecd34271f9ca511f9f090"} err="failed to get container status \"a62cbbfd37df4a009406afce64829ae48246b566b72ecd34271f9ca511f9f090\": rpc error: code = NotFound desc = could not find container \"a62cbbfd37df4a009406afce64829ae48246b566b72ecd34271f9ca511f9f090\": container with ID starting with a62cbbfd37df4a009406afce64829ae48246b566b72ecd34271f9ca511f9f090 not found: ID does not exist"
Jul 27 19:41:56.967754 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.967729    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"]
Jul 27 19:41:56.971071 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:56.971049    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6fceca16-kserve-7b98f45655-tt52h"]
Jul 27 19:41:58.085937 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.085909    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="368f2d84-c714-423a-b6bb-96183725997f" path="/var/lib/kubelet/pods/368f2d84-c714-423a-b6bb-96183725997f/volumes"
Jul 27 19:41:58.087567 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.087535    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c" podUID="0ee12f35-07fb-4f8e-bc65-b2b8d88c2563" containerName="main" containerID="cri-o://5793ada55bd6a58ca692d0373de6342de8446d1cd53988d75926ac8612baac90" gracePeriod=45
Jul 27 19:41:58.340979 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.340925    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:58.463873 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.463846    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvjbw\" (UniqueName: \"kubernetes.io/projected/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-kube-api-access-fvjbw\") pod \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") "
Jul 27 19:41:58.464023 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.463883    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-tls-certs\") pod \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") "
Jul 27 19:41:58.464071 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.464015    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-home\") pod \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") "
Jul 27 19:41:58.464071 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.464046    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-model-cache\") pod \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") "
Jul 27 19:41:58.464171 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.464077    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-dshm\") pod \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") "
Jul 27 19:41:58.464171 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.464095    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-tmp-dir\") pod \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\" (UID: \"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563\") "
Jul 27 19:41:58.464380 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.464337    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ee12f35-07fb-4f8e-bc65-b2b8d88c2563" (UID: "0ee12f35-07fb-4f8e-bc65-b2b8d88c2563"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:41:58.464451 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.464359    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-home" (OuterVolumeSpecName: "home") pod "0ee12f35-07fb-4f8e-bc65-b2b8d88c2563" (UID: "0ee12f35-07fb-4f8e-bc65-b2b8d88c2563"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:41:58.464451 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.464408    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ee12f35-07fb-4f8e-bc65-b2b8d88c2563" (UID: "0ee12f35-07fb-4f8e-bc65-b2b8d88c2563"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:41:58.466320 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.466298    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-dshm" (OuterVolumeSpecName: "dshm") pod "0ee12f35-07fb-4f8e-bc65-b2b8d88c2563" (UID: "0ee12f35-07fb-4f8e-bc65-b2b8d88c2563"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:41:58.466479 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.466455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ee12f35-07fb-4f8e-bc65-b2b8d88c2563" (UID: "0ee12f35-07fb-4f8e-bc65-b2b8d88c2563"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:41:58.466577 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.466485    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-kube-api-access-fvjbw" (OuterVolumeSpecName: "kube-api-access-fvjbw") pod "0ee12f35-07fb-4f8e-bc65-b2b8d88c2563" (UID: "0ee12f35-07fb-4f8e-bc65-b2b8d88c2563"). InnerVolumeSpecName "kube-api-access-fvjbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:41:58.565646 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.565615    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:41:58.565646 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.565642    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:41:58.565646 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.565651    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:41:58.565858 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.565660    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:41:58.565858 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.565673    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvjbw\" (UniqueName: \"kubernetes.io/projected/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-kube-api-access-fvjbw\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:41:58.565858 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.565684    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:41:58.953660 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.953627    2580 generic.go:356] "Generic (PLEG): container finished" podID="0ee12f35-07fb-4f8e-bc65-b2b8d88c2563" containerID="5793ada55bd6a58ca692d0373de6342de8446d1cd53988d75926ac8612baac90" exitCode=0
Jul 27 19:41:58.953846 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.953670    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"
Jul 27 19:41:58.953846 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.953721    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c" event={"ID":"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563","Type":"ContainerDied","Data":"5793ada55bd6a58ca692d0373de6342de8446d1cd53988d75926ac8612baac90"}
Jul 27 19:41:58.953846 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.953767    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c" event={"ID":"0ee12f35-07fb-4f8e-bc65-b2b8d88c2563","Type":"ContainerDied","Data":"1d91c498027bd9af79180914da1a975c5f1f9781d85025d8d7ebff8af09b032e"}
Jul 27 19:41:58.953846 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.953790    2580 scope.go:117] "RemoveContainer" containerID="5793ada55bd6a58ca692d0373de6342de8446d1cd53988d75926ac8612baac90"
Jul 27 19:41:58.962732 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.962716    2580 scope.go:117] "RemoveContainer" containerID="5793ada55bd6a58ca692d0373de6342de8446d1cd53988d75926ac8612baac90"
Jul 27 19:41:58.962994 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:41:58.962976    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5793ada55bd6a58ca692d0373de6342de8446d1cd53988d75926ac8612baac90\": container with ID starting with 5793ada55bd6a58ca692d0373de6342de8446d1cd53988d75926ac8612baac90 not found: ID does not exist" containerID="5793ada55bd6a58ca692d0373de6342de8446d1cd53988d75926ac8612baac90"
Jul 27 19:41:58.963055 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.963002    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5793ada55bd6a58ca692d0373de6342de8446d1cd53988d75926ac8612baac90"} err="failed to get container status \"5793ada55bd6a58ca692d0373de6342de8446d1cd53988d75926ac8612baac90\": rpc error: code = NotFound desc = could not find container \"5793ada55bd6a58ca692d0373de6342de8446d1cd53988d75926ac8612baac90\": container with ID starting with 5793ada55bd6a58ca692d0373de6342de8446d1cd53988d75926ac8612baac90 not found: ID does not exist"
Jul 27 19:41:58.976199 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.976175    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"]
Jul 27 19:41:58.979887 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:41:58.979866    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6fceca16-kserve-84bfc685b7-km99c"]
Jul 27 19:42:00.090863 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:42:00.090829    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ee12f35-07fb-4f8e-bc65-b2b8d88c2563" path="/var/lib/kubelet/pods/0ee12f35-07fb-4f8e-bc65-b2b8d88c2563/volumes"
Jul 27 19:45:55.874548 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:55.874513    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg"]
Jul 27 19:45:55.875005 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:55.874836    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="368f2d84-c714-423a-b6bb-96183725997f" containerName="main"
Jul 27 19:45:55.875005 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:55.874847    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="368f2d84-c714-423a-b6bb-96183725997f" containerName="main"
Jul 27 19:45:55.875005 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:55.874860    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ee12f35-07fb-4f8e-bc65-b2b8d88c2563" containerName="main"
Jul 27 19:45:55.875005 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:55.874866    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ee12f35-07fb-4f8e-bc65-b2b8d88c2563" containerName="main"
Jul 27 19:45:55.875005 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:55.874923    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="368f2d84-c714-423a-b6bb-96183725997f" containerName="main"
Jul 27 19:45:55.875005 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:55.874935    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ee12f35-07fb-4f8e-bc65-b2b8d88c2563" containerName="main"
Jul 27 19:45:55.877798 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:55.877783    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg"
Jul 27 19:45:55.880468 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:55.880448    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\""
Jul 27 19:45:55.880517 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:55.880494    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 19:45:55.886361 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:55.886336    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg"]
Jul 27 19:45:55.963163 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:55.963136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfee450-fd7b-4709-ac31-915c51c31bd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-68303af4-5zxgg\" (UID: \"9dfee450-fd7b-4709-ac31-915c51c31bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg"
Jul 27 19:45:56.064105 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:56.064076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfee450-fd7b-4709-ac31-915c51c31bd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-68303af4-5zxgg\" (UID: \"9dfee450-fd7b-4709-ac31-915c51c31bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg"
Jul 27 19:45:56.064652 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:56.064634    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfee450-fd7b-4709-ac31-915c51c31bd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-68303af4-5zxgg\" (UID: \"9dfee450-fd7b-4709-ac31-915c51c31bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg"
Jul 27 19:45:56.189950 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:56.189880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg"
Jul 27 19:45:56.519929 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:56.519904    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg"]
Jul 27 19:45:56.521740 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:45:56.521713    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfee450_fd7b_4709_ac31_915c51c31bd7.slice/crio-5bf8ea5e5c495b3b76cb70015fb218aab627c818fb45f93eb47f32553dd1b175 WatchSource:0}: Error finding container 5bf8ea5e5c495b3b76cb70015fb218aab627c818fb45f93eb47f32553dd1b175: Status 404 returned error can't find the container with id 5bf8ea5e5c495b3b76cb70015fb218aab627c818fb45f93eb47f32553dd1b175
Jul 27 19:45:56.523650 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:56.523636    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:45:57.125200 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:45:57.125163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg" event={"ID":"9dfee450-fd7b-4709-ac31-915c51c31bd7","Type":"ContainerStarted","Data":"5bf8ea5e5c495b3b76cb70015fb218aab627c818fb45f93eb47f32553dd1b175"}
Jul 27 19:46:00.142345 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:46:00.142309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg" event={"ID":"9dfee450-fd7b-4709-ac31-915c51c31bd7","Type":"ContainerStarted","Data":"5524ca67f42200f9fce815fa6c7645d39f10dbbfe999f8904a7baa9c35121710"}
Jul 27 19:46:00.159609 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:46:00.159561    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg" podStartSLOduration=2.233716833 podStartE2EDuration="5.159546201s" totalImagesPullingTime="2.925829368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:46:00.158536873 +0000 UTC m=+1702.674241376" watchObservedRunningTime="2026-07-27 19:46:00.159546201 +0000 UTC m=+1702.675250762"
Jul 27 19:46:01.148805 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:46:01.148767    2580 generic.go:356] "Generic (PLEG): container finished" podID="9dfee450-fd7b-4709-ac31-915c51c31bd7" containerID="5524ca67f42200f9fce815fa6c7645d39f10dbbfe999f8904a7baa9c35121710" exitCode=0
Jul 27 19:46:01.148805 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:46:01.148808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg" event={"ID":"9dfee450-fd7b-4709-ac31-915c51c31bd7","Type":"ContainerDied","Data":"5524ca67f42200f9fce815fa6c7645d39f10dbbfe999f8904a7baa9c35121710"}
Jul 27 19:46:02.286412 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:46:02.286387    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg"
Jul 27 19:46:02.420222 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:46:02.420141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfee450-fd7b-4709-ac31-915c51c31bd7-maas-api-service-ca\") pod \"9dfee450-fd7b-4709-ac31-915c51c31bd7\" (UID: \"9dfee450-fd7b-4709-ac31-915c51c31bd7\") "
Jul 27 19:46:02.420540 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:46:02.420519    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dfee450-fd7b-4709-ac31-915c51c31bd7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9dfee450-fd7b-4709-ac31-915c51c31bd7" (UID: "9dfee450-fd7b-4709-ac31-915c51c31bd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:46:02.521162 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:46:02.521129    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfee450-fd7b-4709-ac31-915c51c31bd7-maas-api-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:46:03.159556 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:46:03.159528    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg"
Jul 27 19:46:03.159556 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:46:03.159536    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg" event={"ID":"9dfee450-fd7b-4709-ac31-915c51c31bd7","Type":"ContainerDied","Data":"5bf8ea5e5c495b3b76cb70015fb218aab627c818fb45f93eb47f32553dd1b175"}
Jul 27 19:46:03.159800 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:46:03.159581    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bf8ea5e5c495b3b76cb70015fb218aab627c818fb45f93eb47f32553dd1b175"
Jul 27 19:46:11.010903 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:46:11.010869    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg"]
Jul 27 19:46:11.022549 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:46:11.022517    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68303af4-5zxgg"]
Jul 27 19:46:12.086115 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:46:12.086070    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dfee450-fd7b-4709-ac31-915c51c31bd7" path="/var/lib/kubelet/pods/9dfee450-fd7b-4709-ac31-915c51c31bd7/volumes"
Jul 27 19:48:13.372796 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:13.372762    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf"]
Jul 27 19:48:13.373219 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:13.373091    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dfee450-fd7b-4709-ac31-915c51c31bd7" containerName="revoke-keys"
Jul 27 19:48:13.373219 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:13.373102    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dfee450-fd7b-4709-ac31-915c51c31bd7" containerName="revoke-keys"
Jul 27 19:48:13.373219 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:13.373165    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dfee450-fd7b-4709-ac31-915c51c31bd7" containerName="revoke-keys"
Jul 27 19:48:13.376051 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:13.376034    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf"
Jul 27 19:48:13.378611 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:13.378592    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 19:48:13.378722 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:13.378595    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\""
Jul 27 19:48:13.382800 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:13.382779    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf"]
Jul 27 19:48:13.494508 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:13.494472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d3eb01-dcfa-4cb9-b273-a033ca486d1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf\" (UID: \"92d3eb01-dcfa-4cb9-b273-a033ca486d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf"
Jul 27 19:48:13.595830 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:13.595797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d3eb01-dcfa-4cb9-b273-a033ca486d1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf\" (UID: \"92d3eb01-dcfa-4cb9-b273-a033ca486d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf"
Jul 27 19:48:13.596480 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:13.596460    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d3eb01-dcfa-4cb9-b273-a033ca486d1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf\" (UID: \"92d3eb01-dcfa-4cb9-b273-a033ca486d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf"
Jul 27 19:48:13.687415 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:13.687326    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf"
Jul 27 19:48:13.815378 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:13.815351    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf"]
Jul 27 19:48:13.817496 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:48:13.817467    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d3eb01_dcfa_4cb9_b273_a033ca486d1f.slice/crio-d1121eee138f92b7b60f21a8642ae3b2c0da1345ec0aedd8a9441d54bea4575b WatchSource:0}: Error finding container d1121eee138f92b7b60f21a8642ae3b2c0da1345ec0aedd8a9441d54bea4575b: Status 404 returned error can't find the container with id d1121eee138f92b7b60f21a8642ae3b2c0da1345ec0aedd8a9441d54bea4575b
Jul 27 19:48:14.792202 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:14.792112    2580 generic.go:356] "Generic (PLEG): container finished" podID="92d3eb01-dcfa-4cb9-b273-a033ca486d1f" containerID="ee1da6694558ba0aa7fa290626fe1239658535575539d176097d587af607f249" exitCode=0
Jul 27 19:48:14.792624 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:14.792202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf" event={"ID":"92d3eb01-dcfa-4cb9-b273-a033ca486d1f","Type":"ContainerDied","Data":"ee1da6694558ba0aa7fa290626fe1239658535575539d176097d587af607f249"}
Jul 27 19:48:14.792624 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:14.792266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf" event={"ID":"92d3eb01-dcfa-4cb9-b273-a033ca486d1f","Type":"ContainerStarted","Data":"d1121eee138f92b7b60f21a8642ae3b2c0da1345ec0aedd8a9441d54bea4575b"}
Jul 27 19:48:15.923429 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:15.923406    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf"
Jul 27 19:48:16.019653 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:16.019620    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d3eb01-dcfa-4cb9-b273-a033ca486d1f-maas-api-service-ca\") pod \"92d3eb01-dcfa-4cb9-b273-a033ca486d1f\" (UID: \"92d3eb01-dcfa-4cb9-b273-a033ca486d1f\") "
Jul 27 19:48:16.019944 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:16.019923    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d3eb01-dcfa-4cb9-b273-a033ca486d1f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92d3eb01-dcfa-4cb9-b273-a033ca486d1f" (UID: "92d3eb01-dcfa-4cb9-b273-a033ca486d1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:48:16.120378 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:16.120351    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d3eb01-dcfa-4cb9-b273-a033ca486d1f-maas-api-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:48:16.803446 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:16.803412    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf"
Jul 27 19:48:16.803446 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:16.803418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf" event={"ID":"92d3eb01-dcfa-4cb9-b273-a033ca486d1f","Type":"ContainerDied","Data":"d1121eee138f92b7b60f21a8642ae3b2c0da1345ec0aedd8a9441d54bea4575b"}
Jul 27 19:48:16.803726 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:16.803466    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1121eee138f92b7b60f21a8642ae3b2c0da1345ec0aedd8a9441d54bea4575b"
Jul 27 19:48:28.510761 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:28.510729    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf"]
Jul 27 19:48:28.513413 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:28.513387    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-67e1471d-n2tnf"]
Jul 27 19:48:30.085824 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:48:30.085794    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92d3eb01-dcfa-4cb9-b273-a033ca486d1f" path="/var/lib/kubelet/pods/92d3eb01-dcfa-4cb9-b273-a033ca486d1f/volumes"
Jul 27 19:50:27.566226 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:27.566188    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58"]
Jul 27 19:50:27.566655 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:27.566516    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d3eb01-dcfa-4cb9-b273-a033ca486d1f" containerName="revoke-keys"
Jul 27 19:50:27.566655 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:27.566527    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d3eb01-dcfa-4cb9-b273-a033ca486d1f" containerName="revoke-keys"
Jul 27 19:50:27.566655 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:27.566584    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d3eb01-dcfa-4cb9-b273-a033ca486d1f" containerName="revoke-keys"
Jul 27 19:50:27.569377 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:27.569361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58"
Jul 27 19:50:27.571886 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:27.571868    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 19:50:27.572411 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:27.572393    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\""
Jul 27 19:50:27.578691 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:27.578670    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58"]
Jul 27 19:50:27.615976 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:27.615938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2147ad3c-dc28-43ce-8a55-a871d6735170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae5981c-w2b58\" (UID: \"2147ad3c-dc28-43ce-8a55-a871d6735170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58"
Jul 27 19:50:27.716797 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:27.716757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2147ad3c-dc28-43ce-8a55-a871d6735170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae5981c-w2b58\" (UID: \"2147ad3c-dc28-43ce-8a55-a871d6735170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58"
Jul 27 19:50:27.717413 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:27.717394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2147ad3c-dc28-43ce-8a55-a871d6735170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-eae5981c-w2b58\" (UID: \"2147ad3c-dc28-43ce-8a55-a871d6735170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58"
Jul 27 19:50:27.880753 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:27.880673    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58"
Jul 27 19:50:28.010304 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:28.010279    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58"]
Jul 27 19:50:28.011762 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:50:28.011735    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2147ad3c_dc28_43ce_8a55_a871d6735170.slice/crio-51de114691070e1aee8a109e3724ba5a200a75930270d9c840946ed652765174 WatchSource:0}: Error finding container 51de114691070e1aee8a109e3724ba5a200a75930270d9c840946ed652765174: Status 404 returned error can't find the container with id 51de114691070e1aee8a109e3724ba5a200a75930270d9c840946ed652765174
Jul 27 19:50:28.457919 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:28.457827    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58" event={"ID":"2147ad3c-dc28-43ce-8a55-a871d6735170","Type":"ContainerStarted","Data":"487306fbdca7634aba80de83c4ae91e3767911eeb8b68ddb5c555ffd582e3421"}
Jul 27 19:50:28.457919 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:28.457881    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58" event={"ID":"2147ad3c-dc28-43ce-8a55-a871d6735170","Type":"ContainerStarted","Data":"51de114691070e1aee8a109e3724ba5a200a75930270d9c840946ed652765174"}
Jul 27 19:50:28.475975 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:28.475925    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58" podStartSLOduration=1.4759072039999999 podStartE2EDuration="1.475907204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:50:28.474032851 +0000 UTC m=+1970.989737354" watchObservedRunningTime="2026-07-27 19:50:28.475907204 +0000 UTC m=+1970.991611710"
Jul 27 19:50:29.464650 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:29.464563    2580 generic.go:356] "Generic (PLEG): container finished" podID="2147ad3c-dc28-43ce-8a55-a871d6735170" containerID="487306fbdca7634aba80de83c4ae91e3767911eeb8b68ddb5c555ffd582e3421" exitCode=0
Jul 27 19:50:29.464650 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:29.464636    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58" event={"ID":"2147ad3c-dc28-43ce-8a55-a871d6735170","Type":"ContainerDied","Data":"487306fbdca7634aba80de83c4ae91e3767911eeb8b68ddb5c555ffd582e3421"}
Jul 27 19:50:30.595757 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:30.595729    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58"
Jul 27 19:50:30.645216 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:30.645180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2147ad3c-dc28-43ce-8a55-a871d6735170-maas-api-service-ca\") pod \"2147ad3c-dc28-43ce-8a55-a871d6735170\" (UID: \"2147ad3c-dc28-43ce-8a55-a871d6735170\") "
Jul 27 19:50:30.645562 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:30.645535    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2147ad3c-dc28-43ce-8a55-a871d6735170-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2147ad3c-dc28-43ce-8a55-a871d6735170" (UID: "2147ad3c-dc28-43ce-8a55-a871d6735170"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:50:30.746676 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:30.746594    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2147ad3c-dc28-43ce-8a55-a871d6735170-maas-api-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:50:31.475396 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:31.475365    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58"
Jul 27 19:50:31.475396 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:31.475371    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58" event={"ID":"2147ad3c-dc28-43ce-8a55-a871d6735170","Type":"ContainerDied","Data":"51de114691070e1aee8a109e3724ba5a200a75930270d9c840946ed652765174"}
Jul 27 19:50:31.475608 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:31.475420    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51de114691070e1aee8a109e3724ba5a200a75930270d9c840946ed652765174"
Jul 27 19:50:42.647917 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:42.647878    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58"]
Jul 27 19:50:42.649790 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:42.649768    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eae5981c-w2b58"]
Jul 27 19:50:44.086051 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:50:44.086019    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2147ad3c-dc28-43ce-8a55-a871d6735170" path="/var/lib/kubelet/pods/2147ad3c-dc28-43ce-8a55-a871d6735170/volumes"
Jul 27 19:51:38.260448 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.260417    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"]
Jul 27 19:51:38.261070 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.260747    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2147ad3c-dc28-43ce-8a55-a871d6735170" containerName="revoke-keys"
Jul 27 19:51:38.261070 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.260758    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2147ad3c-dc28-43ce-8a55-a871d6735170" containerName="revoke-keys"
Jul 27 19:51:38.261070 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.260817    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2147ad3c-dc28-43ce-8a55-a871d6735170" containerName="revoke-keys"
Jul 27 19:51:38.263599 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.263581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.266194 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.266173    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ef75af22\"/\"default-dockercfg-62n7q\""
Jul 27 19:51:38.266321 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.266212    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ef75af22\"/\"kube-root-ca.crt\""
Jul 27 19:51:38.267028 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.267010    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ef75af22\"/\"e2e-lifecycle-model-ef75af22-kserve-self-signed-certs\""
Jul 27 19:51:38.267115 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.267066    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ef75af22\"/\"openshift-service-ca.crt\""
Jul 27 19:51:38.279885 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.279860    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"]
Jul 27 19:51:38.411213 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.411185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-tmp-dir\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.411393 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.411232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2j4t\" (UniqueName: \"kubernetes.io/projected/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-kube-api-access-s2j4t\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.411393 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.411350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-tls-certs\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.411393 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.411387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-model-cache\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.411498 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.411425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-home\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.411498 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.411476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-dshm\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.512052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.511968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-tls-certs\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.512052 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.512015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-model-cache\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.512299 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.512066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-home\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.512299 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.512179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-dshm\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.512299 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.512234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-tmp-dir\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.512478 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.512323    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2j4t\" (UniqueName: \"kubernetes.io/projected/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-kube-api-access-s2j4t\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.512564 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.512529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-model-cache\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.512564 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.512545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-home\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.512675 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.512651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-tmp-dir\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.514522 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.514501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-dshm\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.514736 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.514715    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-tls-certs\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.521053 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.521028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2j4t\" (UniqueName: \"kubernetes.io/projected/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-kube-api-access-s2j4t\") pod \"e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") " pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.572011 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.571976    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:38.705989 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.705962    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"]
Jul 27 19:51:38.708487 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:51:38.708451    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c6e4f91_a1fd_44a7_8dcf_b2b0cfe751cd.slice/crio-edd5c87dcb73cb07b7733bd4c56e22cb44b41e3b1000fb0d2b6f6889162e70f3 WatchSource:0}: Error finding container edd5c87dcb73cb07b7733bd4c56e22cb44b41e3b1000fb0d2b6f6889162e70f3: Status 404 returned error can't find the container with id edd5c87dcb73cb07b7733bd4c56e22cb44b41e3b1000fb0d2b6f6889162e70f3
Jul 27 19:51:38.818267 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.818214    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk" event={"ID":"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd","Type":"ContainerStarted","Data":"b3272a1f748e72bc9cc78891549ab326eb2d6ef0d95991530a7c068f9669d113"}
Jul 27 19:51:38.818267 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.818270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk" event={"ID":"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd","Type":"ContainerStarted","Data":"edd5c87dcb73cb07b7733bd4c56e22cb44b41e3b1000fb0d2b6f6889162e70f3"}
Jul 27 19:51:38.835535 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:38.835490    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk" podStartSLOduration=0.835474425 podStartE2EDuration="835.474425ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:51:38.834096679 +0000 UTC m=+2041.349801183" watchObservedRunningTime="2026-07-27 19:51:38.835474425 +0000 UTC m=+2041.351178929"
Jul 27 19:51:48.572881 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:48.572850    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:48.572881 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:48.572887    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:48.585487 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:48.585462    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:51:48.878065 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:51:48.877983    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:52:35.384062 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:35.384021    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"]
Jul 27 19:52:38.184547 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:38.184519    2580 scope.go:117] "RemoveContainer" containerID="5524ca67f42200f9fce815fa6c7645d39f10dbbfe999f8904a7baa9c35121710"
Jul 27 19:52:50.402865 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.402818    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk" podUID="2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd" containerName="main" containerID="cri-o://b3272a1f748e72bc9cc78891549ab326eb2d6ef0d95991530a7c068f9669d113" gracePeriod=45
Jul 27 19:52:50.648287 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.648261    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:52:50.827382 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.827353    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-dshm\") pod \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") "
Jul 27 19:52:50.827552 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.827391    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-tls-certs\") pod \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") "
Jul 27 19:52:50.827552 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.827420    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-tmp-dir\") pod \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") "
Jul 27 19:52:50.827552 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.827535    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-model-cache\") pod \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") "
Jul 27 19:52:50.827704 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.827566    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-home\") pod \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") "
Jul 27 19:52:50.827704 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.827603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2j4t\" (UniqueName: \"kubernetes.io/projected/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-kube-api-access-s2j4t\") pod \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\" (UID: \"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd\") "
Jul 27 19:52:50.827704 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.827660    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd" (UID: "2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:52:50.827863 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.827793    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd" (UID: "2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:52:50.827863 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.827847    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-home" (OuterVolumeSpecName: "home") pod "2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd" (UID: "2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:52:50.827954 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.827867    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:52:50.829584 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.829563    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd" (UID: "2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:52:50.830136 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.830116    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-dshm" (OuterVolumeSpecName: "dshm") pod "2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd" (UID: "2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:52:50.830237 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.830218    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-kube-api-access-s2j4t" (OuterVolumeSpecName: "kube-api-access-s2j4t") pod "2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd" (UID: "2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd"). InnerVolumeSpecName "kube-api-access-s2j4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:52:50.929142 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.929108    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:52:50.929142 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.929138    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:52:50.929142 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.929149    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2j4t\" (UniqueName: \"kubernetes.io/projected/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-kube-api-access-s2j4t\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:52:50.929397 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.929159    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:52:50.929397 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:50.929168    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:52:51.184041 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:51.183953    2580 generic.go:356] "Generic (PLEG): container finished" podID="2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd" containerID="b3272a1f748e72bc9cc78891549ab326eb2d6ef0d95991530a7c068f9669d113" exitCode=0
Jul 27 19:52:51.184041 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:51.184006    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"
Jul 27 19:52:51.184229 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:51.184041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk" event={"ID":"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd","Type":"ContainerDied","Data":"b3272a1f748e72bc9cc78891549ab326eb2d6ef0d95991530a7c068f9669d113"}
Jul 27 19:52:51.184229 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:51.184082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk" event={"ID":"2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd","Type":"ContainerDied","Data":"edd5c87dcb73cb07b7733bd4c56e22cb44b41e3b1000fb0d2b6f6889162e70f3"}
Jul 27 19:52:51.184229 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:51.184099    2580 scope.go:117] "RemoveContainer" containerID="b3272a1f748e72bc9cc78891549ab326eb2d6ef0d95991530a7c068f9669d113"
Jul 27 19:52:51.193145 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:51.193129    2580 scope.go:117] "RemoveContainer" containerID="b3272a1f748e72bc9cc78891549ab326eb2d6ef0d95991530a7c068f9669d113"
Jul 27 19:52:51.193420 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:52:51.193402    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3272a1f748e72bc9cc78891549ab326eb2d6ef0d95991530a7c068f9669d113\": container with ID starting with b3272a1f748e72bc9cc78891549ab326eb2d6ef0d95991530a7c068f9669d113 not found: ID does not exist" containerID="b3272a1f748e72bc9cc78891549ab326eb2d6ef0d95991530a7c068f9669d113"
Jul 27 19:52:51.193483 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:51.193427    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3272a1f748e72bc9cc78891549ab326eb2d6ef0d95991530a7c068f9669d113"} err="failed to get container status \"b3272a1f748e72bc9cc78891549ab326eb2d6ef0d95991530a7c068f9669d113\": rpc error: code = NotFound desc = could not find container \"b3272a1f748e72bc9cc78891549ab326eb2d6ef0d95991530a7c068f9669d113\": container with ID starting with b3272a1f748e72bc9cc78891549ab326eb2d6ef0d95991530a7c068f9669d113 not found: ID does not exist"
Jul 27 19:52:51.205969 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:51.205947    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"]
Jul 27 19:52:51.210885 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:51.210862    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ef75af22/e2e-lifecycle-model-ef75af22-kserve-959594475-4hlkk"]
Jul 27 19:52:52.086647 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:52:52.086619    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd" path="/var/lib/kubelet/pods/2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd/volumes"
Jul 27 19:54:38.214571 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:54:38.214544    2580 scope.go:117] "RemoveContainer" containerID="ee1da6694558ba0aa7fa290626fe1239658535575539d176097d587af607f249"
Jul 27 19:55:01.784054 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.783981    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"]
Jul 27 19:55:01.784425 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.784329    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd" containerName="main"
Jul 27 19:55:01.784425 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.784340    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd" containerName="main"
Jul 27 19:55:01.784425 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.784396    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c6e4f91-a1fd-44a7-8dcf-b2b0cfe751cd" containerName="main"
Jul 27 19:55:01.787359 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.787343    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:01.789989 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.789965    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7bbd07\"/\"kube-root-ca.crt\""
Jul 27 19:55:01.790797 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.790750    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7bbd07\"/\"openshift-service-ca.crt\""
Jul 27 19:55:01.790797 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.790778    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7bbd07\"/\"default-dockercfg-699jb\""
Jul 27 19:55:01.790939 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.790837    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7bbd07\"/\"test-model-7bbd07-kserve-self-signed-certs\""
Jul 27 19:55:01.804091 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.804068    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"]
Jul 27 19:55:01.937743 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.937705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccbfh\" (UniqueName: \"kubernetes.io/projected/d20099ad-36c0-46eb-ae91-573226e88351-kube-api-access-ccbfh\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:01.937919 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.937753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-home\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:01.937919 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.937787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-model-cache\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:01.937919 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.937833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20099ad-36c0-46eb-ae91-573226e88351-tls-certs\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:01.937919 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.937866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-tmp-dir\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:01.937919 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:01.937902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-dshm\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:02.038532 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.038507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccbfh\" (UniqueName: \"kubernetes.io/projected/d20099ad-36c0-46eb-ae91-573226e88351-kube-api-access-ccbfh\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:02.038696 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.038548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-home\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:02.038696 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.038576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-model-cache\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:02.038696 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.038593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20099ad-36c0-46eb-ae91-573226e88351-tls-certs\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:02.038696 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.038645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-tmp-dir\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:02.038696 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.038691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-dshm\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:02.039003 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.038981    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-home\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:02.039054 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.039003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-tmp-dir\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:02.039054 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.039040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-model-cache\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:02.040967 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.040948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-dshm\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:02.041209 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.041190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20099ad-36c0-46eb-ae91-573226e88351-tls-certs\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:02.047039 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.047015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccbfh\" (UniqueName: \"kubernetes.io/projected/d20099ad-36c0-46eb-ae91-573226e88351-kube-api-access-ccbfh\") pod \"test-model-7bbd07-kserve-6f5cf596f-ckxzv\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") " pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:02.096983 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.096960    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:02.226216 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.226188    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"]
Jul 27 19:55:02.227413 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:55:02.227381    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd20099ad_36c0_46eb_ae91_573226e88351.slice/crio-ec3c759d6d35d71692be02b8ba7e8fa027a677f5ea713f89ff4280e085cf04fb WatchSource:0}: Error finding container ec3c759d6d35d71692be02b8ba7e8fa027a677f5ea713f89ff4280e085cf04fb: Status 404 returned error can't find the container with id ec3c759d6d35d71692be02b8ba7e8fa027a677f5ea713f89ff4280e085cf04fb
Jul 27 19:55:02.818604 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.818576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv" event={"ID":"d20099ad-36c0-46eb-ae91-573226e88351","Type":"ContainerStarted","Data":"784e1d66d4748260f45457931b634dcaee72f867d37754e752a50fcbe9d45c5b"}
Jul 27 19:55:02.819000 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.818614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv" event={"ID":"d20099ad-36c0-46eb-ae91-573226e88351","Type":"ContainerStarted","Data":"ec3c759d6d35d71692be02b8ba7e8fa027a677f5ea713f89ff4280e085cf04fb"}
Jul 27 19:55:02.836502 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:02.836462    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv" podStartSLOduration=1.836449078 podStartE2EDuration="1.836449078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:55:02.834439157 +0000 UTC m=+2245.350143662" watchObservedRunningTime="2026-07-27 19:55:02.836449078 +0000 UTC m=+2245.352153582"
Jul 27 19:55:12.097832 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:12.097796    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:12.098427 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:12.097847    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:12.112834 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:12.112808    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:12.877141 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:12.877108    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:55:22.887980 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:22.887897    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"]
Jul 27 19:55:22.891131 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:22.891115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:22.895043 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:22.895016    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4ce65d\"/\"kube-root-ca.crt\""
Jul 27 19:55:22.895511 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:22.895491    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4ce65d\"/\"openshift-service-ca.crt\""
Jul 27 19:55:22.895598 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:22.895491    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4ce65d\"/\"test-model-4ce65d-kserve-self-signed-certs\""
Jul 27 19:55:22.895598 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:22.895523    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4ce65d\"/\"default-dockercfg-7cspv\""
Jul 27 19:55:22.904390 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:22.904367    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"]
Jul 27 19:55:22.918729 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:22.918704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a88a19bd-6d79-4d8a-80da-e242c5882277-tls-certs\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:22.918828 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:22.918783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-tmp-dir\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:22.918828 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:22.918807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-home\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:22.918919 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:22.918856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-model-cache\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:22.918963 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:22.918942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-dshm\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:22.918998 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:22.918968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-799kv\" (UniqueName: \"kubernetes.io/projected/a88a19bd-6d79-4d8a-80da-e242c5882277-kube-api-access-799kv\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:23.019809 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.019779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a88a19bd-6d79-4d8a-80da-e242c5882277-tls-certs\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:23.019957 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.019824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-tmp-dir\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:23.019957 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.019841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-home\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:23.019957 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.019892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-model-cache\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:23.019957 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.019944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-dshm\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:23.020234 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.019969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-799kv\" (UniqueName: \"kubernetes.io/projected/a88a19bd-6d79-4d8a-80da-e242c5882277-kube-api-access-799kv\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:23.020320 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.020266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-tmp-dir\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:23.020366 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.020308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-model-cache\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:23.020366 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.020342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-home\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:23.022261 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.022223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-dshm\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:23.022443 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.022423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a88a19bd-6d79-4d8a-80da-e242c5882277-tls-certs\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:23.028728 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.028701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-799kv\" (UniqueName: \"kubernetes.io/projected/a88a19bd-6d79-4d8a-80da-e242c5882277-kube-api-access-799kv\") pod \"test-model-4ce65d-kserve-6b7ff7b54c-vlvp5\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") " pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:23.200868 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.200783    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:23.334088 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.334062    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"]
Jul 27 19:55:23.335105 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:55:23.335080    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda88a19bd_6d79_4d8a_80da_e242c5882277.slice/crio-30435b101e52cc321c2896cacbfba033e620485e595e6eb5272852544094cf69 WatchSource:0}: Error finding container 30435b101e52cc321c2896cacbfba033e620485e595e6eb5272852544094cf69: Status 404 returned error can't find the container with id 30435b101e52cc321c2896cacbfba033e620485e595e6eb5272852544094cf69
Jul 27 19:55:23.923795 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.923763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5" event={"ID":"a88a19bd-6d79-4d8a-80da-e242c5882277","Type":"ContainerStarted","Data":"668d12c61e3ea0f1d579716a4bb48d79ca20c16767fe0b21de85a0465a059942"}
Jul 27 19:55:23.924209 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.923805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5" event={"ID":"a88a19bd-6d79-4d8a-80da-e242c5882277","Type":"ContainerStarted","Data":"30435b101e52cc321c2896cacbfba033e620485e595e6eb5272852544094cf69"}
Jul 27 19:55:23.943835 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:23.943788    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5" podStartSLOduration=1.9437750999999999 podStartE2EDuration="1.9437751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:55:23.941270028 +0000 UTC m=+2266.456974523" watchObservedRunningTime="2026-07-27 19:55:23.9437751 +0000 UTC m=+2266.459479603"
Jul 27 19:55:33.201497 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:33.201450    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:33.201898 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:33.201517    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:33.214380 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:33.214354    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:55:33.989565 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:55:33.989540    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:56:02.615525 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:02.615489    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"]
Jul 27 19:56:03.610799 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:03.610765    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"]
Jul 27 19:56:17.633999 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.633948    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv" podUID="d20099ad-36c0-46eb-ae91-573226e88351" containerName="main" containerID="cri-o://784e1d66d4748260f45457931b634dcaee72f867d37754e752a50fcbe9d45c5b" gracePeriod=45
Jul 27 19:56:17.884503 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.884449    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:56:17.989232 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.989202    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-tmp-dir\") pod \"d20099ad-36c0-46eb-ae91-573226e88351\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") "
Jul 27 19:56:17.989408 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.989270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-dshm\") pod \"d20099ad-36c0-46eb-ae91-573226e88351\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") "
Jul 27 19:56:17.989408 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.989315    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-model-cache\") pod \"d20099ad-36c0-46eb-ae91-573226e88351\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") "
Jul 27 19:56:17.989408 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.989352    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccbfh\" (UniqueName: \"kubernetes.io/projected/d20099ad-36c0-46eb-ae91-573226e88351-kube-api-access-ccbfh\") pod \"d20099ad-36c0-46eb-ae91-573226e88351\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") "
Jul 27 19:56:17.989408 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.989386    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20099ad-36c0-46eb-ae91-573226e88351-tls-certs\") pod \"d20099ad-36c0-46eb-ae91-573226e88351\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") "
Jul 27 19:56:17.989570 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.989442    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-home\") pod \"d20099ad-36c0-46eb-ae91-573226e88351\" (UID: \"d20099ad-36c0-46eb-ae91-573226e88351\") "
Jul 27 19:56:17.989570 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.989443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d20099ad-36c0-46eb-ae91-573226e88351" (UID: "d20099ad-36c0-46eb-ae91-573226e88351"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:56:17.989570 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.989534    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-model-cache" (OuterVolumeSpecName: "model-cache") pod "d20099ad-36c0-46eb-ae91-573226e88351" (UID: "d20099ad-36c0-46eb-ae91-573226e88351"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:56:17.989685 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.989666    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:56:17.989747 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.989696    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-home" (OuterVolumeSpecName: "home") pod "d20099ad-36c0-46eb-ae91-573226e88351" (UID: "d20099ad-36c0-46eb-ae91-573226e88351"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:56:17.989747 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.989709    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:56:17.991576 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.991549    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-dshm" (OuterVolumeSpecName: "dshm") pod "d20099ad-36c0-46eb-ae91-573226e88351" (UID: "d20099ad-36c0-46eb-ae91-573226e88351"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:56:17.991686 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.991593    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20099ad-36c0-46eb-ae91-573226e88351-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d20099ad-36c0-46eb-ae91-573226e88351" (UID: "d20099ad-36c0-46eb-ae91-573226e88351"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:56:17.991686 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:17.991630    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20099ad-36c0-46eb-ae91-573226e88351-kube-api-access-ccbfh" (OuterVolumeSpecName: "kube-api-access-ccbfh") pod "d20099ad-36c0-46eb-ae91-573226e88351" (UID: "d20099ad-36c0-46eb-ae91-573226e88351"). InnerVolumeSpecName "kube-api-access-ccbfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:56:18.090779 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.090747    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccbfh\" (UniqueName: \"kubernetes.io/projected/d20099ad-36c0-46eb-ae91-573226e88351-kube-api-access-ccbfh\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:56:18.090779 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.090774    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20099ad-36c0-46eb-ae91-573226e88351-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:56:18.090779 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.090784    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:56:18.091008 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.090791    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20099ad-36c0-46eb-ae91-573226e88351-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:56:18.192741 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.192644    2580 generic.go:356] "Generic (PLEG): container finished" podID="d20099ad-36c0-46eb-ae91-573226e88351" containerID="784e1d66d4748260f45457931b634dcaee72f867d37754e752a50fcbe9d45c5b" exitCode=0
Jul 27 19:56:18.192741 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.192688    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"
Jul 27 19:56:18.192933 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.192737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv" event={"ID":"d20099ad-36c0-46eb-ae91-573226e88351","Type":"ContainerDied","Data":"784e1d66d4748260f45457931b634dcaee72f867d37754e752a50fcbe9d45c5b"}
Jul 27 19:56:18.192933 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.192780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv" event={"ID":"d20099ad-36c0-46eb-ae91-573226e88351","Type":"ContainerDied","Data":"ec3c759d6d35d71692be02b8ba7e8fa027a677f5ea713f89ff4280e085cf04fb"}
Jul 27 19:56:18.192933 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.192797    2580 scope.go:117] "RemoveContainer" containerID="784e1d66d4748260f45457931b634dcaee72f867d37754e752a50fcbe9d45c5b"
Jul 27 19:56:18.201252 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.201216    2580 scope.go:117] "RemoveContainer" containerID="784e1d66d4748260f45457931b634dcaee72f867d37754e752a50fcbe9d45c5b"
Jul 27 19:56:18.201493 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:56:18.201475    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784e1d66d4748260f45457931b634dcaee72f867d37754e752a50fcbe9d45c5b\": container with ID starting with 784e1d66d4748260f45457931b634dcaee72f867d37754e752a50fcbe9d45c5b not found: ID does not exist" containerID="784e1d66d4748260f45457931b634dcaee72f867d37754e752a50fcbe9d45c5b"
Jul 27 19:56:18.201545 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.201503    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784e1d66d4748260f45457931b634dcaee72f867d37754e752a50fcbe9d45c5b"} err="failed to get container status \"784e1d66d4748260f45457931b634dcaee72f867d37754e752a50fcbe9d45c5b\": rpc error: code = NotFound desc = could not find container \"784e1d66d4748260f45457931b634dcaee72f867d37754e752a50fcbe9d45c5b\": container with ID starting with 784e1d66d4748260f45457931b634dcaee72f867d37754e752a50fcbe9d45c5b not found: ID does not exist"
Jul 27 19:56:18.210235 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.210212    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"]
Jul 27 19:56:18.214106 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.214086    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7bbd07/test-model-7bbd07-kserve-6f5cf596f-ckxzv"]
Jul 27 19:56:18.629328 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.629278    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5" podUID="a88a19bd-6d79-4d8a-80da-e242c5882277" containerName="main" containerID="cri-o://668d12c61e3ea0f1d579716a4bb48d79ca20c16767fe0b21de85a0465a059942" gracePeriod=45
Jul 27 19:56:18.881696 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.881635    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:56:18.996847 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.996817    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-dshm\") pod \"a88a19bd-6d79-4d8a-80da-e242c5882277\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") "
Jul 27 19:56:18.997004 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.996893    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a88a19bd-6d79-4d8a-80da-e242c5882277-tls-certs\") pod \"a88a19bd-6d79-4d8a-80da-e242c5882277\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") "
Jul 27 19:56:18.997004 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.996926    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-799kv\" (UniqueName: \"kubernetes.io/projected/a88a19bd-6d79-4d8a-80da-e242c5882277-kube-api-access-799kv\") pod \"a88a19bd-6d79-4d8a-80da-e242c5882277\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") "
Jul 27 19:56:18.997004 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.996944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-home\") pod \"a88a19bd-6d79-4d8a-80da-e242c5882277\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") "
Jul 27 19:56:18.997004 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.996983    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-model-cache\") pod \"a88a19bd-6d79-4d8a-80da-e242c5882277\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") "
Jul 27 19:56:18.997198 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.997028    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-tmp-dir\") pod \"a88a19bd-6d79-4d8a-80da-e242c5882277\" (UID: \"a88a19bd-6d79-4d8a-80da-e242c5882277\") "
Jul 27 19:56:18.997303 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.997219    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-home" (OuterVolumeSpecName: "home") pod "a88a19bd-6d79-4d8a-80da-e242c5882277" (UID: "a88a19bd-6d79-4d8a-80da-e242c5882277"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:56:18.997367 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.997313    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-model-cache" (OuterVolumeSpecName: "model-cache") pod "a88a19bd-6d79-4d8a-80da-e242c5882277" (UID: "a88a19bd-6d79-4d8a-80da-e242c5882277"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:56:18.997411 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.997364    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a88a19bd-6d79-4d8a-80da-e242c5882277" (UID: "a88a19bd-6d79-4d8a-80da-e242c5882277"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:56:18.999145 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.999119    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-dshm" (OuterVolumeSpecName: "dshm") pod "a88a19bd-6d79-4d8a-80da-e242c5882277" (UID: "a88a19bd-6d79-4d8a-80da-e242c5882277"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:56:18.999234 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.999195    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88a19bd-6d79-4d8a-80da-e242c5882277-kube-api-access-799kv" (OuterVolumeSpecName: "kube-api-access-799kv") pod "a88a19bd-6d79-4d8a-80da-e242c5882277" (UID: "a88a19bd-6d79-4d8a-80da-e242c5882277"). InnerVolumeSpecName "kube-api-access-799kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:56:18.999234 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:18.999199    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a88a19bd-6d79-4d8a-80da-e242c5882277-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a88a19bd-6d79-4d8a-80da-e242c5882277" (UID: "a88a19bd-6d79-4d8a-80da-e242c5882277"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:56:19.098328 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.098301    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a88a19bd-6d79-4d8a-80da-e242c5882277-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:56:19.098328 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.098323    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-799kv\" (UniqueName: \"kubernetes.io/projected/a88a19bd-6d79-4d8a-80da-e242c5882277-kube-api-access-799kv\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:56:19.098328 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.098332    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:56:19.098571 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.098341    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:56:19.098571 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.098350    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:56:19.098571 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.098357    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a88a19bd-6d79-4d8a-80da-e242c5882277-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:56:19.199336 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.199225    2580 generic.go:356] "Generic (PLEG): container finished" podID="a88a19bd-6d79-4d8a-80da-e242c5882277" containerID="668d12c61e3ea0f1d579716a4bb48d79ca20c16767fe0b21de85a0465a059942" exitCode=0
Jul 27 19:56:19.199336 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.199296    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"
Jul 27 19:56:19.199336 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.199316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5" event={"ID":"a88a19bd-6d79-4d8a-80da-e242c5882277","Type":"ContainerDied","Data":"668d12c61e3ea0f1d579716a4bb48d79ca20c16767fe0b21de85a0465a059942"}
Jul 27 19:56:19.199593 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.199366    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5" event={"ID":"a88a19bd-6d79-4d8a-80da-e242c5882277","Type":"ContainerDied","Data":"30435b101e52cc321c2896cacbfba033e620485e595e6eb5272852544094cf69"}
Jul 27 19:56:19.199593 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.199388    2580 scope.go:117] "RemoveContainer" containerID="668d12c61e3ea0f1d579716a4bb48d79ca20c16767fe0b21de85a0465a059942"
Jul 27 19:56:19.208628 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.208610    2580 scope.go:117] "RemoveContainer" containerID="668d12c61e3ea0f1d579716a4bb48d79ca20c16767fe0b21de85a0465a059942"
Jul 27 19:56:19.208886 ip-10-0-129-103 kubenswrapper[2580]: E0727 19:56:19.208864    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"668d12c61e3ea0f1d579716a4bb48d79ca20c16767fe0b21de85a0465a059942\": container with ID starting with 668d12c61e3ea0f1d579716a4bb48d79ca20c16767fe0b21de85a0465a059942 not found: ID does not exist" containerID="668d12c61e3ea0f1d579716a4bb48d79ca20c16767fe0b21de85a0465a059942"
Jul 27 19:56:19.208940 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.208893    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668d12c61e3ea0f1d579716a4bb48d79ca20c16767fe0b21de85a0465a059942"} err="failed to get container status \"668d12c61e3ea0f1d579716a4bb48d79ca20c16767fe0b21de85a0465a059942\": rpc error: code = NotFound desc = could not find container \"668d12c61e3ea0f1d579716a4bb48d79ca20c16767fe0b21de85a0465a059942\": container with ID starting with 668d12c61e3ea0f1d579716a4bb48d79ca20c16767fe0b21de85a0465a059942 not found: ID does not exist"
Jul 27 19:56:19.220843 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.220820    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"]
Jul 27 19:56:19.224901 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:19.224883    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4ce65d/test-model-4ce65d-kserve-6b7ff7b54c-vlvp5"]
Jul 27 19:56:20.086638 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:20.086607    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a88a19bd-6d79-4d8a-80da-e242c5882277" path="/var/lib/kubelet/pods/a88a19bd-6d79-4d8a-80da-e242c5882277/volumes"
Jul 27 19:56:20.086987 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:20.086965    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d20099ad-36c0-46eb-ae91-573226e88351" path="/var/lib/kubelet/pods/d20099ad-36c0-46eb-ae91-573226e88351/volumes"
Jul 27 19:56:25.976923 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:25.976889    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9"]
Jul 27 19:56:25.977318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:25.977190    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a88a19bd-6d79-4d8a-80da-e242c5882277" containerName="main"
Jul 27 19:56:25.977318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:25.977201    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a88a19bd-6d79-4d8a-80da-e242c5882277" containerName="main"
Jul 27 19:56:25.977318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:25.977219    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20099ad-36c0-46eb-ae91-573226e88351" containerName="main"
Jul 27 19:56:25.977318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:25.977226    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20099ad-36c0-46eb-ae91-573226e88351" containerName="main"
Jul 27 19:56:25.977318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:25.977287    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20099ad-36c0-46eb-ae91-573226e88351" containerName="main"
Jul 27 19:56:25.977318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:25.977297    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a88a19bd-6d79-4d8a-80da-e242c5882277" containerName="main"
Jul 27 19:56:25.981529 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:25.981509    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9"
Jul 27 19:56:25.984085 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:25.984065    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\""
Jul 27 19:56:25.984508 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:25.984491    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 19:56:25.991163 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:25.991138    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9"]
Jul 27 19:56:26.059303 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:26.059277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b25478d-fbad-4372-8d59-e0f013713e64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9\" (UID: \"7b25478d-fbad-4372-8d59-e0f013713e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9"
Jul 27 19:56:26.160221 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:26.160194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b25478d-fbad-4372-8d59-e0f013713e64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9\" (UID: \"7b25478d-fbad-4372-8d59-e0f013713e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9"
Jul 27 19:56:26.160788 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:26.160770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b25478d-fbad-4372-8d59-e0f013713e64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9\" (UID: \"7b25478d-fbad-4372-8d59-e0f013713e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9"
Jul 27 19:56:26.292559 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:26.292536    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9"
Jul 27 19:56:26.626160 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:26.626135    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9"]
Jul 27 19:56:26.627603 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:56:26.627576    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b25478d_fbad_4372_8d59_e0f013713e64.slice/crio-17913782be54344af8c42d391aad8d200d0b3efb88bf5fc8e0b183e923d44d73 WatchSource:0}: Error finding container 17913782be54344af8c42d391aad8d200d0b3efb88bf5fc8e0b183e923d44d73: Status 404 returned error can't find the container with id 17913782be54344af8c42d391aad8d200d0b3efb88bf5fc8e0b183e923d44d73
Jul 27 19:56:27.242613 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:27.242530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9" event={"ID":"7b25478d-fbad-4372-8d59-e0f013713e64","Type":"ContainerStarted","Data":"b3b64140ecad56aadf2697916a1355f9dd7c1f8e465308f200fadcb70d4d7b06"}
Jul 27 19:56:27.242613 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:27.242588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9" event={"ID":"7b25478d-fbad-4372-8d59-e0f013713e64","Type":"ContainerStarted","Data":"17913782be54344af8c42d391aad8d200d0b3efb88bf5fc8e0b183e923d44d73"}
Jul 27 19:56:27.260882 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:27.260832    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9" podStartSLOduration=2.260819596 podStartE2EDuration="2.260819596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:56:27.25801356 +0000 UTC m=+2329.773718065" watchObservedRunningTime="2026-07-27 19:56:27.260819596 +0000 UTC m=+2329.776524100"
Jul 27 19:56:28.249414 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:28.249372    2580 generic.go:356] "Generic (PLEG): container finished" podID="7b25478d-fbad-4372-8d59-e0f013713e64" containerID="b3b64140ecad56aadf2697916a1355f9dd7c1f8e465308f200fadcb70d4d7b06" exitCode=0
Jul 27 19:56:28.249822 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:28.249459    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9" event={"ID":"7b25478d-fbad-4372-8d59-e0f013713e64","Type":"ContainerDied","Data":"b3b64140ecad56aadf2697916a1355f9dd7c1f8e465308f200fadcb70d4d7b06"}
Jul 27 19:56:29.376306 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:29.376283    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9"
Jul 27 19:56:29.485192 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:29.485153    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b25478d-fbad-4372-8d59-e0f013713e64-maas-api-service-ca\") pod \"7b25478d-fbad-4372-8d59-e0f013713e64\" (UID: \"7b25478d-fbad-4372-8d59-e0f013713e64\") "
Jul 27 19:56:29.485515 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:29.485497    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b25478d-fbad-4372-8d59-e0f013713e64-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b25478d-fbad-4372-8d59-e0f013713e64" (UID: "7b25478d-fbad-4372-8d59-e0f013713e64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:56:29.585840 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:29.585807    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b25478d-fbad-4372-8d59-e0f013713e64-maas-api-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:56:30.260865 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:30.260834    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9"
Jul 27 19:56:30.260865 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:30.260860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9" event={"ID":"7b25478d-fbad-4372-8d59-e0f013713e64","Type":"ContainerDied","Data":"17913782be54344af8c42d391aad8d200d0b3efb88bf5fc8e0b183e923d44d73"}
Jul 27 19:56:30.261086 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:30.260908    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17913782be54344af8c42d391aad8d200d0b3efb88bf5fc8e0b183e923d44d73"
Jul 27 19:56:38.244128 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:38.244096    2580 scope.go:117] "RemoveContainer" containerID="487306fbdca7634aba80de83c4ae91e3767911eeb8b68ddb5c555ffd582e3421"
Jul 27 19:56:41.065476 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:41.065437    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9"]
Jul 27 19:56:41.068213 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:41.068188    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ce65d-slxn9"]
Jul 27 19:56:42.086032 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:56:42.085996    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b25478d-fbad-4372-8d59-e0f013713e64" path="/var/lib/kubelet/pods/7b25478d-fbad-4372-8d59-e0f013713e64/volumes"
Jul 27 19:57:12.734829 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:12.734794    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw"]
Jul 27 19:57:12.735318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:12.735122    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b25478d-fbad-4372-8d59-e0f013713e64" containerName="revoke-keys"
Jul 27 19:57:12.735318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:12.735133    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b25478d-fbad-4372-8d59-e0f013713e64" containerName="revoke-keys"
Jul 27 19:57:12.735318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:12.735193    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b25478d-fbad-4372-8d59-e0f013713e64" containerName="revoke-keys"
Jul 27 19:57:12.737999 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:12.737984    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw"
Jul 27 19:57:12.740174 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:12.740150    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 19:57:12.740318 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:12.740269    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\""
Jul 27 19:57:12.745915 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:12.745892    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw"]
Jul 27 19:57:12.840367 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:12.840338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c057de9f-5a6e-4dec-ac31-5464f4afd550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw\" (UID: \"c057de9f-5a6e-4dec-ac31-5464f4afd550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw"
Jul 27 19:57:12.940903 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:12.940869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c057de9f-5a6e-4dec-ac31-5464f4afd550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw\" (UID: \"c057de9f-5a6e-4dec-ac31-5464f4afd550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw"
Jul 27 19:57:12.941518 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:12.941500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c057de9f-5a6e-4dec-ac31-5464f4afd550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw\" (UID: \"c057de9f-5a6e-4dec-ac31-5464f4afd550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw"
Jul 27 19:57:13.048781 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:13.048758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw"
Jul 27 19:57:13.386373 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:13.386348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw"]
Jul 27 19:57:13.388396 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:57:13.388374    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc057de9f_5a6e_4dec_ac31_5464f4afd550.slice/crio-6db3699cf9c6ca8e4861bb52868ef20a92b5b79c26f6ce0170a6c75154487f88 WatchSource:0}: Error finding container 6db3699cf9c6ca8e4861bb52868ef20a92b5b79c26f6ce0170a6c75154487f88: Status 404 returned error can't find the container with id 6db3699cf9c6ca8e4861bb52868ef20a92b5b79c26f6ce0170a6c75154487f88
Jul 27 19:57:13.481404 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:13.481364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw" event={"ID":"c057de9f-5a6e-4dec-ac31-5464f4afd550","Type":"ContainerStarted","Data":"6db3699cf9c6ca8e4861bb52868ef20a92b5b79c26f6ce0170a6c75154487f88"}
Jul 27 19:57:14.487740 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:14.487656    2580 generic.go:356] "Generic (PLEG): container finished" podID="c057de9f-5a6e-4dec-ac31-5464f4afd550" containerID="2023586a2b7d2eabc4a71720c2839985b8a7618f72a297267a6aa5b251dcc2f2" exitCode=0
Jul 27 19:57:14.488077 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:14.487743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw" event={"ID":"c057de9f-5a6e-4dec-ac31-5464f4afd550","Type":"ContainerDied","Data":"2023586a2b7d2eabc4a71720c2839985b8a7618f72a297267a6aa5b251dcc2f2"}
Jul 27 19:57:15.640347 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:15.640323    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw"
Jul 27 19:57:15.664510 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:15.664483    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c057de9f-5a6e-4dec-ac31-5464f4afd550-maas-api-service-ca\") pod \"c057de9f-5a6e-4dec-ac31-5464f4afd550\" (UID: \"c057de9f-5a6e-4dec-ac31-5464f4afd550\") "
Jul 27 19:57:15.664841 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:15.664819    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c057de9f-5a6e-4dec-ac31-5464f4afd550-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c057de9f-5a6e-4dec-ac31-5464f4afd550" (UID: "c057de9f-5a6e-4dec-ac31-5464f4afd550"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:57:15.766069 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:15.765974    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c057de9f-5a6e-4dec-ac31-5464f4afd550-maas-api-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:57:16.499802 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:16.499720    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw"
Jul 27 19:57:16.499802 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:16.499756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw" event={"ID":"c057de9f-5a6e-4dec-ac31-5464f4afd550","Type":"ContainerDied","Data":"6db3699cf9c6ca8e4861bb52868ef20a92b5b79c26f6ce0170a6c75154487f88"}
Jul 27 19:57:16.500060 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:16.499804    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6db3699cf9c6ca8e4861bb52868ef20a92b5b79c26f6ce0170a6c75154487f88"
Jul 27 19:57:32.864346 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:32.864307    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw"]
Jul 27 19:57:32.869143 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:32.869118    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2eb089-fm6xw"]
Jul 27 19:57:34.085706 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:34.085677    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c057de9f-5a6e-4dec-ac31-5464f4afd550" path="/var/lib/kubelet/pods/c057de9f-5a6e-4dec-ac31-5464f4afd550/volumes"
Jul 27 19:57:39.975021 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:39.974987    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr"]
Jul 27 19:57:39.975446 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:39.975336    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c057de9f-5a6e-4dec-ac31-5464f4afd550" containerName="revoke-keys"
Jul 27 19:57:39.975446 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:39.975348    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c057de9f-5a6e-4dec-ac31-5464f4afd550" containerName="revoke-keys"
Jul 27 19:57:39.975446 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:39.975415    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c057de9f-5a6e-4dec-ac31-5464f4afd550" containerName="revoke-keys"
Jul 27 19:57:39.978444 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:39.978428    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr"
Jul 27 19:57:39.980927 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:39.980903    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\""
Jul 27 19:57:39.980927 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:39.980920    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 19:57:39.986021 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:39.985998    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr"]
Jul 27 19:57:40.060971 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:40.060936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b9681e-5e99-4a7b-a2c9-84a007b94ad5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-941fec-tfpnr\" (UID: \"d8b9681e-5e99-4a7b-a2c9-84a007b94ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr"
Jul 27 19:57:40.161566 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:40.161522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b9681e-5e99-4a7b-a2c9-84a007b94ad5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-941fec-tfpnr\" (UID: \"d8b9681e-5e99-4a7b-a2c9-84a007b94ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr"
Jul 27 19:57:40.162082 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:40.162062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b9681e-5e99-4a7b-a2c9-84a007b94ad5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-941fec-tfpnr\" (UID: \"d8b9681e-5e99-4a7b-a2c9-84a007b94ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr"
Jul 27 19:57:40.289338 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:40.289308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr"
Jul 27 19:57:40.426172 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:40.426148    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr"]
Jul 27 19:57:40.428195 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:57:40.428155    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b9681e_5e99_4a7b_a2c9_84a007b94ad5.slice/crio-ac53535a6b59785c356396f26d04f5a5ef09bf76441c5408c64c950e2a513509 WatchSource:0}: Error finding container ac53535a6b59785c356396f26d04f5a5ef09bf76441c5408c64c950e2a513509: Status 404 returned error can't find the container with id ac53535a6b59785c356396f26d04f5a5ef09bf76441c5408c64c950e2a513509
Jul 27 19:57:40.620629 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:40.620534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr" event={"ID":"d8b9681e-5e99-4a7b-a2c9-84a007b94ad5","Type":"ContainerStarted","Data":"ac53535a6b59785c356396f26d04f5a5ef09bf76441c5408c64c950e2a513509"}
Jul 27 19:57:41.627487 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:41.627450    2580 generic.go:356] "Generic (PLEG): container finished" podID="d8b9681e-5e99-4a7b-a2c9-84a007b94ad5" containerID="571a91c39d2af1d4654bfe0ac9fa9e11f29d09e313e160d3f5c84591ffcc87ee" exitCode=0
Jul 27 19:57:41.627487 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:41.627490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr" event={"ID":"d8b9681e-5e99-4a7b-a2c9-84a007b94ad5","Type":"ContainerDied","Data":"571a91c39d2af1d4654bfe0ac9fa9e11f29d09e313e160d3f5c84591ffcc87ee"}
Jul 27 19:57:42.760709 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:42.760682    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr"
Jul 27 19:57:42.883967 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:42.883932    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b9681e-5e99-4a7b-a2c9-84a007b94ad5-maas-api-service-ca\") pod \"d8b9681e-5e99-4a7b-a2c9-84a007b94ad5\" (UID: \"d8b9681e-5e99-4a7b-a2c9-84a007b94ad5\") "
Jul 27 19:57:42.884332 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:42.884312    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b9681e-5e99-4a7b-a2c9-84a007b94ad5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8b9681e-5e99-4a7b-a2c9-84a007b94ad5" (UID: "d8b9681e-5e99-4a7b-a2c9-84a007b94ad5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:57:42.984545 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:42.984466    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b9681e-5e99-4a7b-a2c9-84a007b94ad5-maas-api-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 19:57:43.641348 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:43.641313    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr"
Jul 27 19:57:43.641348 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:43.641335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr" event={"ID":"d8b9681e-5e99-4a7b-a2c9-84a007b94ad5","Type":"ContainerDied","Data":"ac53535a6b59785c356396f26d04f5a5ef09bf76441c5408c64c950e2a513509"}
Jul 27 19:57:43.641654 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:57:43.641381    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac53535a6b59785c356396f26d04f5a5ef09bf76441c5408c64c950e2a513509"
Jul 27 19:58:00.070623 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:00.070516    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr"]
Jul 27 19:58:00.072772 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:00.072748    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-941fec-tfpnr"]
Jul 27 19:58:00.090098 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:00.090063    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8b9681e-5e99-4a7b-a2c9-84a007b94ad5" path="/var/lib/kubelet/pods/d8b9681e-5e99-4a7b-a2c9-84a007b94ad5/volumes"
Jul 27 19:58:33.233091 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.233058    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"]
Jul 27 19:58:33.233540 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.233405    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b9681e-5e99-4a7b-a2c9-84a007b94ad5" containerName="revoke-keys"
Jul 27 19:58:33.233540 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.233417    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b9681e-5e99-4a7b-a2c9-84a007b94ad5" containerName="revoke-keys"
Jul 27 19:58:33.233540 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.233475    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b9681e-5e99-4a7b-a2c9-84a007b94ad5" containerName="revoke-keys"
Jul 27 19:58:33.236407 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.236389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.242219 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.242199    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4289a9\"/\"kube-root-ca.crt\""
Jul 27 19:58:33.243124 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.243103    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4289a9\"/\"openshift-service-ca.crt\""
Jul 27 19:58:33.243251 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.243103    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4289a9\"/\"default-dockercfg-hbgqq\""
Jul 27 19:58:33.243251 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.243139    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4289a9\"/\"auth-test-model-4289a9-kserve-self-signed-certs\""
Jul 27 19:58:33.259478 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.259448    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"]
Jul 27 19:58:33.330599 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.330564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf56f\" (UniqueName: \"kubernetes.io/projected/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-kube-api-access-pf56f\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.330785 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.330612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-tmp-dir\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.330785 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.330699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-home\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.330785 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.330726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-tls-certs\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.330785 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.330764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-dshm\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.330991 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.330788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-model-cache\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.431578 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.431545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf56f\" (UniqueName: \"kubernetes.io/projected/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-kube-api-access-pf56f\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.431578 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.431586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-tmp-dir\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.431856 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.431633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-home\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.431856 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.431652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-tls-certs\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.431856 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.431677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-dshm\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.431856 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.431722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-model-cache\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.432271 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.432213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-home\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.432271 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.432234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-model-cache\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.432450 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.432298    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-tmp-dir\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.434133 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.434111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-dshm\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.434417 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.434396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-tls-certs\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.440500 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.440475    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf56f\" (UniqueName: \"kubernetes.io/projected/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-kube-api-access-pf56f\") pod \"auth-test-model-4289a9-kserve-7dc595cdb7-sgvww\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") " pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.545276 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.545229    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:33.677501 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.677474    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"]
Jul 27 19:58:33.679447 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:58:33.679419    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0eb3bb30_ad98_43ba_99a3_77eb2ef1af22.slice/crio-0da077be9c5e6c4fbefa4dc7233a90fb6e1ead046d7e469288ff235fc70686ca WatchSource:0}: Error finding container 0da077be9c5e6c4fbefa4dc7233a90fb6e1ead046d7e469288ff235fc70686ca: Status 404 returned error can't find the container with id 0da077be9c5e6c4fbefa4dc7233a90fb6e1ead046d7e469288ff235fc70686ca
Jul 27 19:58:33.888216 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.888115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww" event={"ID":"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22","Type":"ContainerStarted","Data":"e4b1312095a2a6dc4c9801473fc1e0c4c78694474fe29db442cff8ae1d20096b"}
Jul 27 19:58:33.888216 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.888167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww" event={"ID":"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22","Type":"ContainerStarted","Data":"0da077be9c5e6c4fbefa4dc7233a90fb6e1ead046d7e469288ff235fc70686ca"}
Jul 27 19:58:33.910050 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:33.909994    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww" podStartSLOduration=0.909975235 podStartE2EDuration="909.975235ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:58:33.908375874 +0000 UTC m=+2456.424080379" watchObservedRunningTime="2026-07-27 19:58:33.909975235 +0000 UTC m=+2456.425679743"
Jul 27 19:58:43.546428 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:43.546385    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:43.546914 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:43.546446    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:43.558828 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:43.558802    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:58:43.952579 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:58:43.952492    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 19:59:08.273387 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.273345    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"]
Jul 27 19:59:08.276689 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.276671    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.279991 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.279967    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f99bfd\"/\"auth-test-model-f99bfd-kserve-self-signed-certs\""
Jul 27 19:59:08.280102 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.279993    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f99bfd\"/\"default-dockercfg-n967z\""
Jul 27 19:59:08.280102 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.280056    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f99bfd\"/\"openshift-service-ca.crt\""
Jul 27 19:59:08.280102 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.279998    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f99bfd\"/\"kube-root-ca.crt\""
Jul 27 19:59:08.292166 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.292138    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"]
Jul 27 19:59:08.339555 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.339525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-tmp-dir\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.339555 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.339556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-model-cache\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.339719 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.339575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-tls-certs\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.339719 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.339605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62slv\" (UniqueName: \"kubernetes.io/projected/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-kube-api-access-62slv\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.339719 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.339632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-home\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.339719 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.339653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-dshm\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.441073 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.441034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-tmp-dir\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.441073 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.441072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-model-cache\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.441324 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.441098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-tls-certs\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.441324 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.441285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62slv\" (UniqueName: \"kubernetes.io/projected/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-kube-api-access-62slv\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.441404 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.441331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-home\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.441404 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.441372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-dshm\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.441525 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.441505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-tmp-dir\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.441608 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.441529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-model-cache\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.441693 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.441676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-home\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.443551 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.443525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-dshm\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.443682 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.443667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-tls-certs\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.450046 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.450020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62slv\" (UniqueName: \"kubernetes.io/projected/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-kube-api-access-62slv\") pod \"auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") " pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.585485 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.585450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:08.725950 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:08.725922    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"]
Jul 27 19:59:08.727642 ip-10-0-129-103 kubenswrapper[2580]: W0727 19:59:08.727612    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f7bc6ac_d0ca_4e07_aa2a_f8e11ebf3231.slice/crio-1525b37774de4e7bc5f4784d55ed2c6243e51e050c31d695711793b8be9bed84 WatchSource:0}: Error finding container 1525b37774de4e7bc5f4784d55ed2c6243e51e050c31d695711793b8be9bed84: Status 404 returned error can't find the container with id 1525b37774de4e7bc5f4784d55ed2c6243e51e050c31d695711793b8be9bed84
Jul 27 19:59:09.067065 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:09.067025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt" event={"ID":"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231","Type":"ContainerStarted","Data":"66eac1a754aa50f423ef80c068a3c109016b2bc44a1b6c30276301b31d70f67b"}
Jul 27 19:59:09.067065 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:09.067070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt" event={"ID":"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231","Type":"ContainerStarted","Data":"1525b37774de4e7bc5f4784d55ed2c6243e51e050c31d695711793b8be9bed84"}
Jul 27 19:59:09.085600 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:09.085539    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt" podStartSLOduration=1.085518173 podStartE2EDuration="1.085518173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:59:09.085131926 +0000 UTC m=+2491.600836442" watchObservedRunningTime="2026-07-27 19:59:09.085518173 +0000 UTC m=+2491.601222678"
Jul 27 19:59:18.586660 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:18.586605    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:18.587228 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:18.586679    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:18.599794 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:18.599769    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 19:59:19.131508 ip-10-0-129-103 kubenswrapper[2580]: I0727 19:59:19.131474    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 20:00:00.153774 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:00.153735    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f99bfd-29753040-vbzzx"]
Jul 27 20:00:00.156488 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:00.156469    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f99bfd-29753040-vbzzx"
Jul 27 20:00:00.160340 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:00.160325    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\""
Jul 27 20:00:00.166674 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:00.166654    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f99bfd-29753040-vbzzx"
Jul 27 20:00:00.170502 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:00.170479    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f99bfd-29753040-vbzzx"]
Jul 27 20:00:00.509541 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:00.509462    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f99bfd-29753040-vbzzx"]
Jul 27 20:00:00.511916 ip-10-0-129-103 kubenswrapper[2580]: W0727 20:00:00.511889    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b905e6c_834d_46db_8033_a5637f99728a.slice/crio-9d73aef59e35fd508107afc9cf4e865e300f36323b70be879f709afd01c469d2 WatchSource:0}: Error finding container 9d73aef59e35fd508107afc9cf4e865e300f36323b70be879f709afd01c469d2: Status 404 returned error can't find the container with id 9d73aef59e35fd508107afc9cf4e865e300f36323b70be879f709afd01c469d2
Jul 27 20:00:01.328436 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:01.328399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f99bfd-29753040-vbzzx" event={"ID":"8b905e6c-834d-46db-8033-a5637f99728a","Type":"ContainerStarted","Data":"70a0c83184d0898bebc3e3e5b46c7a00ac219ee39687750bc55831496b1526de"}
Jul 27 20:00:01.328842 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:01.328463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f99bfd-29753040-vbzzx" event={"ID":"8b905e6c-834d-46db-8033-a5637f99728a","Type":"ContainerStarted","Data":"9d73aef59e35fd508107afc9cf4e865e300f36323b70be879f709afd01c469d2"}
Jul 27 20:00:01.374063 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:01.374020    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f99bfd-29753040-vbzzx" podStartSLOduration=1.374003143 podStartE2EDuration="1.374003143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:00:01.372236887 +0000 UTC m=+2543.887941391" watchObservedRunningTime="2026-07-27 20:00:01.374003143 +0000 UTC m=+2543.889707662"
Jul 27 20:00:02.335113 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:02.335074    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b905e6c-834d-46db-8033-a5637f99728a" containerID="70a0c83184d0898bebc3e3e5b46c7a00ac219ee39687750bc55831496b1526de" exitCode=0
Jul 27 20:00:02.335532 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:02.335165    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f99bfd-29753040-vbzzx" event={"ID":"8b905e6c-834d-46db-8033-a5637f99728a","Type":"ContainerDied","Data":"70a0c83184d0898bebc3e3e5b46c7a00ac219ee39687750bc55831496b1526de"}
Jul 27 20:00:03.469629 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:03.469608    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f99bfd-29753040-vbzzx"
Jul 27 20:00:04.347442 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:04.347357    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f99bfd-29753040-vbzzx"
Jul 27 20:00:04.347442 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:04.347369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f99bfd-29753040-vbzzx" event={"ID":"8b905e6c-834d-46db-8033-a5637f99728a","Type":"ContainerDied","Data":"9d73aef59e35fd508107afc9cf4e865e300f36323b70be879f709afd01c469d2"}
Jul 27 20:00:04.347442 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:00:04.347419    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d73aef59e35fd508107afc9cf4e865e300f36323b70be879f709afd01c469d2"
Jul 27 20:02:38.313416 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:02:38.313384    2580 scope.go:117] "RemoveContainer" containerID="b3b64140ecad56aadf2697916a1355f9dd7c1f8e465308f200fadcb70d4d7b06"
Jul 27 20:03:38.332998 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:03:38.332965    2580 scope.go:117] "RemoveContainer" containerID="2023586a2b7d2eabc4a71720c2839985b8a7618f72a297267a6aa5b251dcc2f2"
Jul 27 20:03:58.250830 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:03:58.250793    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"]
Jul 27 20:03:59.181841 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:03:59.181751    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"]
Jul 27 20:04:13.269624 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.269562    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww" podUID="0eb3bb30-ad98-43ba-99a3-77eb2ef1af22" containerName="main" containerID="cri-o://e4b1312095a2a6dc4c9801473fc1e0c4c78694474fe29db442cff8ae1d20096b" gracePeriod=45
Jul 27 20:04:13.510410 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.510384    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 20:04:13.547683 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.547655    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-model-cache\") pod \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") "
Jul 27 20:04:13.547866 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.547719    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-home\") pod \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") "
Jul 27 20:04:13.547866 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.547757    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-tls-certs\") pod \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") "
Jul 27 20:04:13.547866 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.547791    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf56f\" (UniqueName: \"kubernetes.io/projected/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-kube-api-access-pf56f\") pod \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") "
Jul 27 20:04:13.547866 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.547831    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-dshm\") pod \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") "
Jul 27 20:04:13.548082 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.547872    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-tmp-dir\") pod \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\" (UID: \"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22\") "
Jul 27 20:04:13.548082 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.547975    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-model-cache" (OuterVolumeSpecName: "model-cache") pod "0eb3bb30-ad98-43ba-99a3-77eb2ef1af22" (UID: "0eb3bb30-ad98-43ba-99a3-77eb2ef1af22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:04:13.548082 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.548014    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-home" (OuterVolumeSpecName: "home") pod "0eb3bb30-ad98-43ba-99a3-77eb2ef1af22" (UID: "0eb3bb30-ad98-43ba-99a3-77eb2ef1af22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:04:13.548688 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.548144    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:04:13.548688 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.548162    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:04:13.548688 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.548195    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0eb3bb30-ad98-43ba-99a3-77eb2ef1af22" (UID: "0eb3bb30-ad98-43ba-99a3-77eb2ef1af22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:04:13.550029 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.550003    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-dshm" (OuterVolumeSpecName: "dshm") pod "0eb3bb30-ad98-43ba-99a3-77eb2ef1af22" (UID: "0eb3bb30-ad98-43ba-99a3-77eb2ef1af22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:04:13.550119 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.550096    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-kube-api-access-pf56f" (OuterVolumeSpecName: "kube-api-access-pf56f") pod "0eb3bb30-ad98-43ba-99a3-77eb2ef1af22" (UID: "0eb3bb30-ad98-43ba-99a3-77eb2ef1af22"). InnerVolumeSpecName "kube-api-access-pf56f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:04:13.550157 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.550114    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0eb3bb30-ad98-43ba-99a3-77eb2ef1af22" (UID: "0eb3bb30-ad98-43ba-99a3-77eb2ef1af22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:04:13.615437 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.615404    2580 generic.go:356] "Generic (PLEG): container finished" podID="0eb3bb30-ad98-43ba-99a3-77eb2ef1af22" containerID="e4b1312095a2a6dc4c9801473fc1e0c4c78694474fe29db442cff8ae1d20096b" exitCode=0
Jul 27 20:04:13.615597 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.615454    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"
Jul 27 20:04:13.615597 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.615484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww" event={"ID":"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22","Type":"ContainerDied","Data":"e4b1312095a2a6dc4c9801473fc1e0c4c78694474fe29db442cff8ae1d20096b"}
Jul 27 20:04:13.615597 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.615516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww" event={"ID":"0eb3bb30-ad98-43ba-99a3-77eb2ef1af22","Type":"ContainerDied","Data":"0da077be9c5e6c4fbefa4dc7233a90fb6e1ead046d7e469288ff235fc70686ca"}
Jul 27 20:04:13.615597 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.615534    2580 scope.go:117] "RemoveContainer" containerID="e4b1312095a2a6dc4c9801473fc1e0c4c78694474fe29db442cff8ae1d20096b"
Jul 27 20:04:13.624685 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.624664    2580 scope.go:117] "RemoveContainer" containerID="e4b1312095a2a6dc4c9801473fc1e0c4c78694474fe29db442cff8ae1d20096b"
Jul 27 20:04:13.624943 ip-10-0-129-103 kubenswrapper[2580]: E0727 20:04:13.624920    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b1312095a2a6dc4c9801473fc1e0c4c78694474fe29db442cff8ae1d20096b\": container with ID starting with e4b1312095a2a6dc4c9801473fc1e0c4c78694474fe29db442cff8ae1d20096b not found: ID does not exist" containerID="e4b1312095a2a6dc4c9801473fc1e0c4c78694474fe29db442cff8ae1d20096b"
Jul 27 20:04:13.625005 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.624955    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b1312095a2a6dc4c9801473fc1e0c4c78694474fe29db442cff8ae1d20096b"} err="failed to get container status \"e4b1312095a2a6dc4c9801473fc1e0c4c78694474fe29db442cff8ae1d20096b\": rpc error: code = NotFound desc = could not find container \"e4b1312095a2a6dc4c9801473fc1e0c4c78694474fe29db442cff8ae1d20096b\": container with ID starting with e4b1312095a2a6dc4c9801473fc1e0c4c78694474fe29db442cff8ae1d20096b not found: ID does not exist"
Jul 27 20:04:13.636230 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.636203    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"]
Jul 27 20:04:13.639590 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.639570    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/auth-test-model-4289a9-kserve-7dc595cdb7-sgvww"]
Jul 27 20:04:13.648864 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.648847    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:04:13.648919 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.648884    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:04:13.648919 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.648894    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:04:13.648919 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:13.648902    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf56f\" (UniqueName: \"kubernetes.io/projected/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22-kube-api-access-pf56f\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:04:14.085929 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.085889    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eb3bb30-ad98-43ba-99a3-77eb2ef1af22" path="/var/lib/kubelet/pods/0eb3bb30-ad98-43ba-99a3-77eb2ef1af22/volumes"
Jul 27 20:04:14.200193 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.200145    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt" podUID="3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231" containerName="main" containerID="cri-o://66eac1a754aa50f423ef80c068a3c109016b2bc44a1b6c30276301b31d70f67b" gracePeriod=45
Jul 27 20:04:14.451112 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.451087    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 20:04:14.557408 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.557379    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-tmp-dir\") pod \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") "
Jul 27 20:04:14.557584 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.557419    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62slv\" (UniqueName: \"kubernetes.io/projected/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-kube-api-access-62slv\") pod \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") "
Jul 27 20:04:14.557584 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.557438    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-model-cache\") pod \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") "
Jul 27 20:04:14.557584 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.557475    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-dshm\") pod \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") "
Jul 27 20:04:14.557584 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.557494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-home\") pod \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") "
Jul 27 20:04:14.557584 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.557529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-tls-certs\") pod \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\" (UID: \"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231\") "
Jul 27 20:04:14.557855 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.557695    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231" (UID: "3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:04:14.557855 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.557798    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231" (UID: "3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:04:14.557855 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.557809    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-home" (OuterVolumeSpecName: "home") pod "3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231" (UID: "3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:04:14.557997 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.557877    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:04:14.559719 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.559698    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-dshm" (OuterVolumeSpecName: "dshm") pod "3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231" (UID: "3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:04:14.559801 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.559721    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-kube-api-access-62slv" (OuterVolumeSpecName: "kube-api-access-62slv") pod "3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231" (UID: "3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231"). InnerVolumeSpecName "kube-api-access-62slv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:04:14.559839 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.559804    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231" (UID: "3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:04:14.628994 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.628919    2580 generic.go:356] "Generic (PLEG): container finished" podID="3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231" containerID="66eac1a754aa50f423ef80c068a3c109016b2bc44a1b6c30276301b31d70f67b" exitCode=0
Jul 27 20:04:14.628994 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.628967    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"
Jul 27 20:04:14.628994 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.628981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt" event={"ID":"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231","Type":"ContainerDied","Data":"66eac1a754aa50f423ef80c068a3c109016b2bc44a1b6c30276301b31d70f67b"}
Jul 27 20:04:14.629187 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.629017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt" event={"ID":"3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231","Type":"ContainerDied","Data":"1525b37774de4e7bc5f4784d55ed2c6243e51e050c31d695711793b8be9bed84"}
Jul 27 20:04:14.629187 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.629035    2580 scope.go:117] "RemoveContainer" containerID="66eac1a754aa50f423ef80c068a3c109016b2bc44a1b6c30276301b31d70f67b"
Jul 27 20:04:14.638501 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.638487    2580 scope.go:117] "RemoveContainer" containerID="66eac1a754aa50f423ef80c068a3c109016b2bc44a1b6c30276301b31d70f67b"
Jul 27 20:04:14.638727 ip-10-0-129-103 kubenswrapper[2580]: E0727 20:04:14.638708    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66eac1a754aa50f423ef80c068a3c109016b2bc44a1b6c30276301b31d70f67b\": container with ID starting with 66eac1a754aa50f423ef80c068a3c109016b2bc44a1b6c30276301b31d70f67b not found: ID does not exist" containerID="66eac1a754aa50f423ef80c068a3c109016b2bc44a1b6c30276301b31d70f67b"
Jul 27 20:04:14.638774 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.638734    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66eac1a754aa50f423ef80c068a3c109016b2bc44a1b6c30276301b31d70f67b"} err="failed to get container status \"66eac1a754aa50f423ef80c068a3c109016b2bc44a1b6c30276301b31d70f67b\": rpc error: code = NotFound desc = could not find container \"66eac1a754aa50f423ef80c068a3c109016b2bc44a1b6c30276301b31d70f67b\": container with ID starting with 66eac1a754aa50f423ef80c068a3c109016b2bc44a1b6c30276301b31d70f67b not found: ID does not exist"
Jul 27 20:04:14.650108 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.650081    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"]
Jul 27 20:04:14.653091 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.653071    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/auth-test-model-f99bfd-kserve-6bd5b9577f-dplqt"]
Jul 27 20:04:14.658326 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.658309    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:04:14.658389 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.658329    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62slv\" (UniqueName: \"kubernetes.io/projected/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-kube-api-access-62slv\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:04:14.658389 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.658340    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:04:14.658389 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.658349    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:04:14.658389 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:14.658356    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:04:16.086626 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:16.086598    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231" path="/var/lib/kubelet/pods/3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231/volumes"
Jul 27 20:04:24.474952 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.474924    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"]
Jul 27 20:04:24.475321 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.475216    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eb3bb30-ad98-43ba-99a3-77eb2ef1af22" containerName="main"
Jul 27 20:04:24.475321 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.475227    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eb3bb30-ad98-43ba-99a3-77eb2ef1af22" containerName="main"
Jul 27 20:04:24.475321 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.475257    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231" containerName="main"
Jul 27 20:04:24.475321 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.475263    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231" containerName="main"
Jul 27 20:04:24.475321 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.475274    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b905e6c-834d-46db-8033-a5637f99728a" containerName="cleanup"
Jul 27 20:04:24.475321 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.475279    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b905e6c-834d-46db-8033-a5637f99728a" containerName="cleanup"
Jul 27 20:04:24.475515 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.475333    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b905e6c-834d-46db-8033-a5637f99728a" containerName="cleanup"
Jul 27 20:04:24.475515 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.475341    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eb3bb30-ad98-43ba-99a3-77eb2ef1af22" containerName="main"
Jul 27 20:04:24.475515 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.475347    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f7bc6ac-d0ca-4e07-aa2a-f8e11ebf3231" containerName="main"
Jul 27 20:04:24.478294 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.478278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.481496 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.481473    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4289a9\"/\"kube-root-ca.crt\""
Jul 27 20:04:24.481642 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.481516    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4289a9\"/\"openshift-service-ca.crt\""
Jul 27 20:04:24.481642 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.481527    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4289a9\"/\"sub-test-model-4289a9-kserve-self-signed-certs\""
Jul 27 20:04:24.481642 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.481516    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4289a9\"/\"default-dockercfg-hbgqq\""
Jul 27 20:04:24.489879 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.489857    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"]
Jul 27 20:04:24.534895 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.534871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5287d0cf-6677-4406-a0d8-418911fb7aad-tls-certs\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.535017 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.534910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzkhc\" (UniqueName: \"kubernetes.io/projected/5287d0cf-6677-4406-a0d8-418911fb7aad-kube-api-access-dzkhc\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.535017 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.534939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-home\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.535141 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.535049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-model-cache\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.535141 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.535118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-tmp-dir\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.535229 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.535167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-dshm\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.636200 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.636170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-dshm\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.636359 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.636212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5287d0cf-6677-4406-a0d8-418911fb7aad-tls-certs\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.636359 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.636270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkhc\" (UniqueName: \"kubernetes.io/projected/5287d0cf-6677-4406-a0d8-418911fb7aad-kube-api-access-dzkhc\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.636359 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.636300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-home\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.636537 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.636364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-model-cache\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.636537 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.636422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-tmp-dir\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.636738 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.636717    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-home\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.636810 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.636752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-tmp-dir\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.636810 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.636785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-model-cache\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.638686 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.638667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-dshm\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.638850 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.638831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5287d0cf-6677-4406-a0d8-418911fb7aad-tls-certs\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.645898 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.645876    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkhc\" (UniqueName: \"kubernetes.io/projected/5287d0cf-6677-4406-a0d8-418911fb7aad-kube-api-access-dzkhc\") pod \"sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") " pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.787408 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.787386    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:24.941862 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:24.941837    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"]
Jul 27 20:04:24.943004 ip-10-0-129-103 kubenswrapper[2580]: W0727 20:04:24.942981    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5287d0cf_6677_4406_a0d8_418911fb7aad.slice/crio-061722e03f501d75e953738efa81f639cf381e0f25d6b0935b7ec765e0f06dc7 WatchSource:0}: Error finding container 061722e03f501d75e953738efa81f639cf381e0f25d6b0935b7ec765e0f06dc7: Status 404 returned error can't find the container with id 061722e03f501d75e953738efa81f639cf381e0f25d6b0935b7ec765e0f06dc7
Jul 27 20:04:25.685444 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:25.685407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k" event={"ID":"5287d0cf-6677-4406-a0d8-418911fb7aad","Type":"ContainerStarted","Data":"7abb4350521939dfd4901c2f1386b85d86f89e3618d47f8b363ea683b2c09e0f"}
Jul 27 20:04:25.685807 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:25.685454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k" event={"ID":"5287d0cf-6677-4406-a0d8-418911fb7aad","Type":"ContainerStarted","Data":"061722e03f501d75e953738efa81f639cf381e0f25d6b0935b7ec765e0f06dc7"}
Jul 27 20:04:25.705746 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:25.705696    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k" podStartSLOduration=1.7056802370000002 podStartE2EDuration="1.705680237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:04:25.704053029 +0000 UTC m=+2808.219757532" watchObservedRunningTime="2026-07-27 20:04:25.705680237 +0000 UTC m=+2808.221384744"
Jul 27 20:04:34.787990 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:34.787955    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:34.788460 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:34.788004    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:34.800945 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:34.800922    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:35.746607 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:35.746575    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:04:38.351900 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:38.351871    2580 scope.go:117] "RemoveContainer" containerID="571a91c39d2af1d4654bfe0ac9fa9e11f29d09e313e160d3f5c84591ffcc87ee"
Jul 27 20:04:42.182329 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.182290    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"]
Jul 27 20:04:42.187162 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.187142    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.189613 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.189578    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f99bfd\"/\"openshift-service-ca.crt\""
Jul 27 20:04:42.190433 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.190417    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f99bfd\"/\"default-dockercfg-n967z\""
Jul 27 20:04:42.190537 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.190512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f99bfd\"/\"kube-root-ca.crt\""
Jul 27 20:04:42.190593 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.190544    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f99bfd\"/\"sub-test-model-f99bfd-kserve-self-signed-certs\""
Jul 27 20:04:42.199260 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.199222    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"]
Jul 27 20:04:42.286358 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.286326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-model-cache\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.286499 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.286367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf6px\" (UniqueName: \"kubernetes.io/projected/4191b9b2-d526-4f25-b3f5-900827312739-kube-api-access-pf6px\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.286499 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.286398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-dshm\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.286499 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.286456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4191b9b2-d526-4f25-b3f5-900827312739-tls-certs\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.286499 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.286478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-tmp-dir\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.286635 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.286523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-home\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.386940 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.386907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-home\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.387054 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.386957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-model-cache\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.387054 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.386983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf6px\" (UniqueName: \"kubernetes.io/projected/4191b9b2-d526-4f25-b3f5-900827312739-kube-api-access-pf6px\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.387054 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.387010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-dshm\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.387054 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.387035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4191b9b2-d526-4f25-b3f5-900827312739-tls-certs\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.387296 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.387057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-tmp-dir\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.387362 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.387316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-home\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.387440 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.387418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-tmp-dir\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.387500 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.387448    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-model-cache\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.389256 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.389226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-dshm\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.389397 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.389379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4191b9b2-d526-4f25-b3f5-900827312739-tls-certs\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.396093 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.396072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf6px\" (UniqueName: \"kubernetes.io/projected/4191b9b2-d526-4f25-b3f5-900827312739-kube-api-access-pf6px\") pod \"sub-test-model-f99bfd-kserve-74d59bf979-v9xls\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") " pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.496331 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.496261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:42.631728 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.631700    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"]
Jul 27 20:04:42.633495 ip-10-0-129-103 kubenswrapper[2580]: W0727 20:04:42.633459    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4191b9b2_d526_4f25_b3f5_900827312739.slice/crio-2403ad5306b230dfccee1f65f00143334ab9841d22af6461b7a7aa27648f29ac WatchSource:0}: Error finding container 2403ad5306b230dfccee1f65f00143334ab9841d22af6461b7a7aa27648f29ac: Status 404 returned error can't find the container with id 2403ad5306b230dfccee1f65f00143334ab9841d22af6461b7a7aa27648f29ac
Jul 27 20:04:42.772578 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.772488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls" event={"ID":"4191b9b2-d526-4f25-b3f5-900827312739","Type":"ContainerStarted","Data":"2a85eea9fd245d704ceb36462a00425937d236d4c4cf3f62fde79c5d882ebc78"}
Jul 27 20:04:42.772578 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.772537    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls" event={"ID":"4191b9b2-d526-4f25-b3f5-900827312739","Type":"ContainerStarted","Data":"2403ad5306b230dfccee1f65f00143334ab9841d22af6461b7a7aa27648f29ac"}
Jul 27 20:04:42.793114 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:42.793068    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls" podStartSLOduration=0.793051772 podStartE2EDuration="793.051772ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:04:42.791528344 +0000 UTC m=+2825.307232862" watchObservedRunningTime="2026-07-27 20:04:42.793051772 +0000 UTC m=+2825.308756276"
Jul 27 20:04:52.496730 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:52.496686    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:52.496730 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:52.496736    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:52.509032 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:52.509009    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:04:52.830599 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:04:52.830569    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:05:43.904021 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:43.903985    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"]
Jul 27 20:05:44.916081 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:44.916048    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"]
Jul 27 20:05:54.706031 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.705901    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"]
Jul 27 20:05:54.709062 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.709045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.712101 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.712079    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4289a9\"/\"rate-test-model-4289a9-kserve-self-signed-certs\""
Jul 27 20:05:54.723665 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.723643    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"]
Jul 27 20:05:54.898256 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.898219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-tls-certs\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.898439 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.898274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-tmp-dir\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.898439 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.898343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-dshm\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.898439 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.898369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmq82\" (UniqueName: \"kubernetes.io/projected/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-kube-api-access-rmq82\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.898562 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.898439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-model-cache\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.898562 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.898470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-home\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.999297 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.999196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-dshm\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.999297 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.999234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmq82\" (UniqueName: \"kubernetes.io/projected/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-kube-api-access-rmq82\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.999519 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.999303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-model-cache\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.999519 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.999325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-home\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.999519 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.999347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-tls-certs\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.999519 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.999364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-tmp-dir\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.999762 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.999738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-model-cache\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.999828 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.999752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-tmp-dir\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:54.999828 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:54.999797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-home\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:55.001429 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:55.001412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-dshm\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:55.001907 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:55.001886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-tls-certs\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:55.013288 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:55.013264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmq82\" (UniqueName: \"kubernetes.io/projected/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-kube-api-access-rmq82\") pod \"rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") " pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:55.017944 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:55.017924    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:05:55.157619 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:55.157594    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"]
Jul 27 20:05:55.158780 ip-10-0-129-103 kubenswrapper[2580]: W0727 20:05:55.158748    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dd3e429_bd0e_48a6_8c1c_632d7fe3180b.slice/crio-7c1e9157ddab4347a69e2fce4ad9ca48c9d724edf544c23bb4a30a55ee85c86d WatchSource:0}: Error finding container 7c1e9157ddab4347a69e2fce4ad9ca48c9d724edf544c23bb4a30a55ee85c86d: Status 404 returned error can't find the container with id 7c1e9157ddab4347a69e2fce4ad9ca48c9d724edf544c23bb4a30a55ee85c86d
Jul 27 20:05:56.137505 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:56.137470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6" event={"ID":"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b","Type":"ContainerStarted","Data":"b4202ffeddb86557dbe8d2bbb9b2f62e2fb9e2d55b1d8f809466a6497ef490b9"}
Jul 27 20:05:56.137505 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:56.137505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6" event={"ID":"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b","Type":"ContainerStarted","Data":"7c1e9157ddab4347a69e2fce4ad9ca48c9d724edf544c23bb4a30a55ee85c86d"}
Jul 27 20:05:56.162637 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:56.162582    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6" podStartSLOduration=2.162567227 podStartE2EDuration="2.162567227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:05:56.161158146 +0000 UTC m=+2898.676862651" watchObservedRunningTime="2026-07-27 20:05:56.162567227 +0000 UTC m=+2898.678271731"
Jul 27 20:05:58.922749 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:58.922710    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k" podUID="5287d0cf-6677-4406-a0d8-418911fb7aad" containerName="main" containerID="cri-o://7abb4350521939dfd4901c2f1386b85d86f89e3618d47f8b363ea683b2c09e0f" gracePeriod=45
Jul 27 20:05:59.153484 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.153455    2580 generic.go:356] "Generic (PLEG): container finished" podID="5287d0cf-6677-4406-a0d8-418911fb7aad" containerID="7abb4350521939dfd4901c2f1386b85d86f89e3618d47f8b363ea683b2c09e0f" exitCode=0
Jul 27 20:05:59.153646 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.153538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k" event={"ID":"5287d0cf-6677-4406-a0d8-418911fb7aad","Type":"ContainerDied","Data":"7abb4350521939dfd4901c2f1386b85d86f89e3618d47f8b363ea683b2c09e0f"}
Jul 27 20:05:59.178375 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.178320    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:05:59.334380 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.334346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5287d0cf-6677-4406-a0d8-418911fb7aad-tls-certs\") pod \"5287d0cf-6677-4406-a0d8-418911fb7aad\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") "
Jul 27 20:05:59.334380 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.334383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-home\") pod \"5287d0cf-6677-4406-a0d8-418911fb7aad\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") "
Jul 27 20:05:59.334599 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.334406    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-model-cache\") pod \"5287d0cf-6677-4406-a0d8-418911fb7aad\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") "
Jul 27 20:05:59.334599 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.334458    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-dshm\") pod \"5287d0cf-6677-4406-a0d8-418911fb7aad\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") "
Jul 27 20:05:59.334599 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.334493    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-tmp-dir\") pod \"5287d0cf-6677-4406-a0d8-418911fb7aad\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") "
Jul 27 20:05:59.334599 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.334559    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzkhc\" (UniqueName: \"kubernetes.io/projected/5287d0cf-6677-4406-a0d8-418911fb7aad-kube-api-access-dzkhc\") pod \"5287d0cf-6677-4406-a0d8-418911fb7aad\" (UID: \"5287d0cf-6677-4406-a0d8-418911fb7aad\") "
Jul 27 20:05:59.334817 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.334697    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-home" (OuterVolumeSpecName: "home") pod "5287d0cf-6677-4406-a0d8-418911fb7aad" (UID: "5287d0cf-6677-4406-a0d8-418911fb7aad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:05:59.334817 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.334728    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-model-cache" (OuterVolumeSpecName: "model-cache") pod "5287d0cf-6677-4406-a0d8-418911fb7aad" (UID: "5287d0cf-6677-4406-a0d8-418911fb7aad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:05:59.334903 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.334827    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:05:59.334903 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.334845    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:05:59.335015 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.334923    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5287d0cf-6677-4406-a0d8-418911fb7aad" (UID: "5287d0cf-6677-4406-a0d8-418911fb7aad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:05:59.336506 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.336477    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-dshm" (OuterVolumeSpecName: "dshm") pod "5287d0cf-6677-4406-a0d8-418911fb7aad" (UID: "5287d0cf-6677-4406-a0d8-418911fb7aad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:05:59.336615 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.336571    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5287d0cf-6677-4406-a0d8-418911fb7aad-kube-api-access-dzkhc" (OuterVolumeSpecName: "kube-api-access-dzkhc") pod "5287d0cf-6677-4406-a0d8-418911fb7aad" (UID: "5287d0cf-6677-4406-a0d8-418911fb7aad"). InnerVolumeSpecName "kube-api-access-dzkhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:05:59.336615 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.336579    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5287d0cf-6677-4406-a0d8-418911fb7aad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5287d0cf-6677-4406-a0d8-418911fb7aad" (UID: "5287d0cf-6677-4406-a0d8-418911fb7aad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:05:59.436119 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.436052    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:05:59.436119 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.436091    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5287d0cf-6677-4406-a0d8-418911fb7aad-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:05:59.436119 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.436102    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzkhc\" (UniqueName: \"kubernetes.io/projected/5287d0cf-6677-4406-a0d8-418911fb7aad-kube-api-access-dzkhc\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:05:59.436119 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.436111    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5287d0cf-6677-4406-a0d8-418911fb7aad-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:05:59.934655 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:05:59.934608    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls" podUID="4191b9b2-d526-4f25-b3f5-900827312739" containerName="main" containerID="cri-o://2a85eea9fd245d704ceb36462a00425937d236d4c4cf3f62fde79c5d882ebc78" gracePeriod=45
Jul 27 20:06:00.160908 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.160883    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"
Jul 27 20:06:00.160908 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.160890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k" event={"ID":"5287d0cf-6677-4406-a0d8-418911fb7aad","Type":"ContainerDied","Data":"061722e03f501d75e953738efa81f639cf381e0f25d6b0935b7ec765e0f06dc7"}
Jul 27 20:06:00.161161 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.160951    2580 scope.go:117] "RemoveContainer" containerID="7abb4350521939dfd4901c2f1386b85d86f89e3618d47f8b363ea683b2c09e0f"
Jul 27 20:06:00.162693 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.162673    2580 generic.go:356] "Generic (PLEG): container finished" podID="4191b9b2-d526-4f25-b3f5-900827312739" containerID="2a85eea9fd245d704ceb36462a00425937d236d4c4cf3f62fde79c5d882ebc78" exitCode=0
Jul 27 20:06:00.162823 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.162713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls" event={"ID":"4191b9b2-d526-4f25-b3f5-900827312739","Type":"ContainerDied","Data":"2a85eea9fd245d704ceb36462a00425937d236d4c4cf3f62fde79c5d882ebc78"}
Jul 27 20:06:00.179981 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.179950    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"]
Jul 27 20:06:00.181869 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.181849    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/sub-test-model-4289a9-kserve-84fd8dbf45-9hk2k"]
Jul 27 20:06:00.191939 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.191894    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:06:00.343554 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.343514    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-dshm\") pod \"4191b9b2-d526-4f25-b3f5-900827312739\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") "
Jul 27 20:06:00.343554 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.343566    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-tmp-dir\") pod \"4191b9b2-d526-4f25-b3f5-900827312739\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") "
Jul 27 20:06:00.343823 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.343609    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf6px\" (UniqueName: \"kubernetes.io/projected/4191b9b2-d526-4f25-b3f5-900827312739-kube-api-access-pf6px\") pod \"4191b9b2-d526-4f25-b3f5-900827312739\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") "
Jul 27 20:06:00.343823 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.343631    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-home\") pod \"4191b9b2-d526-4f25-b3f5-900827312739\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") "
Jul 27 20:06:00.343823 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.343664    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4191b9b2-d526-4f25-b3f5-900827312739-tls-certs\") pod \"4191b9b2-d526-4f25-b3f5-900827312739\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") "
Jul 27 20:06:00.343823 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.343696    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-model-cache\") pod \"4191b9b2-d526-4f25-b3f5-900827312739\" (UID: \"4191b9b2-d526-4f25-b3f5-900827312739\") "
Jul 27 20:06:00.344084 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.343863    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4191b9b2-d526-4f25-b3f5-900827312739" (UID: "4191b9b2-d526-4f25-b3f5-900827312739"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:06:00.344084 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.343913    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-home" (OuterVolumeSpecName: "home") pod "4191b9b2-d526-4f25-b3f5-900827312739" (UID: "4191b9b2-d526-4f25-b3f5-900827312739"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:06:00.344084 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.343980    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-model-cache" (OuterVolumeSpecName: "model-cache") pod "4191b9b2-d526-4f25-b3f5-900827312739" (UID: "4191b9b2-d526-4f25-b3f5-900827312739"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:06:00.345737 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.345709    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-dshm" (OuterVolumeSpecName: "dshm") pod "4191b9b2-d526-4f25-b3f5-900827312739" (UID: "4191b9b2-d526-4f25-b3f5-900827312739"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:06:00.345819 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.345782    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4191b9b2-d526-4f25-b3f5-900827312739-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4191b9b2-d526-4f25-b3f5-900827312739" (UID: "4191b9b2-d526-4f25-b3f5-900827312739"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:06:00.345819 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.345787    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4191b9b2-d526-4f25-b3f5-900827312739-kube-api-access-pf6px" (OuterVolumeSpecName: "kube-api-access-pf6px") pod "4191b9b2-d526-4f25-b3f5-900827312739" (UID: "4191b9b2-d526-4f25-b3f5-900827312739"). InnerVolumeSpecName "kube-api-access-pf6px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:06:00.444300 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.444190    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:06:00.444300 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.444215    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:06:00.444300 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.444225    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf6px\" (UniqueName: \"kubernetes.io/projected/4191b9b2-d526-4f25-b3f5-900827312739-kube-api-access-pf6px\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:06:00.444300 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.444234    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:06:00.444300 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.444274    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4191b9b2-d526-4f25-b3f5-900827312739-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:06:00.444300 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:00.444283    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4191b9b2-d526-4f25-b3f5-900827312739-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:06:01.169643 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:01.169614    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"
Jul 27 20:06:01.170092 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:01.169648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls" event={"ID":"4191b9b2-d526-4f25-b3f5-900827312739","Type":"ContainerDied","Data":"2403ad5306b230dfccee1f65f00143334ab9841d22af6461b7a7aa27648f29ac"}
Jul 27 20:06:01.170092 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:01.169697    2580 scope.go:117] "RemoveContainer" containerID="2a85eea9fd245d704ceb36462a00425937d236d4c4cf3f62fde79c5d882ebc78"
Jul 27 20:06:01.194423 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:01.194396    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"]
Jul 27 20:06:01.197206 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:01.197185    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/sub-test-model-f99bfd-kserve-74d59bf979-v9xls"]
Jul 27 20:06:02.085843 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:02.085812    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4191b9b2-d526-4f25-b3f5-900827312739" path="/var/lib/kubelet/pods/4191b9b2-d526-4f25-b3f5-900827312739/volumes"
Jul 27 20:06:02.086183 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:02.086170    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5287d0cf-6677-4406-a0d8-418911fb7aad" path="/var/lib/kubelet/pods/5287d0cf-6677-4406-a0d8-418911fb7aad/volumes"
Jul 27 20:06:05.018973 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:05.018939    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:06:05.019418 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:05.018990    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:06:05.031858 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:05.031836    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:06:05.199891 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:05.199865    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:06:27.591050 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.591016    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"]
Jul 27 20:06:27.591529 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.591336    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5287d0cf-6677-4406-a0d8-418911fb7aad" containerName="main"
Jul 27 20:06:27.591529 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.591347    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5287d0cf-6677-4406-a0d8-418911fb7aad" containerName="main"
Jul 27 20:06:27.591529 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.591366    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4191b9b2-d526-4f25-b3f5-900827312739" containerName="main"
Jul 27 20:06:27.591529 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.591372    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4191b9b2-d526-4f25-b3f5-900827312739" containerName="main"
Jul 27 20:06:27.591529 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.591420    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4191b9b2-d526-4f25-b3f5-900827312739" containerName="main"
Jul 27 20:06:27.591529 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.591429    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5287d0cf-6677-4406-a0d8-418911fb7aad" containerName="main"
Jul 27 20:06:27.594352 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.594337    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.597077 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.597054    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f99bfd\"/\"default-dockercfg-n967z\""
Jul 27 20:06:27.597757 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.597737    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f99bfd\"/\"rate-test-model-f99bfd-kserve-self-signed-certs\""
Jul 27 20:06:27.598334 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.598315    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f99bfd\"/\"openshift-service-ca.crt\""
Jul 27 20:06:27.598409 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.598370    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f99bfd\"/\"kube-root-ca.crt\""
Jul 27 20:06:27.604350 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.604330    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"]
Jul 27 20:06:27.644538 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.644511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-model-cache\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.644672 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.644541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-dshm\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.644672 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.644567    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-home\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.644672 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.644583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-tmp-dir\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.644839 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.644671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-tls-certs\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.644839 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.644731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58564\" (UniqueName: \"kubernetes.io/projected/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-kube-api-access-58564\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.745726 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.745698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-model-cache\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.745726 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.745728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-dshm\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.745935 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.745750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-home\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.745935 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.745767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-tmp-dir\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.745935 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.745795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-tls-certs\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.745935 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.745826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58564\" (UniqueName: \"kubernetes.io/projected/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-kube-api-access-58564\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.746160 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.746130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-model-cache\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.746207 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.746188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-tmp-dir\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.746270 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.746214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-home\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.747898 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.747880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-dshm\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.748158 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.748142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-tls-certs\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.754018 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.753998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58564\" (UniqueName: \"kubernetes.io/projected/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-kube-api-access-58564\") pod \"rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") " pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:27.904098 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:27.904019    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:28.039214 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:28.039184    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"]
Jul 27 20:06:28.039876 ip-10-0-129-103 kubenswrapper[2580]: W0727 20:06:28.039840    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda991a82e_125d_4ed1_ba36_c4cbf3c3f04f.slice/crio-5e311116fd21564d0969923faae6caeaace13455f86e4a0f659761f3fcec81b7 WatchSource:0}: Error finding container 5e311116fd21564d0969923faae6caeaace13455f86e4a0f659761f3fcec81b7: Status 404 returned error can't find the container with id 5e311116fd21564d0969923faae6caeaace13455f86e4a0f659761f3fcec81b7
Jul 27 20:06:28.303235 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:28.303199    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck" event={"ID":"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f","Type":"ContainerStarted","Data":"f73c559482d12989eacdd92b5ecc66619fb5c57dbc35995bf3f7da33e1d49080"}
Jul 27 20:06:28.303424 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:28.303262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck" event={"ID":"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f","Type":"ContainerStarted","Data":"5e311116fd21564d0969923faae6caeaace13455f86e4a0f659761f3fcec81b7"}
Jul 27 20:06:28.326832 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:28.326788    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck" podStartSLOduration=1.3267721940000001 podStartE2EDuration="1.326772194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:28.325405475 +0000 UTC m=+2930.841109981" watchObservedRunningTime="2026-07-27 20:06:28.326772194 +0000 UTC m=+2930.842476697"
Jul 27 20:06:37.904106 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:37.904075    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:37.904506 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:37.904120    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:37.916637 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:37.916612    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:06:38.365171 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:06:38.365143    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:07:42.635049 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:42.635010    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"]
Jul 27 20:07:43.691546 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:43.691512    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"]
Jul 27 20:07:47.683116 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:47.683080    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7498-140224303309488"]
Jul 27 20:07:47.686167 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:47.686150    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224303309488"
Jul 27 20:07:47.688690 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:47.688666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8xh8n\""
Jul 27 20:07:47.694813 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:47.694783    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-140224303309488"]
Jul 27 20:07:47.862536 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:47.862496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz4lg\" (UniqueName: \"kubernetes.io/projected/d5f36693-9f89-4639-a46e-0ff92c3ab2b6-kube-api-access-hz4lg\") pod \"test-curl-7498-140224303309488\" (UID: \"d5f36693-9f89-4639-a46e-0ff92c3ab2b6\") " pod="opendatahub/test-curl-7498-140224303309488"
Jul 27 20:07:47.964135 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:47.964033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz4lg\" (UniqueName: \"kubernetes.io/projected/d5f36693-9f89-4639-a46e-0ff92c3ab2b6-kube-api-access-hz4lg\") pod \"test-curl-7498-140224303309488\" (UID: \"d5f36693-9f89-4639-a46e-0ff92c3ab2b6\") " pod="opendatahub/test-curl-7498-140224303309488"
Jul 27 20:07:47.973370 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:47.973340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz4lg\" (UniqueName: \"kubernetes.io/projected/d5f36693-9f89-4639-a46e-0ff92c3ab2b6-kube-api-access-hz4lg\") pod \"test-curl-7498-140224303309488\" (UID: \"d5f36693-9f89-4639-a46e-0ff92c3ab2b6\") " pod="opendatahub/test-curl-7498-140224303309488"
Jul 27 20:07:47.996358 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:47.996322    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224303309488"
Jul 27 20:07:48.132984 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:48.131027    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-140224303309488"]
Jul 27 20:07:48.134071 ip-10-0-129-103 kubenswrapper[2580]: W0727 20:07:48.134044    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5f36693_9f89_4639_a46e_0ff92c3ab2b6.slice/crio-10c09da53c053e008031405408b7caadb1e0aa5678e57564e586c6a95f77a637 WatchSource:0}: Error finding container 10c09da53c053e008031405408b7caadb1e0aa5678e57564e586c6a95f77a637: Status 404 returned error can't find the container with id 10c09da53c053e008031405408b7caadb1e0aa5678e57564e586c6a95f77a637
Jul 27 20:07:48.135602 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:48.135584    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:07:48.705647 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:48.705599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-140224303309488" event={"ID":"d5f36693-9f89-4639-a46e-0ff92c3ab2b6","Type":"ContainerStarted","Data":"10c09da53c053e008031405408b7caadb1e0aa5678e57564e586c6a95f77a637"}
Jul 27 20:07:49.712032 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:49.711989    2580 generic.go:356] "Generic (PLEG): container finished" podID="d5f36693-9f89-4639-a46e-0ff92c3ab2b6" containerID="c7b54195f2ef85b761bbd11a0de1062327cd084f420ea6c8ebfa960832b81687" exitCode=0
Jul 27 20:07:49.712032 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:49.712027    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-140224303309488" event={"ID":"d5f36693-9f89-4639-a46e-0ff92c3ab2b6","Type":"ContainerDied","Data":"c7b54195f2ef85b761bbd11a0de1062327cd084f420ea6c8ebfa960832b81687"}
Jul 27 20:07:50.849556 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:50.849522    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224303309488"
Jul 27 20:07:50.889773 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:50.889738    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7498-140224303309488_d5f36693-9f89-4639-a46e-0ff92c3ab2b6/test-curl-7498-140224303309488/0.log"
Jul 27 20:07:50.919168 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:50.919075    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7498-140224303309488"]
Jul 27 20:07:50.921298 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:50.921274    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7498-140224303309488"]
Jul 27 20:07:50.990653 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:50.990559    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz4lg\" (UniqueName: \"kubernetes.io/projected/d5f36693-9f89-4639-a46e-0ff92c3ab2b6-kube-api-access-hz4lg\") pod \"d5f36693-9f89-4639-a46e-0ff92c3ab2b6\" (UID: \"d5f36693-9f89-4639-a46e-0ff92c3ab2b6\") "
Jul 27 20:07:50.992773 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:50.992746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f36693-9f89-4639-a46e-0ff92c3ab2b6-kube-api-access-hz4lg" (OuterVolumeSpecName: "kube-api-access-hz4lg") pod "d5f36693-9f89-4639-a46e-0ff92c3ab2b6" (UID: "d5f36693-9f89-4639-a46e-0ff92c3ab2b6"). InnerVolumeSpecName "kube-api-access-hz4lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:07:51.080058 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.080020    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7498-140224297900048"]
Jul 27 20:07:51.080400 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.080386    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f36693-9f89-4639-a46e-0ff92c3ab2b6" containerName="test-curl-7498-140224303309488"
Jul 27 20:07:51.080466 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.080402    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f36693-9f89-4639-a46e-0ff92c3ab2b6" containerName="test-curl-7498-140224303309488"
Jul 27 20:07:51.080466 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.080461    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f36693-9f89-4639-a46e-0ff92c3ab2b6" containerName="test-curl-7498-140224303309488"
Jul 27 20:07:51.083449 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.083434    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297900048"
Jul 27 20:07:51.091545 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.091517    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz4lg\" (UniqueName: \"kubernetes.io/projected/d5f36693-9f89-4639-a46e-0ff92c3ab2b6-kube-api-access-hz4lg\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:07:51.091670 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.091656    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-140224297900048"]
Jul 27 20:07:51.192987 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.192945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qn46\" (UniqueName: \"kubernetes.io/projected/8d31fc09-8d99-471f-b76f-a1136bb578e7-kube-api-access-4qn46\") pod \"test-curl-7498-140224297900048\" (UID: \"8d31fc09-8d99-471f-b76f-a1136bb578e7\") " pod="opendatahub/test-curl-7498-140224297900048"
Jul 27 20:07:51.293693 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.293656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qn46\" (UniqueName: \"kubernetes.io/projected/8d31fc09-8d99-471f-b76f-a1136bb578e7-kube-api-access-4qn46\") pod \"test-curl-7498-140224297900048\" (UID: \"8d31fc09-8d99-471f-b76f-a1136bb578e7\") " pod="opendatahub/test-curl-7498-140224297900048"
Jul 27 20:07:51.303796 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.303766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qn46\" (UniqueName: \"kubernetes.io/projected/8d31fc09-8d99-471f-b76f-a1136bb578e7-kube-api-access-4qn46\") pod \"test-curl-7498-140224297900048\" (UID: \"8d31fc09-8d99-471f-b76f-a1136bb578e7\") " pod="opendatahub/test-curl-7498-140224297900048"
Jul 27 20:07:51.394351 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.394313    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297900048"
Jul 27 20:07:51.526406 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.526337    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-140224297900048"]
Jul 27 20:07:51.528506 ip-10-0-129-103 kubenswrapper[2580]: W0727 20:07:51.528478    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d31fc09_8d99_471f_b76f_a1136bb578e7.slice/crio-c64ed2b1e06639f9f9eaad9905f63386967c6eea675f3c02856cc24b449c4aa3 WatchSource:0}: Error finding container c64ed2b1e06639f9f9eaad9905f63386967c6eea675f3c02856cc24b449c4aa3: Status 404 returned error can't find the container with id c64ed2b1e06639f9f9eaad9905f63386967c6eea675f3c02856cc24b449c4aa3
Jul 27 20:07:51.726615 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.726587    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224303309488"
Jul 27 20:07:51.726771 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.726624    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10c09da53c053e008031405408b7caadb1e0aa5678e57564e586c6a95f77a637"
Jul 27 20:07:51.727849 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:51.727825    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-140224297900048" event={"ID":"8d31fc09-8d99-471f-b76f-a1136bb578e7","Type":"ContainerStarted","Data":"c64ed2b1e06639f9f9eaad9905f63386967c6eea675f3c02856cc24b449c4aa3"}
Jul 27 20:07:51.737090 ip-10-0-129-103 kubenswrapper[2580]: E0727 20:07:51.737063    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7498-140224303309488\" is forbidden: User \"system:node:ip-10-0-129-103.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-103.ec2.internal' and this object" podUID="d5f36693-9f89-4639-a46e-0ff92c3ab2b6" pod="opendatahub/test-curl-7498-140224303309488"
Jul 27 20:07:52.086355 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:52.086262    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5f36693-9f89-4639-a46e-0ff92c3ab2b6" path="/var/lib/kubelet/pods/d5f36693-9f89-4639-a46e-0ff92c3ab2b6/volumes"
Jul 27 20:07:52.734173 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:52.734133    2580 generic.go:356] "Generic (PLEG): container finished" podID="8d31fc09-8d99-471f-b76f-a1136bb578e7" containerID="8d36a31fdb4a80616f928b24bd4a15b91f67f9acb2d0c43b07e7efc68455720e" exitCode=0
Jul 27 20:07:52.734379 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:52.734223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-140224297900048" event={"ID":"8d31fc09-8d99-471f-b76f-a1136bb578e7","Type":"ContainerDied","Data":"8d36a31fdb4a80616f928b24bd4a15b91f67f9acb2d0c43b07e7efc68455720e"}
Jul 27 20:07:53.869334 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:53.869312    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297900048"
Jul 27 20:07:53.895605 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:53.895578    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7498-140224297900048_8d31fc09-8d99-471f-b76f-a1136bb578e7/test-curl-7498-140224297900048/0.log"
Jul 27 20:07:53.915886 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:53.915862    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qn46\" (UniqueName: \"kubernetes.io/projected/8d31fc09-8d99-471f-b76f-a1136bb578e7-kube-api-access-4qn46\") pod \"8d31fc09-8d99-471f-b76f-a1136bb578e7\" (UID: \"8d31fc09-8d99-471f-b76f-a1136bb578e7\") "
Jul 27 20:07:53.917862 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:53.917831    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d31fc09-8d99-471f-b76f-a1136bb578e7-kube-api-access-4qn46" (OuterVolumeSpecName: "kube-api-access-4qn46") pod "8d31fc09-8d99-471f-b76f-a1136bb578e7" (UID: "8d31fc09-8d99-471f-b76f-a1136bb578e7"). InnerVolumeSpecName "kube-api-access-4qn46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:07:53.922377 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:53.922328    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7498-140224297900048"]
Jul 27 20:07:53.924927 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:53.924908    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7498-140224297900048"]
Jul 27 20:07:54.016879 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:54.016807    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qn46\" (UniqueName: \"kubernetes.io/projected/8d31fc09-8d99-471f-b76f-a1136bb578e7-kube-api-access-4qn46\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:07:54.089675 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:54.089639    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d31fc09-8d99-471f-b76f-a1136bb578e7" path="/var/lib/kubelet/pods/8d31fc09-8d99-471f-b76f-a1136bb578e7/volumes"
Jul 27 20:07:54.745764 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:54.745729    2580 scope.go:117] "RemoveContainer" containerID="8d36a31fdb4a80616f928b24bd4a15b91f67f9acb2d0c43b07e7efc68455720e"
Jul 27 20:07:54.745764 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:54.745751    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297900048"
Jul 27 20:07:57.653698 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.653657    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6" podUID="2dd3e429-bd0e-48a6-8c1c-632d7fe3180b" containerName="main" containerID="cri-o://b4202ffeddb86557dbe8d2bbb9b2f62e2fb9e2d55b1d8f809466a6497ef490b9" gracePeriod=45
Jul 27 20:07:57.893346 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.893325    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:07:57.949203 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.949128    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-tmp-dir\") pod \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") "
Jul 27 20:07:57.949203 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.949199    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-tls-certs\") pod \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") "
Jul 27 20:07:57.949418 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.949259    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-home\") pod \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") "
Jul 27 20:07:57.949418 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.949287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmq82\" (UniqueName: \"kubernetes.io/projected/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-kube-api-access-rmq82\") pod \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") "
Jul 27 20:07:57.949418 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.949324    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-dshm\") pod \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") "
Jul 27 20:07:57.949418 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.949341    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-model-cache\") pod \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\" (UID: \"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b\") "
Jul 27 20:07:57.949628 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.949438    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2dd3e429-bd0e-48a6-8c1c-632d7fe3180b" (UID: "2dd3e429-bd0e-48a6-8c1c-632d7fe3180b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:57.949683 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.949625    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:07:57.949683 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.949640    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-home" (OuterVolumeSpecName: "home") pod "2dd3e429-bd0e-48a6-8c1c-632d7fe3180b" (UID: "2dd3e429-bd0e-48a6-8c1c-632d7fe3180b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:57.949683 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.949649    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-model-cache" (OuterVolumeSpecName: "model-cache") pod "2dd3e429-bd0e-48a6-8c1c-632d7fe3180b" (UID: "2dd3e429-bd0e-48a6-8c1c-632d7fe3180b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:57.951303 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.951282    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2dd3e429-bd0e-48a6-8c1c-632d7fe3180b" (UID: "2dd3e429-bd0e-48a6-8c1c-632d7fe3180b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:57.951387 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.951310    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-dshm" (OuterVolumeSpecName: "dshm") pod "2dd3e429-bd0e-48a6-8c1c-632d7fe3180b" (UID: "2dd3e429-bd0e-48a6-8c1c-632d7fe3180b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:57.951422 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:57.951393    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-kube-api-access-rmq82" (OuterVolumeSpecName: "kube-api-access-rmq82") pod "2dd3e429-bd0e-48a6-8c1c-632d7fe3180b" (UID: "2dd3e429-bd0e-48a6-8c1c-632d7fe3180b"). InnerVolumeSpecName "kube-api-access-rmq82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:07:58.050049 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.050014    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmq82\" (UniqueName: \"kubernetes.io/projected/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-kube-api-access-rmq82\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:07:58.050049 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.050043    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:07:58.050049 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.050053    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:07:58.050287 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.050062    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:07:58.050287 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.050070    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:07:58.709793 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.709749    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck" podUID="a991a82e-125d-4ed1-ba36-c4cbf3c3f04f" containerName="main" containerID="cri-o://f73c559482d12989eacdd92b5ecc66619fb5c57dbc35995bf3f7da33e1d49080" gracePeriod=45
Jul 27 20:07:58.767025 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.766997    2580 generic.go:356] "Generic (PLEG): container finished" podID="2dd3e429-bd0e-48a6-8c1c-632d7fe3180b" containerID="b4202ffeddb86557dbe8d2bbb9b2f62e2fb9e2d55b1d8f809466a6497ef490b9" exitCode=0
Jul 27 20:07:58.767146 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.767058    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"
Jul 27 20:07:58.767146 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.767082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6" event={"ID":"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b","Type":"ContainerDied","Data":"b4202ffeddb86557dbe8d2bbb9b2f62e2fb9e2d55b1d8f809466a6497ef490b9"}
Jul 27 20:07:58.767146 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.767135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6" event={"ID":"2dd3e429-bd0e-48a6-8c1c-632d7fe3180b","Type":"ContainerDied","Data":"7c1e9157ddab4347a69e2fce4ad9ca48c9d724edf544c23bb4a30a55ee85c86d"}
Jul 27 20:07:58.767276 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.767160    2580 scope.go:117] "RemoveContainer" containerID="b4202ffeddb86557dbe8d2bbb9b2f62e2fb9e2d55b1d8f809466a6497ef490b9"
Jul 27 20:07:58.788444 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.788423    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"]
Jul 27 20:07:58.793570 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.793549    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4289a9/rate-test-model-4289a9-kserve-66d64dcdd7-7dfl6"]
Jul 27 20:07:58.820778 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.820759    2580 scope.go:117] "RemoveContainer" containerID="b4202ffeddb86557dbe8d2bbb9b2f62e2fb9e2d55b1d8f809466a6497ef490b9"
Jul 27 20:07:58.821042 ip-10-0-129-103 kubenswrapper[2580]: E0727 20:07:58.821022    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4202ffeddb86557dbe8d2bbb9b2f62e2fb9e2d55b1d8f809466a6497ef490b9\": container with ID starting with b4202ffeddb86557dbe8d2bbb9b2f62e2fb9e2d55b1d8f809466a6497ef490b9 not found: ID does not exist" containerID="b4202ffeddb86557dbe8d2bbb9b2f62e2fb9e2d55b1d8f809466a6497ef490b9"
Jul 27 20:07:58.821102 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.821058    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4202ffeddb86557dbe8d2bbb9b2f62e2fb9e2d55b1d8f809466a6497ef490b9"} err="failed to get container status \"b4202ffeddb86557dbe8d2bbb9b2f62e2fb9e2d55b1d8f809466a6497ef490b9\": rpc error: code = NotFound desc = could not find container \"b4202ffeddb86557dbe8d2bbb9b2f62e2fb9e2d55b1d8f809466a6497ef490b9\": container with ID starting with b4202ffeddb86557dbe8d2bbb9b2f62e2fb9e2d55b1d8f809466a6497ef490b9 not found: ID does not exist"
Jul 27 20:07:58.958143 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:58.958116    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:07:59.058783 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.058751    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-dshm\") pod \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") "
Jul 27 20:07:59.058936 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.058857    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-tls-certs\") pod \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") "
Jul 27 20:07:59.058936 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.058911    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-model-cache\") pod \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") "
Jul 27 20:07:59.059017 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.058937    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58564\" (UniqueName: \"kubernetes.io/projected/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-kube-api-access-58564\") pod \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") "
Jul 27 20:07:59.059017 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.058979    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-tmp-dir\") pod \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") "
Jul 27 20:07:59.059017 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.059008    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-home\") pod \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\" (UID: \"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f\") "
Jul 27 20:07:59.059172 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.059151    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a991a82e-125d-4ed1-ba36-c4cbf3c3f04f" (UID: "a991a82e-125d-4ed1-ba36-c4cbf3c3f04f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:59.059322 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.059277    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a991a82e-125d-4ed1-ba36-c4cbf3c3f04f" (UID: "a991a82e-125d-4ed1-ba36-c4cbf3c3f04f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:59.059445 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.059307    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:07:59.059445 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.059350    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-home" (OuterVolumeSpecName: "home") pod "a991a82e-125d-4ed1-ba36-c4cbf3c3f04f" (UID: "a991a82e-125d-4ed1-ba36-c4cbf3c3f04f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:59.060820 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.060789    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-dshm" (OuterVolumeSpecName: "dshm") pod "a991a82e-125d-4ed1-ba36-c4cbf3c3f04f" (UID: "a991a82e-125d-4ed1-ba36-c4cbf3c3f04f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:59.061006 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.060981    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a991a82e-125d-4ed1-ba36-c4cbf3c3f04f" (UID: "a991a82e-125d-4ed1-ba36-c4cbf3c3f04f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:59.061093 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.060986    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-kube-api-access-58564" (OuterVolumeSpecName: "kube-api-access-58564") pod "a991a82e-125d-4ed1-ba36-c4cbf3c3f04f" (UID: "a991a82e-125d-4ed1-ba36-c4cbf3c3f04f"). InnerVolumeSpecName "kube-api-access-58564". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:07:59.160113 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.160080    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:07:59.160113 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.160109    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:07:59.160113 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.160118    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:07:59.160386 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.160129    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:07:59.160386 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.160142    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58564\" (UniqueName: \"kubernetes.io/projected/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f-kube-api-access-58564\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:07:59.757015 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.756980    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7498-140224297901056"]
Jul 27 20:07:59.757393 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.757296    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd3e429-bd0e-48a6-8c1c-632d7fe3180b" containerName="main"
Jul 27 20:07:59.757393 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.757309    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd3e429-bd0e-48a6-8c1c-632d7fe3180b" containerName="main"
Jul 27 20:07:59.757393 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.757321    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d31fc09-8d99-471f-b76f-a1136bb578e7" containerName="test-curl-7498-140224297900048"
Jul 27 20:07:59.757393 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.757327    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d31fc09-8d99-471f-b76f-a1136bb578e7" containerName="test-curl-7498-140224297900048"
Jul 27 20:07:59.757393 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.757336    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a991a82e-125d-4ed1-ba36-c4cbf3c3f04f" containerName="main"
Jul 27 20:07:59.757393 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.757341    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a991a82e-125d-4ed1-ba36-c4cbf3c3f04f" containerName="main"
Jul 27 20:07:59.757393 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.757395    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd3e429-bd0e-48a6-8c1c-632d7fe3180b" containerName="main"
Jul 27 20:07:59.757626 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.757405    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d31fc09-8d99-471f-b76f-a1136bb578e7" containerName="test-curl-7498-140224297900048"
Jul 27 20:07:59.757626 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.757412    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a991a82e-125d-4ed1-ba36-c4cbf3c3f04f" containerName="main"
Jul 27 20:07:59.760558 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.760542    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297901056"
Jul 27 20:07:59.763053 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.763034    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8xh8n\""
Jul 27 20:07:59.769071 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.769040    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-140224297901056"]
Jul 27 20:07:59.774523 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.774500    2580 generic.go:356] "Generic (PLEG): container finished" podID="a991a82e-125d-4ed1-ba36-c4cbf3c3f04f" containerID="f73c559482d12989eacdd92b5ecc66619fb5c57dbc35995bf3f7da33e1d49080" exitCode=0
Jul 27 20:07:59.774621 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.774536    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"
Jul 27 20:07:59.774621 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.774552    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck" event={"ID":"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f","Type":"ContainerDied","Data":"f73c559482d12989eacdd92b5ecc66619fb5c57dbc35995bf3f7da33e1d49080"}
Jul 27 20:07:59.774621 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.774583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck" event={"ID":"a991a82e-125d-4ed1-ba36-c4cbf3c3f04f","Type":"ContainerDied","Data":"5e311116fd21564d0969923faae6caeaace13455f86e4a0f659761f3fcec81b7"}
Jul 27 20:07:59.774621 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.774600    2580 scope.go:117] "RemoveContainer" containerID="f73c559482d12989eacdd92b5ecc66619fb5c57dbc35995bf3f7da33e1d49080"
Jul 27 20:07:59.785227 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.785198    2580 scope.go:117] "RemoveContainer" containerID="f73c559482d12989eacdd92b5ecc66619fb5c57dbc35995bf3f7da33e1d49080"
Jul 27 20:07:59.785516 ip-10-0-129-103 kubenswrapper[2580]: E0727 20:07:59.785489    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73c559482d12989eacdd92b5ecc66619fb5c57dbc35995bf3f7da33e1d49080\": container with ID starting with f73c559482d12989eacdd92b5ecc66619fb5c57dbc35995bf3f7da33e1d49080 not found: ID does not exist" containerID="f73c559482d12989eacdd92b5ecc66619fb5c57dbc35995bf3f7da33e1d49080"
Jul 27 20:07:59.785608 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.785521    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73c559482d12989eacdd92b5ecc66619fb5c57dbc35995bf3f7da33e1d49080"} err="failed to get container status \"f73c559482d12989eacdd92b5ecc66619fb5c57dbc35995bf3f7da33e1d49080\": rpc error: code = NotFound desc = could not find container \"f73c559482d12989eacdd92b5ecc66619fb5c57dbc35995bf3f7da33e1d49080\": container with ID starting with f73c559482d12989eacdd92b5ecc66619fb5c57dbc35995bf3f7da33e1d49080 not found: ID does not exist"
Jul 27 20:07:59.801768 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.801739    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"]
Jul 27 20:07:59.809553 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.809528    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f99bfd/rate-test-model-f99bfd-kserve-58c4d4b4f9-jjcck"]
Jul 27 20:07:59.867218 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.867182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpcl6\" (UniqueName: \"kubernetes.io/projected/40e965f2-28fd-4eb8-a995-6f4665ddbc8e-kube-api-access-zpcl6\") pod \"test-curl-7498-140224297901056\" (UID: \"40e965f2-28fd-4eb8-a995-6f4665ddbc8e\") " pod="opendatahub/test-curl-7498-140224297901056"
Jul 27 20:07:59.968313 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.968262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpcl6\" (UniqueName: \"kubernetes.io/projected/40e965f2-28fd-4eb8-a995-6f4665ddbc8e-kube-api-access-zpcl6\") pod \"test-curl-7498-140224297901056\" (UID: \"40e965f2-28fd-4eb8-a995-6f4665ddbc8e\") " pod="opendatahub/test-curl-7498-140224297901056"
Jul 27 20:07:59.977297 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:07:59.977272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpcl6\" (UniqueName: \"kubernetes.io/projected/40e965f2-28fd-4eb8-a995-6f4665ddbc8e-kube-api-access-zpcl6\") pod \"test-curl-7498-140224297901056\" (UID: \"40e965f2-28fd-4eb8-a995-6f4665ddbc8e\") " pod="opendatahub/test-curl-7498-140224297901056"
Jul 27 20:08:00.070364 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:00.070327    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297901056"
Jul 27 20:08:00.086945 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:00.086918    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd3e429-bd0e-48a6-8c1c-632d7fe3180b" path="/var/lib/kubelet/pods/2dd3e429-bd0e-48a6-8c1c-632d7fe3180b/volumes"
Jul 27 20:08:00.087380 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:00.087362    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a991a82e-125d-4ed1-ba36-c4cbf3c3f04f" path="/var/lib/kubelet/pods/a991a82e-125d-4ed1-ba36-c4cbf3c3f04f/volumes"
Jul 27 20:08:00.217582 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:00.217546    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-140224297901056"]
Jul 27 20:08:00.220193 ip-10-0-129-103 kubenswrapper[2580]: W0727 20:08:00.220165    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40e965f2_28fd_4eb8_a995_6f4665ddbc8e.slice/crio-f63383339aa9f1b00ebfe30c503f36de6aed7f94d31e62155e5646c4c175ef26 WatchSource:0}: Error finding container f63383339aa9f1b00ebfe30c503f36de6aed7f94d31e62155e5646c4c175ef26: Status 404 returned error can't find the container with id f63383339aa9f1b00ebfe30c503f36de6aed7f94d31e62155e5646c4c175ef26
Jul 27 20:08:00.781990 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:00.781899    2580 generic.go:356] "Generic (PLEG): container finished" podID="40e965f2-28fd-4eb8-a995-6f4665ddbc8e" containerID="24e4458212b451a282b3b11dd03d8f43123fb3f676fe1f567b2efdfc6aeadda9" exitCode=0
Jul 27 20:08:00.782396 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:00.781990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-140224297901056" event={"ID":"40e965f2-28fd-4eb8-a995-6f4665ddbc8e","Type":"ContainerDied","Data":"24e4458212b451a282b3b11dd03d8f43123fb3f676fe1f567b2efdfc6aeadda9"}
Jul 27 20:08:00.782396 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:00.782032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-140224297901056" event={"ID":"40e965f2-28fd-4eb8-a995-6f4665ddbc8e","Type":"ContainerStarted","Data":"f63383339aa9f1b00ebfe30c503f36de6aed7f94d31e62155e5646c4c175ef26"}
Jul 27 20:08:01.919053 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:01.919031    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297901056"
Jul 27 20:08:01.947550 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:01.947529    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7498-140224297901056_40e965f2-28fd-4eb8-a995-6f4665ddbc8e/test-curl-7498-140224297901056/0.log"
Jul 27 20:08:01.968909 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:01.968885    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7498-140224297901056"]
Jul 27 20:08:01.970860 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:01.970837    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7498-140224297901056"]
Jul 27 20:08:01.984974 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:01.984949    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpcl6\" (UniqueName: \"kubernetes.io/projected/40e965f2-28fd-4eb8-a995-6f4665ddbc8e-kube-api-access-zpcl6\") pod \"40e965f2-28fd-4eb8-a995-6f4665ddbc8e\" (UID: \"40e965f2-28fd-4eb8-a995-6f4665ddbc8e\") "
Jul 27 20:08:01.986915 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:01.986886    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e965f2-28fd-4eb8-a995-6f4665ddbc8e-kube-api-access-zpcl6" (OuterVolumeSpecName: "kube-api-access-zpcl6") pod "40e965f2-28fd-4eb8-a995-6f4665ddbc8e" (UID: "40e965f2-28fd-4eb8-a995-6f4665ddbc8e"). InnerVolumeSpecName "kube-api-access-zpcl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:08:02.085617 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.085549    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpcl6\" (UniqueName: \"kubernetes.io/projected/40e965f2-28fd-4eb8-a995-6f4665ddbc8e-kube-api-access-zpcl6\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:08:02.086315 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.086296    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40e965f2-28fd-4eb8-a995-6f4665ddbc8e" path="/var/lib/kubelet/pods/40e965f2-28fd-4eb8-a995-6f4665ddbc8e/volumes"
Jul 27 20:08:02.795203 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.795175    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297901056"
Jul 27 20:08:02.795203 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.795203    2580 scope.go:117] "RemoveContainer" containerID="24e4458212b451a282b3b11dd03d8f43123fb3f676fe1f567b2efdfc6aeadda9"
Jul 27 20:08:02.829163 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.829137    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"]
Jul 27 20:08:02.829470 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.829456    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e965f2-28fd-4eb8-a995-6f4665ddbc8e" containerName="test-curl-7498-140224297901056"
Jul 27 20:08:02.829510 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.829473    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e965f2-28fd-4eb8-a995-6f4665ddbc8e" containerName="test-curl-7498-140224297901056"
Jul 27 20:08:02.829552 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.829534    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e965f2-28fd-4eb8-a995-6f4665ddbc8e" containerName="test-curl-7498-140224297901056"
Jul 27 20:08:02.833552 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.833537    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.836793 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.836763    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-58c998-openshift-default-dockercfg-7tfs7\""
Jul 27 20:08:02.836793 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.836779    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 20:08:02.848078 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.848055    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"]
Jul 27 20:08:02.891567 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.891537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b40db455-a2a9-4eb3-9be9-fce3b9462512-istiod-ca-cert\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.891690 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.891571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-podinfo\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.891690 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.891595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-token\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.891775 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.891699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-data\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.891775 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.891745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-credential-socket\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.891839 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.891778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-workload-certs\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.891839 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.891807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-workload-socket\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.891839 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.891830    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-envoy\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.891931 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.891856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg2vd\" (UniqueName: \"kubernetes.io/projected/b40db455-a2a9-4eb3-9be9-fce3b9462512-kube-api-access-pg2vd\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.992366 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.992327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-envoy\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.992366 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.992372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg2vd\" (UniqueName: \"kubernetes.io/projected/b40db455-a2a9-4eb3-9be9-fce3b9462512-kube-api-access-pg2vd\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.992887 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.992408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b40db455-a2a9-4eb3-9be9-fce3b9462512-istiod-ca-cert\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.992887 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.992425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-podinfo\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.992887 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.992442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-token\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.992887 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.992471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-data\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.992887 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.992688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-credential-socket\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.992887 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.992763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-workload-certs\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.992887 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.992792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-workload-socket\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.993268 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.992888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-data\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.993268 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.993068    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-workload-certs\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.993268 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.993075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-credential-socket\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.993268 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.993167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-workload-socket\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.993443 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.993329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b40db455-a2a9-4eb3-9be9-fce3b9462512-istiod-ca-cert\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.994632 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.994612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-envoy\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:02.994896 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:02.994877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-podinfo\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:03.003097 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:03.003074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-token\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:03.003566 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:03.003544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg2vd\" (UniqueName: \"kubernetes.io/projected/b40db455-a2a9-4eb3-9be9-fce3b9462512-kube-api-access-pg2vd\") pod \"e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") " pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:03.143703 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:03.143622    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:03.280532 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:03.280509    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"]
Jul 27 20:08:03.282447 ip-10-0-129-103 kubenswrapper[2580]: W0727 20:08:03.282421    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40db455_a2a9_4eb3_9be9_fce3b9462512.slice/crio-bc3e2270cea0bb9333588cd3a6df5438bd9d4ace4c7f9eb33c06d71e2c7990da WatchSource:0}: Error finding container bc3e2270cea0bb9333588cd3a6df5438bd9d4ace4c7f9eb33c06d71e2c7990da: Status 404 returned error can't find the container with id bc3e2270cea0bb9333588cd3a6df5438bd9d4ace4c7f9eb33c06d71e2c7990da
Jul 27 20:08:03.802782 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:03.802748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c" event={"ID":"b40db455-a2a9-4eb3-9be9-fce3b9462512","Type":"ContainerStarted","Data":"bc3e2270cea0bb9333588cd3a6df5438bd9d4ace4c7f9eb33c06d71e2c7990da"}
Jul 27 20:08:06.065213 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:06.065157    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:08:06.065490 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:06.065265    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:08:06.065490 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:06.065296    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:08:06.822629 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:06.822590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c" event={"ID":"b40db455-a2a9-4eb3-9be9-fce3b9462512","Type":"ContainerStarted","Data":"564017b5211f99037a51bfa4f2bc7715a5c6c84ac92029a348cd9a46d87389ab"}
Jul 27 20:08:06.845748 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:06.845699    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c" podStartSLOduration=2.065491059 podStartE2EDuration="4.845685704s" totalImagesPullingTime="2.780194645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:08:06.843611554 +0000 UTC m=+3029.359316085" watchObservedRunningTime="2026-07-27 20:08:06.845685704 +0000 UTC m=+3029.361390207"
Jul 27 20:08:07.144626 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:07.144548    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:07.149467 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:07.149435    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:07.828156 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:07.828125    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:07.829071 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:07.829055    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:20.903550 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:20.903522    2580 generic.go:356] "Generic (PLEG): container finished" podID="b40db455-a2a9-4eb3-9be9-fce3b9462512" containerID="564017b5211f99037a51bfa4f2bc7715a5c6c84ac92029a348cd9a46d87389ab" exitCode=0
Jul 27 20:08:20.903893 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:20.903583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c" event={"ID":"b40db455-a2a9-4eb3-9be9-fce3b9462512","Type":"ContainerDied","Data":"564017b5211f99037a51bfa4f2bc7715a5c6c84ac92029a348cd9a46d87389ab"}
Jul 27 20:08:20.903981 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:20.903968    2580 scope.go:117] "RemoveContainer" containerID="564017b5211f99037a51bfa4f2bc7715a5c6c84ac92029a348cd9a46d87389ab"
Jul 27 20:08:20.905092 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:20.905065    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:08:20.905160 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:20.905137    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:08:20.905197 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:20.905168    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:08:21.911286 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:21.911224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c" event={"ID":"b40db455-a2a9-4eb3-9be9-fce3b9462512","Type":"ContainerStarted","Data":"f82a271b871ce047331f223d883f3f3689592cc368faeeb4356517e371da7e06"}
Jul 27 20:08:22.144654 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:22.144613    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:22.145733 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:22.145690    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 27 20:08:22.145857 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:22.145757    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c" podUID="b40db455-a2a9-4eb3-9be9-fce3b9462512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 27 20:08:22.828416 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:22.828362    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:23.144991 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:23.144882    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 27 20:08:23.144991 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:23.144955    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c" podUID="b40db455-a2a9-4eb3-9be9-fce3b9462512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 27 20:08:24.148170 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.147665    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 27 20:08:24.148170 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.147732    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c" podUID="b40db455-a2a9-4eb3-9be9-fce3b9462512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 27 20:08:24.532841 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.532746    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"]
Jul 27 20:08:24.535256 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.535222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.537981 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.537958    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-791fb5-openshift-default-dockercfg-4q59k\""
Jul 27 20:08:24.569991 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.569944    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"]
Jul 27 20:08:24.690924 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.690887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-workload-socket\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.690924 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.690928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ac46c6-257b-4086-bb0a-7e293480800d-istio-token\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.691221 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.690951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-istio-data\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.691221 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.690966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ac46c6-257b-4086-bb0a-7e293480800d-istiod-ca-cert\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.691221 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.691088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ac46c6-257b-4086-bb0a-7e293480800d-istio-podinfo\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.691221 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.691128    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9rl6\" (UniqueName: \"kubernetes.io/projected/71ac46c6-257b-4086-bb0a-7e293480800d-kube-api-access-n9rl6\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.691221 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.691190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-credential-socket\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.691472 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.691225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-istio-envoy\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.691472 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.691310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-workload-certs\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.792374 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.792336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-istio-data\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.792580 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.792379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ac46c6-257b-4086-bb0a-7e293480800d-istiod-ca-cert\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.792580 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.792439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ac46c6-257b-4086-bb0a-7e293480800d-istio-podinfo\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.792580 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.792459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9rl6\" (UniqueName: \"kubernetes.io/projected/71ac46c6-257b-4086-bb0a-7e293480800d-kube-api-access-n9rl6\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.792580 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.792502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-credential-socket\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.792824 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.792679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-istio-envoy\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.792824 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.792769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-workload-certs\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.792932 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.792863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-istio-data\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.792932 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.792917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-credential-socket\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.793119 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.793086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-workload-certs\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.793271 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.793129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-workload-socket\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.793271 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.793160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-workload-socket\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.793271 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.793171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ac46c6-257b-4086-bb0a-7e293480800d-istio-token\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.793460 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.793426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ac46c6-257b-4086-bb0a-7e293480800d-istiod-ca-cert\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.795558 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.795526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-istio-envoy\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.795680 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.795659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ac46c6-257b-4086-bb0a-7e293480800d-istio-podinfo\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.801832 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.801809    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ac46c6-257b-4086-bb0a-7e293480800d-istio-token\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.801981 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.801959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9rl6\" (UniqueName: \"kubernetes.io/projected/71ac46c6-257b-4086-bb0a-7e293480800d-kube-api-access-n9rl6\") pod \"e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") " pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:24.861681 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:24.861642    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:25.036220 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:25.036194    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"]
Jul 27 20:08:25.037810 ip-10-0-129-103 kubenswrapper[2580]: W0727 20:08:25.037769    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ac46c6_257b_4086_bb0a_7e293480800d.slice/crio-e626918a02fc94dc9e9d50c0d145ca7281e8df619eda23588d294f6cb7d07af2 WatchSource:0}: Error finding container e626918a02fc94dc9e9d50c0d145ca7281e8df619eda23588d294f6cb7d07af2: Status 404 returned error can't find the container with id e626918a02fc94dc9e9d50c0d145ca7281e8df619eda23588d294f6cb7d07af2
Jul 27 20:08:25.040126 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:25.040094    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:08:25.040209 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:25.040163    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:08:25.040209 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:25.040191    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:08:25.155120 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:25.154695    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 27 20:08:25.155120 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:25.154757    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c" podUID="b40db455-a2a9-4eb3-9be9-fce3b9462512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 27 20:08:25.935536 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:25.935497    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn" event={"ID":"71ac46c6-257b-4086-bb0a-7e293480800d","Type":"ContainerStarted","Data":"78317a4a21cf70035d7e15690abe85aff86bfff2b8c9e3e599e10833c73130c4"}
Jul 27 20:08:25.935752 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:25.935549    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn" event={"ID":"71ac46c6-257b-4086-bb0a-7e293480800d","Type":"ContainerStarted","Data":"e626918a02fc94dc9e9d50c0d145ca7281e8df619eda23588d294f6cb7d07af2"}
Jul 27 20:08:25.993942 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:25.993878    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn" podStartSLOduration=1.9938588720000001 podStartE2EDuration="1.993858872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:08:25.99169443 +0000 UTC m=+3048.507398933" watchObservedRunningTime="2026-07-27 20:08:25.993858872 +0000 UTC m=+3048.509563377"
Jul 27 20:08:26.145087 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:26.145040    2580 patch_prober.go:28] interesting pod/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 27 20:08:26.145313 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:26.145149    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c" podUID="b40db455-a2a9-4eb3-9be9-fce3b9462512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 27 20:08:26.862143 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:26.862101    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:26.867497 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:26.867473    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:26.940963 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:26.940923    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:26.941782 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:26.941765    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:27.148544 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:27.148467    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:27.149409 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:27.149392    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:08:42.982507 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:42.982464    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 20:08:42.983979 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:42.982555    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn" podUID="71ac46c6-257b-4086-bb0a-7e293480800d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 20:08:43.034821 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:43.034788    2580 generic.go:356] "Generic (PLEG): container finished" podID="71ac46c6-257b-4086-bb0a-7e293480800d" containerID="78317a4a21cf70035d7e15690abe85aff86bfff2b8c9e3e599e10833c73130c4" exitCode=0
Jul 27 20:08:43.034977 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:43.034854    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn" event={"ID":"71ac46c6-257b-4086-bb0a-7e293480800d","Type":"ContainerDied","Data":"78317a4a21cf70035d7e15690abe85aff86bfff2b8c9e3e599e10833c73130c4"}
Jul 27 20:08:43.035289 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:43.035275    2580 scope.go:117] "RemoveContainer" containerID="78317a4a21cf70035d7e15690abe85aff86bfff2b8c9e3e599e10833c73130c4"
Jul 27 20:08:43.036356 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:43.036328    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:08:43.036462 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:43.036410    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:08:43.036531 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:43.036473    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:08:44.042091 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:44.042042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn" event={"ID":"71ac46c6-257b-4086-bb0a-7e293480800d","Type":"ContainerStarted","Data":"505dd7064bc547630b066e6a27d42b28061cee328c71e2cbadfe97c9ca59a610"}
Jul 27 20:08:44.862791 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:44.862753    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:44.863200 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:44.863174    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 27 20:08:44.863398 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:44.863368    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn" podUID="71ac46c6-257b-4086-bb0a-7e293480800d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 27 20:08:45.863010 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:45.862963    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 27 20:08:45.863538 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:45.863035    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn" podUID="71ac46c6-257b-4086-bb0a-7e293480800d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 27 20:08:46.862879 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:46.862837    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 27 20:08:46.863676 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:46.863636    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn" podUID="71ac46c6-257b-4086-bb0a-7e293480800d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 27 20:08:47.862933 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:47.862889    2580 patch_prober.go:28] interesting pod/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 27 20:08:47.863696 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:47.863236    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn" podUID="71ac46c6-257b-4086-bb0a-7e293480800d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 27 20:08:48.865602 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:48.865572    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:48.866130 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:48.865843    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:48.866669 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:48.866653    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:08:59.010565 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.010512    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"]
Jul 27 20:08:59.014619 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.014599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.018640 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.018601    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-58c998\"/\"ipp-route-58c998-kserve-self-signed-certs\""
Jul 27 20:08:59.018783 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.018700    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-58c998\"/\"openshift-service-ca.crt\""
Jul 27 20:08:59.018783 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.018724    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-58c998\"/\"kube-root-ca.crt\""
Jul 27 20:08:59.018783 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.018728    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-58c998\"/\"default-dockercfg-t9zw7\""
Jul 27 20:08:59.028542 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.028521    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"]
Jul 27 20:08:59.106410 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.106373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-dshm\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.106588 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.106421    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024e7232-f884-4dce-be6f-f3f9ef370d80-tls-certs\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.106588 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.106453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-model-cache\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.106588 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.106482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-home\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.106588 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.106505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fph5\" (UniqueName: \"kubernetes.io/projected/024e7232-f884-4dce-be6f-f3f9ef370d80-kube-api-access-8fph5\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.106732 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.106613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-tmp-dir\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.207764 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.207721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024e7232-f884-4dce-be6f-f3f9ef370d80-tls-certs\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.207764 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.207769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-model-cache\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.208038 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.207798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-home\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.208038 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.207825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fph5\" (UniqueName: \"kubernetes.io/projected/024e7232-f884-4dce-be6f-f3f9ef370d80-kube-api-access-8fph5\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.208038 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.207897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-tmp-dir\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.208038 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.207942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-dshm\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.208282 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.208228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-model-cache\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.208353 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.208277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-home\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.208353 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.208336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-tmp-dir\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.210208 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.210185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-dshm\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.210524 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.210505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024e7232-f884-4dce-be6f-f3f9ef370d80-tls-certs\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.216557 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.216536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fph5\" (UniqueName: \"kubernetes.io/projected/024e7232-f884-4dce-be6f-f3f9ef370d80-kube-api-access-8fph5\") pod \"ipp-route-58c998-kserve-7f487b747-vtqdp\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") " pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.323920 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.323882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:08:59.464620 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:08:59.464593    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"]
Jul 27 20:08:59.465757 ip-10-0-129-103 kubenswrapper[2580]: W0727 20:08:59.465727    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod024e7232_f884_4dce_be6f_f3f9ef370d80.slice/crio-289b1f7a9884663499f8a0afaf39f9e1d0b25ec540420ceef18b981ef8296d08 WatchSource:0}: Error finding container 289b1f7a9884663499f8a0afaf39f9e1d0b25ec540420ceef18b981ef8296d08: Status 404 returned error can't find the container with id 289b1f7a9884663499f8a0afaf39f9e1d0b25ec540420ceef18b981ef8296d08
Jul 27 20:09:00.134499 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:00.134454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp" event={"ID":"024e7232-f884-4dce-be6f-f3f9ef370d80","Type":"ContainerStarted","Data":"e3d71ed77e799ef511b43a0a21e6d0be14c275d3326a46ec2af1c49f069ab95e"}
Jul 27 20:09:00.134902 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:00.134512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp" event={"ID":"024e7232-f884-4dce-be6f-f3f9ef370d80","Type":"ContainerStarted","Data":"289b1f7a9884663499f8a0afaf39f9e1d0b25ec540420ceef18b981ef8296d08"}
Jul 27 20:09:00.155633 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:00.155580    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp" podStartSLOduration=2.155565263 podStartE2EDuration="2.155565263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:09:00.154079666 +0000 UTC m=+3082.669784170" watchObservedRunningTime="2026-07-27 20:09:00.155565263 +0000 UTC m=+3082.671269766"
Jul 27 20:09:09.324669 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:09.324630    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:09:09.325133 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:09.324690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:09:09.337382 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:09.337353    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:09:10.197379 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:10.197348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:09:42.583554 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.583515    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"]
Jul 27 20:09:42.587336 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.587312    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.589846 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.589823    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-d8e48b-openshift-default-dockercfg-hp8dv\""
Jul 27 20:09:42.614084 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.614052    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"]
Jul 27 20:09:42.725463 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.725430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-workload-socket\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.725668 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.725472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-workload-certs\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.725668 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.725492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61fb9496-7142-445d-a970-06f33e1314a3-istio-token\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.725668 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.725589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61fb9496-7142-445d-a970-06f33e1314a3-istio-podinfo\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.725668 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.725628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-credential-socket\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.725891 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.725712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61fb9496-7142-445d-a970-06f33e1314a3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.725891 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.725754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-istio-data\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.725891 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.725800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-istio-envoy\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.725891 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.725832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jpq4\" (UniqueName: \"kubernetes.io/projected/61fb9496-7142-445d-a970-06f33e1314a3-kube-api-access-6jpq4\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.827396 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.827356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-istio-data\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.827396 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.827400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-istio-envoy\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.827656 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.827420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jpq4\" (UniqueName: \"kubernetes.io/projected/61fb9496-7142-445d-a970-06f33e1314a3-kube-api-access-6jpq4\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.827656 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.827503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-workload-socket\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.827656 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.827534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-workload-certs\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.827656 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.827557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61fb9496-7142-445d-a970-06f33e1314a3-istio-token\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.827656 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.827590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61fb9496-7142-445d-a970-06f33e1314a3-istio-podinfo\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.827656 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.827618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-credential-socket\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.827656 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.827655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61fb9496-7142-445d-a970-06f33e1314a3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.827988 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.827830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-istio-data\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.828068 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.828042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-workload-socket\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.828145 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.828096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-workload-certs\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.828145 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.828100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-credential-socket\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.828549 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.828530    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61fb9496-7142-445d-a970-06f33e1314a3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.829891 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.829869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-istio-envoy\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.829977 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.829937    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61fb9496-7142-445d-a970-06f33e1314a3-istio-podinfo\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.836903 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.836833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jpq4\" (UniqueName: \"kubernetes.io/projected/61fb9496-7142-445d-a970-06f33e1314a3-kube-api-access-6jpq4\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.838551 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.838529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61fb9496-7142-445d-a970-06f33e1314a3-istio-token\") pod \"e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") " pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:42.899831 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:42.899786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:43.050911 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:43.050882    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"]
Jul 27 20:09:43.052623 ip-10-0-129-103 kubenswrapper[2580]: W0727 20:09:43.052596    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fb9496_7142_445d_a970_06f33e1314a3.slice/crio-9a8d7e8cea9e4ecae51e20d5ccad86c44a650520f77232dc4d341837ef1d7318 WatchSource:0}: Error finding container 9a8d7e8cea9e4ecae51e20d5ccad86c44a650520f77232dc4d341837ef1d7318: Status 404 returned error can't find the container with id 9a8d7e8cea9e4ecae51e20d5ccad86c44a650520f77232dc4d341837ef1d7318
Jul 27 20:09:43.054873 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:43.054838    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:09:43.054946 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:43.054912    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:09:43.054946 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:43.054938    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:09:43.363946 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:43.363852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn" event={"ID":"61fb9496-7142-445d-a970-06f33e1314a3","Type":"ContainerStarted","Data":"671336c825bdde11086c7e769ddc552482b1bdb811baec4d96b3b83eec7f123b"}
Jul 27 20:09:43.363946 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:43.363923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn" event={"ID":"61fb9496-7142-445d-a970-06f33e1314a3","Type":"ContainerStarted","Data":"9a8d7e8cea9e4ecae51e20d5ccad86c44a650520f77232dc4d341837ef1d7318"}
Jul 27 20:09:43.395831 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:43.395766    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn" podStartSLOduration=1.395743286 podStartE2EDuration="1.395743286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:09:43.394335777 +0000 UTC m=+3125.910040281" watchObservedRunningTime="2026-07-27 20:09:43.395743286 +0000 UTC m=+3125.911447797"
Jul 27 20:09:43.900498 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:43.900463    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:44.905270 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:44.905225    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:45.375215 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:45.375180    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:45.376086 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:45.376065    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:09:59.449626 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:59.449544    2580 generic.go:356] "Generic (PLEG): container finished" podID="61fb9496-7142-445d-a970-06f33e1314a3" containerID="671336c825bdde11086c7e769ddc552482b1bdb811baec4d96b3b83eec7f123b" exitCode=0
Jul 27 20:09:59.449626 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:59.449593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn" event={"ID":"61fb9496-7142-445d-a970-06f33e1314a3","Type":"ContainerDied","Data":"671336c825bdde11086c7e769ddc552482b1bdb811baec4d96b3b83eec7f123b"}
Jul 27 20:09:59.450076 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:59.449954    2580 scope.go:117] "RemoveContainer" containerID="671336c825bdde11086c7e769ddc552482b1bdb811baec4d96b3b83eec7f123b"
Jul 27 20:09:59.450987 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:59.450962    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:09:59.451046 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:59.451029    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:09:59.451088 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:09:59.451057    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 20:10:00.375969 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:00.375918    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:10:00.471949 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:00.471909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn" event={"ID":"61fb9496-7142-445d-a970-06f33e1314a3","Type":"ContainerStarted","Data":"7a4bec948d3fc97f5c13e9dca6d4c797d1ebcc56db3b96017563fb5c0dcaf22c"}
Jul 27 20:10:00.900850 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:00.900805    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:10:00.902024 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:00.901994    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 27 20:10:00.902152 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:00.902053    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn" podUID="61fb9496-7142-445d-a970-06f33e1314a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 27 20:10:01.900722 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:01.900672    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 27 20:10:01.901128 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:01.900736    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn" podUID="61fb9496-7142-445d-a970-06f33e1314a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 27 20:10:02.900632 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:02.900592    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 27 20:10:02.900816 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:02.900648    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn" podUID="61fb9496-7142-445d-a970-06f33e1314a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 27 20:10:03.900738 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:03.900697    2580 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 27 20:10:03.900911 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:03.900759    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn" podUID="61fb9496-7142-445d-a970-06f33e1314a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 27 20:10:04.904640 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:04.904608    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:10:04.905130 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:04.905010    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:10:04.905780 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:04.905763    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:10:22.851051 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:22.851013    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"]
Jul 27 20:10:22.851498 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:22.851226    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn" podUID="61fb9496-7142-445d-a970-06f33e1314a3" containerName="istio-proxy" containerID="cri-o://7a4bec948d3fc97f5c13e9dca6d4c797d1ebcc56db3b96017563fb5c0dcaf22c" gracePeriod=30
Jul 27 20:10:28.110859 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.110833    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:10:28.255988 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.255900    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-workload-certs\") pod \"61fb9496-7142-445d-a970-06f33e1314a3\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") "
Jul 27 20:10:28.255988 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.255943    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-workload-socket\") pod \"61fb9496-7142-445d-a970-06f33e1314a3\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") "
Jul 27 20:10:28.255988 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.255965    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61fb9496-7142-445d-a970-06f33e1314a3-istio-podinfo\") pod \"61fb9496-7142-445d-a970-06f33e1314a3\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") "
Jul 27 20:10:28.256312 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.256010    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61fb9496-7142-445d-a970-06f33e1314a3-istio-token\") pod \"61fb9496-7142-445d-a970-06f33e1314a3\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") "
Jul 27 20:10:28.256312 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.256048    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-credential-socket\") pod \"61fb9496-7142-445d-a970-06f33e1314a3\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") "
Jul 27 20:10:28.256312 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.256063    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-istio-envoy\") pod \"61fb9496-7142-445d-a970-06f33e1314a3\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") "
Jul 27 20:10:28.256312 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.256099    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61fb9496-7142-445d-a970-06f33e1314a3-istiod-ca-cert\") pod \"61fb9496-7142-445d-a970-06f33e1314a3\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") "
Jul 27 20:10:28.256312 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.256122    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jpq4\" (UniqueName: \"kubernetes.io/projected/61fb9496-7142-445d-a970-06f33e1314a3-kube-api-access-6jpq4\") pod \"61fb9496-7142-445d-a970-06f33e1314a3\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") "
Jul 27 20:10:28.256312 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.256156    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-istio-data\") pod \"61fb9496-7142-445d-a970-06f33e1314a3\" (UID: \"61fb9496-7142-445d-a970-06f33e1314a3\") "
Jul 27 20:10:28.256312 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.256199    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "61fb9496-7142-445d-a970-06f33e1314a3" (UID: "61fb9496-7142-445d-a970-06f33e1314a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:10:28.256312 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.256267    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "61fb9496-7142-445d-a970-06f33e1314a3" (UID: "61fb9496-7142-445d-a970-06f33e1314a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:10:28.256735 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.256443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "61fb9496-7142-445d-a970-06f33e1314a3" (UID: "61fb9496-7142-445d-a970-06f33e1314a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:10:28.256735 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.256518    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-credential-socket\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:10:28.256735 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.256538    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-workload-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:10:28.256735 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.256534    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61fb9496-7142-445d-a970-06f33e1314a3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "61fb9496-7142-445d-a970-06f33e1314a3" (UID: "61fb9496-7142-445d-a970-06f33e1314a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:10:28.256735 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.256557    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-workload-socket\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:10:28.257038 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.257017    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-istio-data" (OuterVolumeSpecName: "istio-data") pod "61fb9496-7142-445d-a970-06f33e1314a3" (UID: "61fb9496-7142-445d-a970-06f33e1314a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:10:28.258527 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.258499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "61fb9496-7142-445d-a970-06f33e1314a3" (UID: "61fb9496-7142-445d-a970-06f33e1314a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:10:28.258626 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.258549    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fb9496-7142-445d-a970-06f33e1314a3-istio-token" (OuterVolumeSpecName: "istio-token") pod "61fb9496-7142-445d-a970-06f33e1314a3" (UID: "61fb9496-7142-445d-a970-06f33e1314a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:10:28.258709 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.258693    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fb9496-7142-445d-a970-06f33e1314a3-kube-api-access-6jpq4" (OuterVolumeSpecName: "kube-api-access-6jpq4") pod "61fb9496-7142-445d-a970-06f33e1314a3" (UID: "61fb9496-7142-445d-a970-06f33e1314a3"). InnerVolumeSpecName "kube-api-access-6jpq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:10:28.258750 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.258713    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61fb9496-7142-445d-a970-06f33e1314a3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "61fb9496-7142-445d-a970-06f33e1314a3" (UID: "61fb9496-7142-445d-a970-06f33e1314a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:10:28.357944 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.357905    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61fb9496-7142-445d-a970-06f33e1314a3-istiod-ca-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:10:28.357944 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.357938    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jpq4\" (UniqueName: \"kubernetes.io/projected/61fb9496-7142-445d-a970-06f33e1314a3-kube-api-access-6jpq4\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:10:28.357944 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.357952    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-istio-data\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:10:28.358187 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.357964    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61fb9496-7142-445d-a970-06f33e1314a3-istio-podinfo\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:10:28.358187 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.357974    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61fb9496-7142-445d-a970-06f33e1314a3-istio-token\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:10:28.358187 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.357986    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61fb9496-7142-445d-a970-06f33e1314a3-istio-envoy\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:10:28.627382 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.627348    2580 generic.go:356] "Generic (PLEG): container finished" podID="61fb9496-7142-445d-a970-06f33e1314a3" containerID="7a4bec948d3fc97f5c13e9dca6d4c797d1ebcc56db3b96017563fb5c0dcaf22c" exitCode=0
Jul 27 20:10:28.627563 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.627395    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"
Jul 27 20:10:28.627563 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.627433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn" event={"ID":"61fb9496-7142-445d-a970-06f33e1314a3","Type":"ContainerDied","Data":"7a4bec948d3fc97f5c13e9dca6d4c797d1ebcc56db3b96017563fb5c0dcaf22c"}
Jul 27 20:10:28.627563 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.627485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn" event={"ID":"61fb9496-7142-445d-a970-06f33e1314a3","Type":"ContainerDied","Data":"9a8d7e8cea9e4ecae51e20d5ccad86c44a650520f77232dc4d341837ef1d7318"}
Jul 27 20:10:28.627563 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.627509    2580 scope.go:117] "RemoveContainer" containerID="7a4bec948d3fc97f5c13e9dca6d4c797d1ebcc56db3b96017563fb5c0dcaf22c"
Jul 27 20:10:28.638358 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.638336    2580 scope.go:117] "RemoveContainer" containerID="671336c825bdde11086c7e769ddc552482b1bdb811baec4d96b3b83eec7f123b"
Jul 27 20:10:28.646921 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.646904    2580 scope.go:117] "RemoveContainer" containerID="7a4bec948d3fc97f5c13e9dca6d4c797d1ebcc56db3b96017563fb5c0dcaf22c"
Jul 27 20:10:28.647154 ip-10-0-129-103 kubenswrapper[2580]: E0727 20:10:28.647135    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a4bec948d3fc97f5c13e9dca6d4c797d1ebcc56db3b96017563fb5c0dcaf22c\": container with ID starting with 7a4bec948d3fc97f5c13e9dca6d4c797d1ebcc56db3b96017563fb5c0dcaf22c not found: ID does not exist" containerID="7a4bec948d3fc97f5c13e9dca6d4c797d1ebcc56db3b96017563fb5c0dcaf22c"
Jul 27 20:10:28.647229 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.647166    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a4bec948d3fc97f5c13e9dca6d4c797d1ebcc56db3b96017563fb5c0dcaf22c"} err="failed to get container status \"7a4bec948d3fc97f5c13e9dca6d4c797d1ebcc56db3b96017563fb5c0dcaf22c\": rpc error: code = NotFound desc = could not find container \"7a4bec948d3fc97f5c13e9dca6d4c797d1ebcc56db3b96017563fb5c0dcaf22c\": container with ID starting with 7a4bec948d3fc97f5c13e9dca6d4c797d1ebcc56db3b96017563fb5c0dcaf22c not found: ID does not exist"
Jul 27 20:10:28.647229 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.647189    2580 scope.go:117] "RemoveContainer" containerID="671336c825bdde11086c7e769ddc552482b1bdb811baec4d96b3b83eec7f123b"
Jul 27 20:10:28.647477 ip-10-0-129-103 kubenswrapper[2580]: E0727 20:10:28.647456    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671336c825bdde11086c7e769ddc552482b1bdb811baec4d96b3b83eec7f123b\": container with ID starting with 671336c825bdde11086c7e769ddc552482b1bdb811baec4d96b3b83eec7f123b not found: ID does not exist" containerID="671336c825bdde11086c7e769ddc552482b1bdb811baec4d96b3b83eec7f123b"
Jul 27 20:10:28.647534 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.647486    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671336c825bdde11086c7e769ddc552482b1bdb811baec4d96b3b83eec7f123b"} err="failed to get container status \"671336c825bdde11086c7e769ddc552482b1bdb811baec4d96b3b83eec7f123b\": rpc error: code = NotFound desc = could not find container \"671336c825bdde11086c7e769ddc552482b1bdb811baec4d96b3b83eec7f123b\": container with ID starting with 671336c825bdde11086c7e769ddc552482b1bdb811baec4d96b3b83eec7f123b not found: ID does not exist"
Jul 27 20:10:28.657070 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.657039    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"]
Jul 27 20:10:28.663682 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:28.663662    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d8e48b-openshift-default-7d88794578-bbcxn"]
Jul 27 20:10:30.086063 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:30.086027    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61fb9496-7142-445d-a970-06f33e1314a3" path="/var/lib/kubelet/pods/61fb9496-7142-445d-a970-06f33e1314a3/volumes"
Jul 27 20:10:52.115643 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:10:52.115601    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"]
Jul 27 20:11:07.134451 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.134404    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp" podUID="024e7232-f884-4dce-be6f-f3f9ef370d80" containerName="main" containerID="cri-o://e3d71ed77e799ef511b43a0a21e6d0be14c275d3326a46ec2af1c49f069ab95e" gracePeriod=45
Jul 27 20:11:07.383448 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.383421    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:11:07.516045 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.515955    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-tmp-dir\") pod \"024e7232-f884-4dce-be6f-f3f9ef370d80\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") "
Jul 27 20:11:07.516045 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.516008    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fph5\" (UniqueName: \"kubernetes.io/projected/024e7232-f884-4dce-be6f-f3f9ef370d80-kube-api-access-8fph5\") pod \"024e7232-f884-4dce-be6f-f3f9ef370d80\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") "
Jul 27 20:11:07.516304 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.516079    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-model-cache\") pod \"024e7232-f884-4dce-be6f-f3f9ef370d80\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") "
Jul 27 20:11:07.516304 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.516106    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-home\") pod \"024e7232-f884-4dce-be6f-f3f9ef370d80\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") "
Jul 27 20:11:07.516304 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.516146    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-dshm\") pod \"024e7232-f884-4dce-be6f-f3f9ef370d80\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") "
Jul 27 20:11:07.516304 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.516174    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024e7232-f884-4dce-be6f-f3f9ef370d80-tls-certs\") pod \"024e7232-f884-4dce-be6f-f3f9ef370d80\" (UID: \"024e7232-f884-4dce-be6f-f3f9ef370d80\") "
Jul 27 20:11:07.516508 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.516349    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "024e7232-f884-4dce-be6f-f3f9ef370d80" (UID: "024e7232-f884-4dce-be6f-f3f9ef370d80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:11:07.516508 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.516387    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-model-cache" (OuterVolumeSpecName: "model-cache") pod "024e7232-f884-4dce-be6f-f3f9ef370d80" (UID: "024e7232-f884-4dce-be6f-f3f9ef370d80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:11:07.516508 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.516445    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-home" (OuterVolumeSpecName: "home") pod "024e7232-f884-4dce-be6f-f3f9ef370d80" (UID: "024e7232-f884-4dce-be6f-f3f9ef370d80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:11:07.516622 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.516546    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-tmp-dir\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:07.516622 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.516561    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-model-cache\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:07.516622 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.516570    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-home\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:07.518368 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.518345    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-dshm" (OuterVolumeSpecName: "dshm") pod "024e7232-f884-4dce-be6f-f3f9ef370d80" (UID: "024e7232-f884-4dce-be6f-f3f9ef370d80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:11:07.518491 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.518469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024e7232-f884-4dce-be6f-f3f9ef370d80-kube-api-access-8fph5" (OuterVolumeSpecName: "kube-api-access-8fph5") pod "024e7232-f884-4dce-be6f-f3f9ef370d80" (UID: "024e7232-f884-4dce-be6f-f3f9ef370d80"). InnerVolumeSpecName "kube-api-access-8fph5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:11:07.518539 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.518484    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024e7232-f884-4dce-be6f-f3f9ef370d80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "024e7232-f884-4dce-be6f-f3f9ef370d80" (UID: "024e7232-f884-4dce-be6f-f3f9ef370d80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:11:07.617148 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.617105    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fph5\" (UniqueName: \"kubernetes.io/projected/024e7232-f884-4dce-be6f-f3f9ef370d80-kube-api-access-8fph5\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:07.617148 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.617141    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024e7232-f884-4dce-be6f-f3f9ef370d80-dshm\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:07.617148 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.617153    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024e7232-f884-4dce-be6f-f3f9ef370d80-tls-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:07.834476 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.834438    2580 generic.go:356] "Generic (PLEG): container finished" podID="024e7232-f884-4dce-be6f-f3f9ef370d80" containerID="e3d71ed77e799ef511b43a0a21e6d0be14c275d3326a46ec2af1c49f069ab95e" exitCode=0
Jul 27 20:11:07.834670 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.834489    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"
Jul 27 20:11:07.834670 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.834524    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp" event={"ID":"024e7232-f884-4dce-be6f-f3f9ef370d80","Type":"ContainerDied","Data":"e3d71ed77e799ef511b43a0a21e6d0be14c275d3326a46ec2af1c49f069ab95e"}
Jul 27 20:11:07.834670 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.834593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp" event={"ID":"024e7232-f884-4dce-be6f-f3f9ef370d80","Type":"ContainerDied","Data":"289b1f7a9884663499f8a0afaf39f9e1d0b25ec540420ceef18b981ef8296d08"}
Jul 27 20:11:07.834670 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.834615    2580 scope.go:117] "RemoveContainer" containerID="e3d71ed77e799ef511b43a0a21e6d0be14c275d3326a46ec2af1c49f069ab95e"
Jul 27 20:11:07.844166 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.844142    2580 scope.go:117] "RemoveContainer" containerID="e3d71ed77e799ef511b43a0a21e6d0be14c275d3326a46ec2af1c49f069ab95e"
Jul 27 20:11:07.844488 ip-10-0-129-103 kubenswrapper[2580]: E0727 20:11:07.844467    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d71ed77e799ef511b43a0a21e6d0be14c275d3326a46ec2af1c49f069ab95e\": container with ID starting with e3d71ed77e799ef511b43a0a21e6d0be14c275d3326a46ec2af1c49f069ab95e not found: ID does not exist" containerID="e3d71ed77e799ef511b43a0a21e6d0be14c275d3326a46ec2af1c49f069ab95e"
Jul 27 20:11:07.844560 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.844502    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d71ed77e799ef511b43a0a21e6d0be14c275d3326a46ec2af1c49f069ab95e"} err="failed to get container status \"e3d71ed77e799ef511b43a0a21e6d0be14c275d3326a46ec2af1c49f069ab95e\": rpc error: code = NotFound desc = could not find container \"e3d71ed77e799ef511b43a0a21e6d0be14c275d3326a46ec2af1c49f069ab95e\": container with ID starting with e3d71ed77e799ef511b43a0a21e6d0be14c275d3326a46ec2af1c49f069ab95e not found: ID does not exist"
Jul 27 20:11:07.855899 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.855813    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"]
Jul 27 20:11:07.858515 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:07.858484    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-58c998/ipp-route-58c998-kserve-7f487b747-vtqdp"]
Jul 27 20:11:08.087273 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:08.087162    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="024e7232-f884-4dce-be6f-f3f9ef370d80" path="/var/lib/kubelet/pods/024e7232-f884-4dce-be6f-f3f9ef370d80/volumes"
Jul 27 20:11:35.483765 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:35.483692    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"]
Jul 27 20:11:35.484267 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:35.483903    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c" podUID="b40db455-a2a9-4eb3-9be9-fce3b9462512" containerName="istio-proxy" containerID="cri-o://f82a271b871ce047331f223d883f3f3689592cc368faeeb4356517e371da7e06" gracePeriod=30
Jul 27 20:11:40.736121 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.736097    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:11:40.813178 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813147    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-credential-socket\") pod \"b40db455-a2a9-4eb3-9be9-fce3b9462512\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") "
Jul 27 20:11:40.813386 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813187    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-workload-certs\") pod \"b40db455-a2a9-4eb3-9be9-fce3b9462512\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") "
Jul 27 20:11:40.813386 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813223    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-token\") pod \"b40db455-a2a9-4eb3-9be9-fce3b9462512\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") "
Jul 27 20:11:40.813386 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813295    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-podinfo\") pod \"b40db455-a2a9-4eb3-9be9-fce3b9462512\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") "
Jul 27 20:11:40.813386 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813316    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-workload-socket\") pod \"b40db455-a2a9-4eb3-9be9-fce3b9462512\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") "
Jul 27 20:11:40.813386 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813335    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b40db455-a2a9-4eb3-9be9-fce3b9462512-istiod-ca-cert\") pod \"b40db455-a2a9-4eb3-9be9-fce3b9462512\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") "
Jul 27 20:11:40.813386 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813363    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-data\") pod \"b40db455-a2a9-4eb3-9be9-fce3b9462512\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") "
Jul 27 20:11:40.813711 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813394    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg2vd\" (UniqueName: \"kubernetes.io/projected/b40db455-a2a9-4eb3-9be9-fce3b9462512-kube-api-access-pg2vd\") pod \"b40db455-a2a9-4eb3-9be9-fce3b9462512\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") "
Jul 27 20:11:40.813711 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-envoy\") pod \"b40db455-a2a9-4eb3-9be9-fce3b9462512\" (UID: \"b40db455-a2a9-4eb3-9be9-fce3b9462512\") "
Jul 27 20:11:40.813711 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813492    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b40db455-a2a9-4eb3-9be9-fce3b9462512" (UID: "b40db455-a2a9-4eb3-9be9-fce3b9462512"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:11:40.813711 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813514    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b40db455-a2a9-4eb3-9be9-fce3b9462512" (UID: "b40db455-a2a9-4eb3-9be9-fce3b9462512"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:11:40.813711 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813666    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b40db455-a2a9-4eb3-9be9-fce3b9462512" (UID: "b40db455-a2a9-4eb3-9be9-fce3b9462512"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:11:40.813971 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813754    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-workload-socket\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:40.813971 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813799    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-credential-socket\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:40.813971 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813818    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-workload-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:40.813971 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.813870    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40db455-a2a9-4eb3-9be9-fce3b9462512-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b40db455-a2a9-4eb3-9be9-fce3b9462512" (UID: "b40db455-a2a9-4eb3-9be9-fce3b9462512"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:11:40.814319 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.814293    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-data" (OuterVolumeSpecName: "istio-data") pod "b40db455-a2a9-4eb3-9be9-fce3b9462512" (UID: "b40db455-a2a9-4eb3-9be9-fce3b9462512"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:11:40.815741 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.815719    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b40db455-a2a9-4eb3-9be9-fce3b9462512" (UID: "b40db455-a2a9-4eb3-9be9-fce3b9462512"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:11:40.815822 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.815735    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b40db455-a2a9-4eb3-9be9-fce3b9462512" (UID: "b40db455-a2a9-4eb3-9be9-fce3b9462512"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:11:40.815926 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.815907    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-token" (OuterVolumeSpecName: "istio-token") pod "b40db455-a2a9-4eb3-9be9-fce3b9462512" (UID: "b40db455-a2a9-4eb3-9be9-fce3b9462512"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:11:40.815926 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.815915    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40db455-a2a9-4eb3-9be9-fce3b9462512-kube-api-access-pg2vd" (OuterVolumeSpecName: "kube-api-access-pg2vd") pod "b40db455-a2a9-4eb3-9be9-fce3b9462512" (UID: "b40db455-a2a9-4eb3-9be9-fce3b9462512"). InnerVolumeSpecName "kube-api-access-pg2vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:11:40.914777 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.914735    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-token\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:40.914777 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.914766    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-podinfo\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:40.914777 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.914779    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b40db455-a2a9-4eb3-9be9-fce3b9462512-istiod-ca-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:40.915024 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.914790    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-data\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:40.915024 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.914802    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg2vd\" (UniqueName: \"kubernetes.io/projected/b40db455-a2a9-4eb3-9be9-fce3b9462512-kube-api-access-pg2vd\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:40.915024 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:40.914814    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b40db455-a2a9-4eb3-9be9-fce3b9462512-istio-envoy\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:11:41.009001 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:41.008963    2580 generic.go:356] "Generic (PLEG): container finished" podID="b40db455-a2a9-4eb3-9be9-fce3b9462512" containerID="f82a271b871ce047331f223d883f3f3689592cc368faeeb4356517e371da7e06" exitCode=0
Jul 27 20:11:41.009197 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:41.009023    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"
Jul 27 20:11:41.009197 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:41.009031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c" event={"ID":"b40db455-a2a9-4eb3-9be9-fce3b9462512","Type":"ContainerDied","Data":"f82a271b871ce047331f223d883f3f3689592cc368faeeb4356517e371da7e06"}
Jul 27 20:11:41.009197 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:41.009074    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c" event={"ID":"b40db455-a2a9-4eb3-9be9-fce3b9462512","Type":"ContainerDied","Data":"bc3e2270cea0bb9333588cd3a6df5438bd9d4ace4c7f9eb33c06d71e2c7990da"}
Jul 27 20:11:41.009197 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:41.009096    2580 scope.go:117] "RemoveContainer" containerID="f82a271b871ce047331f223d883f3f3689592cc368faeeb4356517e371da7e06"
Jul 27 20:11:41.018464 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:41.018446    2580 scope.go:117] "RemoveContainer" containerID="564017b5211f99037a51bfa4f2bc7715a5c6c84ac92029a348cd9a46d87389ab"
Jul 27 20:11:41.026467 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:41.026446    2580 scope.go:117] "RemoveContainer" containerID="f82a271b871ce047331f223d883f3f3689592cc368faeeb4356517e371da7e06"
Jul 27 20:11:41.026708 ip-10-0-129-103 kubenswrapper[2580]: E0727 20:11:41.026693    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f82a271b871ce047331f223d883f3f3689592cc368faeeb4356517e371da7e06\": container with ID starting with f82a271b871ce047331f223d883f3f3689592cc368faeeb4356517e371da7e06 not found: ID does not exist" containerID="f82a271b871ce047331f223d883f3f3689592cc368faeeb4356517e371da7e06"
Jul 27 20:11:41.026748 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:41.026716    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82a271b871ce047331f223d883f3f3689592cc368faeeb4356517e371da7e06"} err="failed to get container status \"f82a271b871ce047331f223d883f3f3689592cc368faeeb4356517e371da7e06\": rpc error: code = NotFound desc = could not find container \"f82a271b871ce047331f223d883f3f3689592cc368faeeb4356517e371da7e06\": container with ID starting with f82a271b871ce047331f223d883f3f3689592cc368faeeb4356517e371da7e06 not found: ID does not exist"
Jul 27 20:11:41.026748 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:41.026731    2580 scope.go:117] "RemoveContainer" containerID="564017b5211f99037a51bfa4f2bc7715a5c6c84ac92029a348cd9a46d87389ab"
Jul 27 20:11:41.026944 ip-10-0-129-103 kubenswrapper[2580]: E0727 20:11:41.026929    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"564017b5211f99037a51bfa4f2bc7715a5c6c84ac92029a348cd9a46d87389ab\": container with ID starting with 564017b5211f99037a51bfa4f2bc7715a5c6c84ac92029a348cd9a46d87389ab not found: ID does not exist" containerID="564017b5211f99037a51bfa4f2bc7715a5c6c84ac92029a348cd9a46d87389ab"
Jul 27 20:11:41.026981 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:41.026949    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"564017b5211f99037a51bfa4f2bc7715a5c6c84ac92029a348cd9a46d87389ab"} err="failed to get container status \"564017b5211f99037a51bfa4f2bc7715a5c6c84ac92029a348cd9a46d87389ab\": rpc error: code = NotFound desc = could not find container \"564017b5211f99037a51bfa4f2bc7715a5c6c84ac92029a348cd9a46d87389ab\": container with ID starting with 564017b5211f99037a51bfa4f2bc7715a5c6c84ac92029a348cd9a46d87389ab not found: ID does not exist"
Jul 27 20:11:41.031958 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:41.031935    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"]
Jul 27 20:11:41.036017 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:41.035995    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-58c998-openshift-default-76f5ffd674-bdz8c"]
Jul 27 20:11:42.085620 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:42.085586    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b40db455-a2a9-4eb3-9be9-fce3b9462512" path="/var/lib/kubelet/pods/b40db455-a2a9-4eb3-9be9-fce3b9462512/volumes"
Jul 27 20:11:57.861289 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:57.861234    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"]
Jul 27 20:11:57.861767 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:11:57.861509    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn" podUID="71ac46c6-257b-4086-bb0a-7e293480800d" containerName="istio-proxy" containerID="cri-o://505dd7064bc547630b066e6a27d42b28061cee328c71e2cbadfe97c9ca59a610" gracePeriod=30
Jul 27 20:12:03.110503 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.110479    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:12:03.126965 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.126905    2580 generic.go:356] "Generic (PLEG): container finished" podID="71ac46c6-257b-4086-bb0a-7e293480800d" containerID="505dd7064bc547630b066e6a27d42b28061cee328c71e2cbadfe97c9ca59a610" exitCode=0
Jul 27 20:12:03.126965 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.126945    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"
Jul 27 20:12:03.127110 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.126977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn" event={"ID":"71ac46c6-257b-4086-bb0a-7e293480800d","Type":"ContainerDied","Data":"505dd7064bc547630b066e6a27d42b28061cee328c71e2cbadfe97c9ca59a610"}
Jul 27 20:12:03.127110 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.127024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn" event={"ID":"71ac46c6-257b-4086-bb0a-7e293480800d","Type":"ContainerDied","Data":"e626918a02fc94dc9e9d50c0d145ca7281e8df619eda23588d294f6cb7d07af2"}
Jul 27 20:12:03.127110 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.127048    2580 scope.go:117] "RemoveContainer" containerID="505dd7064bc547630b066e6a27d42b28061cee328c71e2cbadfe97c9ca59a610"
Jul 27 20:12:03.136982 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.136962    2580 scope.go:117] "RemoveContainer" containerID="78317a4a21cf70035d7e15690abe85aff86bfff2b8c9e3e599e10833c73130c4"
Jul 27 20:12:03.145727 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.145711    2580 scope.go:117] "RemoveContainer" containerID="505dd7064bc547630b066e6a27d42b28061cee328c71e2cbadfe97c9ca59a610"
Jul 27 20:12:03.145981 ip-10-0-129-103 kubenswrapper[2580]: E0727 20:12:03.145963    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505dd7064bc547630b066e6a27d42b28061cee328c71e2cbadfe97c9ca59a610\": container with ID starting with 505dd7064bc547630b066e6a27d42b28061cee328c71e2cbadfe97c9ca59a610 not found: ID does not exist" containerID="505dd7064bc547630b066e6a27d42b28061cee328c71e2cbadfe97c9ca59a610"
Jul 27 20:12:03.146062 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.145991    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505dd7064bc547630b066e6a27d42b28061cee328c71e2cbadfe97c9ca59a610"} err="failed to get container status \"505dd7064bc547630b066e6a27d42b28061cee328c71e2cbadfe97c9ca59a610\": rpc error: code = NotFound desc = could not find container \"505dd7064bc547630b066e6a27d42b28061cee328c71e2cbadfe97c9ca59a610\": container with ID starting with 505dd7064bc547630b066e6a27d42b28061cee328c71e2cbadfe97c9ca59a610 not found: ID does not exist"
Jul 27 20:12:03.146062 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.146014    2580 scope.go:117] "RemoveContainer" containerID="78317a4a21cf70035d7e15690abe85aff86bfff2b8c9e3e599e10833c73130c4"
Jul 27 20:12:03.146300 ip-10-0-129-103 kubenswrapper[2580]: E0727 20:12:03.146280    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78317a4a21cf70035d7e15690abe85aff86bfff2b8c9e3e599e10833c73130c4\": container with ID starting with 78317a4a21cf70035d7e15690abe85aff86bfff2b8c9e3e599e10833c73130c4 not found: ID does not exist" containerID="78317a4a21cf70035d7e15690abe85aff86bfff2b8c9e3e599e10833c73130c4"
Jul 27 20:12:03.146360 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.146305    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78317a4a21cf70035d7e15690abe85aff86bfff2b8c9e3e599e10833c73130c4"} err="failed to get container status \"78317a4a21cf70035d7e15690abe85aff86bfff2b8c9e3e599e10833c73130c4\": rpc error: code = NotFound desc = could not find container \"78317a4a21cf70035d7e15690abe85aff86bfff2b8c9e3e599e10833c73130c4\": container with ID starting with 78317a4a21cf70035d7e15690abe85aff86bfff2b8c9e3e599e10833c73130c4 not found: ID does not exist"
Jul 27 20:12:03.208794 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.208757    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ac46c6-257b-4086-bb0a-7e293480800d-istio-token\") pod \"71ac46c6-257b-4086-bb0a-7e293480800d\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") "
Jul 27 20:12:03.208989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.208805    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ac46c6-257b-4086-bb0a-7e293480800d-istiod-ca-cert\") pod \"71ac46c6-257b-4086-bb0a-7e293480800d\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") "
Jul 27 20:12:03.208989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.208860    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-workload-socket\") pod \"71ac46c6-257b-4086-bb0a-7e293480800d\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") "
Jul 27 20:12:03.208989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.208899    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-istio-data\") pod \"71ac46c6-257b-4086-bb0a-7e293480800d\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") "
Jul 27 20:12:03.208989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.208932    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ac46c6-257b-4086-bb0a-7e293480800d-istio-podinfo\") pod \"71ac46c6-257b-4086-bb0a-7e293480800d\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") "
Jul 27 20:12:03.208989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.208959    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-istio-envoy\") pod \"71ac46c6-257b-4086-bb0a-7e293480800d\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") "
Jul 27 20:12:03.209370 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.209015    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-workload-certs\") pod \"71ac46c6-257b-4086-bb0a-7e293480800d\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") "
Jul 27 20:12:03.209370 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.209229    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "71ac46c6-257b-4086-bb0a-7e293480800d" (UID: "71ac46c6-257b-4086-bb0a-7e293480800d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:12:03.209370 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.209332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "71ac46c6-257b-4086-bb0a-7e293480800d" (UID: "71ac46c6-257b-4086-bb0a-7e293480800d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:12:03.209370 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.209342    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ac46c6-257b-4086-bb0a-7e293480800d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "71ac46c6-257b-4086-bb0a-7e293480800d" (UID: "71ac46c6-257b-4086-bb0a-7e293480800d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:12:03.209551 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.209430    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9rl6\" (UniqueName: \"kubernetes.io/projected/71ac46c6-257b-4086-bb0a-7e293480800d-kube-api-access-n9rl6\") pod \"71ac46c6-257b-4086-bb0a-7e293480800d\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") "
Jul 27 20:12:03.209551 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.209453    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-credential-socket\") pod \"71ac46c6-257b-4086-bb0a-7e293480800d\" (UID: \"71ac46c6-257b-4086-bb0a-7e293480800d\") "
Jul 27 20:12:03.209858 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.209755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-istio-data" (OuterVolumeSpecName: "istio-data") pod "71ac46c6-257b-4086-bb0a-7e293480800d" (UID: "71ac46c6-257b-4086-bb0a-7e293480800d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:12:03.209858 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.209851    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "71ac46c6-257b-4086-bb0a-7e293480800d" (UID: "71ac46c6-257b-4086-bb0a-7e293480800d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:12:03.210037 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.210019    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-workload-certs\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:12:03.210098 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.210043    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ac46c6-257b-4086-bb0a-7e293480800d-istiod-ca-cert\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:12:03.210098 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.210058    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-workload-socket\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:12:03.211442 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.211401    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71ac46c6-257b-4086-bb0a-7e293480800d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "71ac46c6-257b-4086-bb0a-7e293480800d" (UID: "71ac46c6-257b-4086-bb0a-7e293480800d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:12:03.211442 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.211439    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ac46c6-257b-4086-bb0a-7e293480800d-kube-api-access-n9rl6" (OuterVolumeSpecName: "kube-api-access-n9rl6") pod "71ac46c6-257b-4086-bb0a-7e293480800d" (UID: "71ac46c6-257b-4086-bb0a-7e293480800d"). InnerVolumeSpecName "kube-api-access-n9rl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:12:03.211615 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.211557    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "71ac46c6-257b-4086-bb0a-7e293480800d" (UID: "71ac46c6-257b-4086-bb0a-7e293480800d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:12:03.211615 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.211572    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ac46c6-257b-4086-bb0a-7e293480800d-istio-token" (OuterVolumeSpecName: "istio-token") pod "71ac46c6-257b-4086-bb0a-7e293480800d" (UID: "71ac46c6-257b-4086-bb0a-7e293480800d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:12:03.311403 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.311375    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ac46c6-257b-4086-bb0a-7e293480800d-istio-podinfo\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:12:03.311403 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.311399    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-istio-envoy\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:12:03.311403 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.311409    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9rl6\" (UniqueName: \"kubernetes.io/projected/71ac46c6-257b-4086-bb0a-7e293480800d-kube-api-access-n9rl6\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:12:03.311638 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.311419    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-credential-socket\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:12:03.311638 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.311427    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ac46c6-257b-4086-bb0a-7e293480800d-istio-token\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:12:03.311638 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.311436    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ac46c6-257b-4086-bb0a-7e293480800d-istio-data\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:12:03.449625 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.449596    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"]
Jul 27 20:12:03.453745 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:03.453721    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-791fb5-openshift-default-58687ccd56-ndzbn"]
Jul 27 20:12:04.086047 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:04.086013    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ac46c6-257b-4086-bb0a-7e293480800d" path="/var/lib/kubelet/pods/71ac46c6-257b-4086-bb0a-7e293480800d/volumes"
Jul 27 20:12:20.757349 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757313    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp"]
Jul 27 20:12:20.757989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757772    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61fb9496-7142-445d-a970-06f33e1314a3" containerName="istio-proxy"
Jul 27 20:12:20.757989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757790    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="61fb9496-7142-445d-a970-06f33e1314a3" containerName="istio-proxy"
Jul 27 20:12:20.757989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757803    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ac46c6-257b-4086-bb0a-7e293480800d" containerName="istio-proxy"
Jul 27 20:12:20.757989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757813    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ac46c6-257b-4086-bb0a-7e293480800d" containerName="istio-proxy"
Jul 27 20:12:20.757989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757833    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b40db455-a2a9-4eb3-9be9-fce3b9462512" containerName="istio-proxy"
Jul 27 20:12:20.757989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757842    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b40db455-a2a9-4eb3-9be9-fce3b9462512" containerName="istio-proxy"
Jul 27 20:12:20.757989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757851    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61fb9496-7142-445d-a970-06f33e1314a3" containerName="istio-proxy"
Jul 27 20:12:20.757989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757859    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="61fb9496-7142-445d-a970-06f33e1314a3" containerName="istio-proxy"
Jul 27 20:12:20.757989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757872    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="024e7232-f884-4dce-be6f-f3f9ef370d80" containerName="main"
Jul 27 20:12:20.757989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757880    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="024e7232-f884-4dce-be6f-f3f9ef370d80" containerName="main"
Jul 27 20:12:20.757989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757965    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ac46c6-257b-4086-bb0a-7e293480800d" containerName="istio-proxy"
Jul 27 20:12:20.757989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757976    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="024e7232-f884-4dce-be6f-f3f9ef370d80" containerName="main"
Jul 27 20:12:20.757989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757984    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="61fb9496-7142-445d-a970-06f33e1314a3" containerName="istio-proxy"
Jul 27 20:12:20.757989 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.757993    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ac46c6-257b-4086-bb0a-7e293480800d" containerName="istio-proxy"
Jul 27 20:12:20.758697 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.758004    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b40db455-a2a9-4eb3-9be9-fce3b9462512" containerName="istio-proxy"
Jul 27 20:12:20.758697 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.758012    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b40db455-a2a9-4eb3-9be9-fce3b9462512" containerName="istio-proxy"
Jul 27 20:12:20.760354 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.760333    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp"
Jul 27 20:12:20.762532 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.762510    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\""
Jul 27 20:12:20.762693 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.762676    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 20:12:20.768367 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.768342    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp"]
Jul 27 20:12:20.861528 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.861481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp\" (UID: \"94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp"
Jul 27 20:12:20.962807 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.962761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp\" (UID: \"94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp"
Jul 27 20:12:20.963448 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:20.963428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp\" (UID: \"94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp"
Jul 27 20:12:21.072467 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:21.072437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp"
Jul 27 20:12:21.208538 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:21.208512    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp"]
Jul 27 20:12:21.210948 ip-10-0-129-103 kubenswrapper[2580]: W0727 20:12:21.210923    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d4e3de_41f8_46a1_a9fa_2f4aa0b187ec.slice/crio-1311909d32a9555c126e69a94560275b86a71f2e47aba9b3578098adde44de4b WatchSource:0}: Error finding container 1311909d32a9555c126e69a94560275b86a71f2e47aba9b3578098adde44de4b: Status 404 returned error can't find the container with id 1311909d32a9555c126e69a94560275b86a71f2e47aba9b3578098adde44de4b
Jul 27 20:12:21.217551 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:21.217524    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp" event={"ID":"94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec","Type":"ContainerStarted","Data":"1311909d32a9555c126e69a94560275b86a71f2e47aba9b3578098adde44de4b"}
Jul 27 20:12:22.223871 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:22.223779    2580 generic.go:356] "Generic (PLEG): container finished" podID="94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec" containerID="8001e7dd67a3690f159516c94318766766a993150b76c5fa8ddee7d98c7ac337" exitCode=0
Jul 27 20:12:22.224233 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:22.223865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp" event={"ID":"94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec","Type":"ContainerDied","Data":"8001e7dd67a3690f159516c94318766766a993150b76c5fa8ddee7d98c7ac337"}
Jul 27 20:12:23.361056 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:23.361031    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp"
Jul 27 20:12:23.485080 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:23.484991    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec-maas-api-service-ca\") pod \"94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec\" (UID: \"94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec\") "
Jul 27 20:12:23.485400 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:23.485375    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec" (UID: "94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:12:23.586019 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:23.585967    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec-maas-api-service-ca\") on node \"ip-10-0-129-103.ec2.internal\" DevicePath \"\""
Jul 27 20:12:24.234913 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:24.234879    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp" event={"ID":"94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec","Type":"ContainerDied","Data":"1311909d32a9555c126e69a94560275b86a71f2e47aba9b3578098adde44de4b"}
Jul 27 20:12:24.234913 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:24.234905    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp"
Jul 27 20:12:24.235147 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:24.234932    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1311909d32a9555c126e69a94560275b86a71f2e47aba9b3578098adde44de4b"
Jul 27 20:12:31.381286 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:31.381227    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f99bfd-29753040-vbzzx"]
Jul 27 20:12:31.383105 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:31.383081    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f99bfd-29753040-vbzzx"]
Jul 27 20:12:32.086338 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:32.086305    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b905e6c-834d-46db-8033-a5637f99728a" path="/var/lib/kubelet/pods/8b905e6c-834d-46db-8033-a5637f99728a/volumes"
Jul 27 20:12:35.841662 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:35.841623    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp"]
Jul 27 20:12:35.844467 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:35.844439    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f99bfd-5fshp"]
Jul 27 20:12:36.085533 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:36.085498    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec" path="/var/lib/kubelet/pods/94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec/volumes"
Jul 27 20:12:38.475425 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:38.475395    2580 scope.go:117] "RemoveContainer" containerID="70a0c83184d0898bebc3e3e5b46c7a00ac219ee39687750bc55831496b1526de"
Jul 27 20:12:43.844652 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:43.844615    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67884bbdc-4mxts_7ef2bdcb-9a9b-496d-8c5f-a05f4293947d/authorino/0.log"
Jul 27 20:12:49.744890 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:49.744857    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d79566f86-jwj8m_b08034f2-0acf-44fc-8dae-42db64288bab/copy-manifests/0.log"
Jul 27 20:12:49.752065 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:49.752037    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d79566f86-jwj8m_b08034f2-0acf-44fc-8dae-42db64288bab/manager/0.log"
Jul 27 20:12:51.578743 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:51.578710    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m_2a2c055d-801f-4365-b9b8-487b48ef669b/util/0.log"
Jul 27 20:12:51.586416 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:51.586390    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m_2a2c055d-801f-4365-b9b8-487b48ef669b/pull/0.log"
Jul 27 20:12:51.593116 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:51.593086    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m_2a2c055d-801f-4365-b9b8-487b48ef669b/extract/0.log"
Jul 27 20:12:51.704765 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:51.704729    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm_c8b28b09-f556-4607-88de-8e687b695110/pull/0.log"
Jul 27 20:12:51.711302 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:51.711274    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm_c8b28b09-f556-4607-88de-8e687b695110/extract/0.log"
Jul 27 20:12:51.717214 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:51.717192    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm_c8b28b09-f556-4607-88de-8e687b695110/util/0.log"
Jul 27 20:12:51.830105 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:51.830024    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw_f47d2dd3-c40a-48ae-b792-986c76171d77/util/0.log"
Jul 27 20:12:51.837408 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:51.837381    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw_f47d2dd3-c40a-48ae-b792-986c76171d77/pull/0.log"
Jul 27 20:12:51.844488 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:51.844461    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw_f47d2dd3-c40a-48ae-b792-986c76171d77/extract/0.log"
Jul 27 20:12:51.956147 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:51.956118    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7_1ed7bb6d-0419-4b71-bac9-833132273e22/util/0.log"
Jul 27 20:12:51.964502 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:51.964473    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7_1ed7bb6d-0419-4b71-bac9-833132273e22/pull/0.log"
Jul 27 20:12:51.974064 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:51.974040    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7_1ed7bb6d-0419-4b71-bac9-833132273e22/extract/0.log"
Jul 27 20:12:52.100201 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:52.100115    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67884bbdc-4mxts_7ef2bdcb-9a9b-496d-8c5f-a05f4293947d/authorino/0.log"
Jul 27 20:12:52.896793 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:52.896755    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-sdjnc_d715763c-b7ef-4584-ac16-c78e14462f37/manager/0.log"
Jul 27 20:12:53.460338 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:53.460308    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-949fd4987-nmvwf_0b40ea9c-865c-4631-afe4-6d48cfa7ffd6/kube-auth-proxy/0.log"
Jul 27 20:12:54.574901 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:54.574865    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-mqgcm_199a9ce2-abc8-409b-93c0-5dd920214dde/main/0.log"
Jul 27 20:12:54.682923 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:54.682896    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwfbmd_f6ef021d-b933-4209-abb1-4cd5d10f634c/main/0.log"
Jul 27 20:12:54.902717 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:12:54.902635    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-4v22l_8e95eaa4-c410-4444-a5d3-a0cf342e9798/main/0.log"
Jul 27 20:13:03.170140 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:03.170058    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2f68s_19c1c8b7-cc69-4cad-aa77-67102d7353d8/global-pull-secret-syncer/0.log"
Jul 27 20:13:03.363597 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:03.363562    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mwdjc_16340d27-1b23-4e10-be1e-7328952623c0/konnectivity-agent/0.log"
Jul 27 20:13:03.432404 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:03.432316    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-103.ec2.internal_1ed3ed7be1c2695b0c97f92390b9feef/haproxy/0.log"
Jul 27 20:13:08.217930 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:08.217897    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m_2a2c055d-801f-4365-b9b8-487b48ef669b/extract/0.log"
Jul 27 20:13:08.243832 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:08.243798    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m_2a2c055d-801f-4365-b9b8-487b48ef669b/util/0.log"
Jul 27 20:13:08.265653 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:08.265625    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mdq2m_2a2c055d-801f-4365-b9b8-487b48ef669b/pull/0.log"
Jul 27 20:13:08.294949 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:08.294910    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm_c8b28b09-f556-4607-88de-8e687b695110/extract/0.log"
Jul 27 20:13:08.321366 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:08.321338    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm_c8b28b09-f556-4607-88de-8e687b695110/util/0.log"
Jul 27 20:13:08.347493 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:08.347471    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lfxdm_c8b28b09-f556-4607-88de-8e687b695110/pull/0.log"
Jul 27 20:13:08.381814 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:08.381784    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw_f47d2dd3-c40a-48ae-b792-986c76171d77/extract/0.log"
Jul 27 20:13:08.404721 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:08.404690    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw_f47d2dd3-c40a-48ae-b792-986c76171d77/util/0.log"
Jul 27 20:13:08.433784 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:08.433749    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73mqfkw_f47d2dd3-c40a-48ae-b792-986c76171d77/pull/0.log"
Jul 27 20:13:08.467396 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:08.467373    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7_1ed7bb6d-0419-4b71-bac9-833132273e22/extract/0.log"
Jul 27 20:13:08.493734 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:08.493669    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7_1ed7bb6d-0419-4b71-bac9-833132273e22/util/0.log"
Jul 27 20:13:08.527294 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:08.527255    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1rnnf7_1ed7bb6d-0419-4b71-bac9-833132273e22/pull/0.log"
Jul 27 20:13:09.143611 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:09.143573    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67884bbdc-4mxts_7ef2bdcb-9a9b-496d-8c5f-a05f4293947d/authorino/0.log"
Jul 27 20:13:09.726967 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:09.726931    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-sdjnc_d715763c-b7ef-4584-ac16-c78e14462f37/manager/0.log"
Jul 27 20:13:11.725673 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:11.725607    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7cc967c9d4-7sqn2_eff50f1b-297b-451f-9f18-8a76669dd4d4/metrics-server/0.log"
Jul 27 20:13:11.750560 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:11.750531    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-snj9q_dd83cc2e-4ca8-418c-8b3a-e3de88c57df7/monitoring-plugin/0.log"
Jul 27 20:13:11.946309 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:11.946274    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wh48w_77479c54-bff3-4d49-a067-26c7be1d304c/node-exporter/0.log"
Jul 27 20:13:11.966081 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:11.966055    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wh48w_77479c54-bff3-4d49-a067-26c7be1d304c/kube-rbac-proxy/0.log"
Jul 27 20:13:11.998141 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:11.998068    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wh48w_77479c54-bff3-4d49-a067-26c7be1d304c/init-textfile/0.log"
Jul 27 20:13:12.269631 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:12.269547    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-xnv97_109f3c2b-5122-44fe-8d17-63e0ffc51857/prometheus-operator/0.log"
Jul 27 20:13:12.288302 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:12.288277    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-xnv97_109f3c2b-5122-44fe-8d17-63e0ffc51857/kube-rbac-proxy/0.log"
Jul 27 20:13:12.313762 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:12.313738    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-c8j6s_e04b8dd0-d698-46e7-b992-9cb211e8ff1c/prometheus-operator-admission-webhook/0.log"
Jul 27 20:13:13.211868 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.211832    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"]
Jul 27 20:13:13.212256 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.212137    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b40db455-a2a9-4eb3-9be9-fce3b9462512" containerName="istio-proxy"
Jul 27 20:13:13.212256 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.212148    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b40db455-a2a9-4eb3-9be9-fce3b9462512" containerName="istio-proxy"
Jul 27 20:13:13.212256 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.212168    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ac46c6-257b-4086-bb0a-7e293480800d" containerName="istio-proxy"
Jul 27 20:13:13.212256 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.212174    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ac46c6-257b-4086-bb0a-7e293480800d" containerName="istio-proxy"
Jul 27 20:13:13.212256 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.212180    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec" containerName="revoke-keys"
Jul 27 20:13:13.212256 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.212185    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec" containerName="revoke-keys"
Jul 27 20:13:13.212256 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.212232    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="94d4e3de-41f8-46a1-a9fa-2f4aa0b187ec" containerName="revoke-keys"
Jul 27 20:13:13.212256 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.212254    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="61fb9496-7142-445d-a970-06f33e1314a3" containerName="istio-proxy"
Jul 27 20:13:13.214913 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.214897    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.217289 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.217268    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5b4zt\"/\"default-dockercfg-m29w6\""
Jul 27 20:13:13.217436 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.217355    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5b4zt\"/\"openshift-service-ca.crt\""
Jul 27 20:13:13.218223 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.218204    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5b4zt\"/\"kube-root-ca.crt\""
Jul 27 20:13:13.223922 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.223898    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"]
Jul 27 20:13:13.301202 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.301167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d746eccf-82fb-45dc-badb-a6bcb23b8a76-proc\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.301377 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.301231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d746eccf-82fb-45dc-badb-a6bcb23b8a76-lib-modules\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.301377 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.301274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d746eccf-82fb-45dc-badb-a6bcb23b8a76-sys\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.301377 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.301327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d746eccf-82fb-45dc-badb-a6bcb23b8a76-podres\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.301377 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.301364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx89t\" (UniqueName: \"kubernetes.io/projected/d746eccf-82fb-45dc-badb-a6bcb23b8a76-kube-api-access-nx89t\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.402597 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.402556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d746eccf-82fb-45dc-badb-a6bcb23b8a76-lib-modules\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.402597 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.402596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d746eccf-82fb-45dc-badb-a6bcb23b8a76-sys\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.402773 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.402629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d746eccf-82fb-45dc-badb-a6bcb23b8a76-podres\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.402773 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.402653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx89t\" (UniqueName: \"kubernetes.io/projected/d746eccf-82fb-45dc-badb-a6bcb23b8a76-kube-api-access-nx89t\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.402773 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.402696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d746eccf-82fb-45dc-badb-a6bcb23b8a76-sys\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.402773 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.402705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d746eccf-82fb-45dc-badb-a6bcb23b8a76-proc\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.402773 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.402745    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d746eccf-82fb-45dc-badb-a6bcb23b8a76-lib-modules\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.402952 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.402789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d746eccf-82fb-45dc-badb-a6bcb23b8a76-podres\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.402952 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.402788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d746eccf-82fb-45dc-badb-a6bcb23b8a76-proc\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.412611 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.412585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx89t\" (UniqueName: \"kubernetes.io/projected/d746eccf-82fb-45dc-badb-a6bcb23b8a76-kube-api-access-nx89t\") pod \"perf-node-gather-daemonset-tq9sn\" (UID: \"d746eccf-82fb-45dc-badb-a6bcb23b8a76\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.525334 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.525227    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:13.652993 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:13.652959    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"]
Jul 27 20:13:13.655663 ip-10-0-129-103 kubenswrapper[2580]: W0727 20:13:13.655628    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd746eccf_82fb_45dc_badb_a6bcb23b8a76.slice/crio-538ad5dcc4f82210c1fc92789f38a2e7eec5f32cd123a5bd080f67d82eb091a0 WatchSource:0}: Error finding container 538ad5dcc4f82210c1fc92789f38a2e7eec5f32cd123a5bd080f67d82eb091a0: Status 404 returned error can't find the container with id 538ad5dcc4f82210c1fc92789f38a2e7eec5f32cd123a5bd080f67d82eb091a0
Jul 27 20:13:14.486662 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:14.486624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn" event={"ID":"d746eccf-82fb-45dc-badb-a6bcb23b8a76","Type":"ContainerStarted","Data":"f8f21537660f22e79ce61a7a19c812ecc1eab812ef8f739ede1e174b356d65c9"}
Jul 27 20:13:14.487072 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:14.486684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn" event={"ID":"d746eccf-82fb-45dc-badb-a6bcb23b8a76","Type":"ContainerStarted","Data":"538ad5dcc4f82210c1fc92789f38a2e7eec5f32cd123a5bd080f67d82eb091a0"}
Jul 27 20:13:14.487072 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:14.486793    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:14.515363 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:14.515312    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn" podStartSLOduration=1.515296749 podStartE2EDuration="1.515296749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:13:14.513114835 +0000 UTC m=+3337.028819339" watchObservedRunningTime="2026-07-27 20:13:14.515296749 +0000 UTC m=+3337.031001252"
Jul 27 20:13:15.155061 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:15.155029    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-8dmg4_70705846-cc85-405d-b05a-ada57468d4ad/download-server/0.log"
Jul 27 20:13:17.028805 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:17.028769    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lbsxn_a6bfc017-280a-4902-8f87-008710dab93d/dns/0.log"
Jul 27 20:13:17.067732 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:17.067706    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lbsxn_a6bfc017-280a-4902-8f87-008710dab93d/kube-rbac-proxy/0.log"
Jul 27 20:13:17.169447 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:17.169421    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w9hmm_379c2805-26eb-4141-bd63-49ce6ad17fee/dns-node-resolver/0.log"
Jul 27 20:13:17.801420 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:17.801383    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-fcd5949f6-b67bh_2864c883-0507-46b5-bfb1-8dc3451ef61c/registry/0.log"
Jul 27 20:13:17.835705 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:17.835676    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8rsqx_dafcc8b6-00fd-4c7b-adb0-1cd6ba1560f3/node-ca/0.log"
Jul 27 20:13:19.129833 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:19.129807    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-949fd4987-nmvwf_0b40ea9c-865c-4631-afe4-6d48cfa7ffd6/kube-auth-proxy/0.log"
Jul 27 20:13:20.197509 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:20.197479    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r4tc4_0eaedab2-c519-483f-b30d-8aede92ec1d8/serve-healthcheck-canary/0.log"
Jul 27 20:13:20.507429 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:20.507353    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-tq9sn"
Jul 27 20:13:20.808056 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:20.808026    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9bx4h_3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4/kube-rbac-proxy/0.log"
Jul 27 20:13:20.831471 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:20.831449    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9bx4h_3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4/exporter/0.log"
Jul 27 20:13:20.853072 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:20.853046    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9bx4h_3fe8111c-eeef-4de3-af3a-f6fa03cbc3a4/extractor/0.log"
Jul 27 20:13:23.825577 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:23.825548    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d79566f86-jwj8m_b08034f2-0acf-44fc-8dae-42db64288bab/manager/0.log"
Jul 27 20:13:23.850108 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:23.850075    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d79566f86-jwj8m_b08034f2-0acf-44fc-8dae-42db64288bab/copy-manifests/0.log"
Jul 27 20:13:24.794357 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:24.794326    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-949fd4987-nmvwf_0b40ea9c-865c-4631-afe4-6d48cfa7ffd6/kube-auth-proxy/0.log"
Jul 27 20:13:25.413395 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:25.413362    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6fd758c6bf-cjt5j_ab1c6f30-49e8-4c0a-bacf-53263f7458fc/manager/0.log"
Jul 27 20:13:25.472903 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:25.472871    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-6w6gf_145dde04-5be2-4ac6-b7d3-c67edd7aee45/openshift-lws-operator/0.log"
Jul 27 20:13:26.589559 ip-10-0-129-103 kubenswrapper[2580]: I0727 20:13:26.589526    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-949fd4987-nmvwf_0b40ea9c-865c-4631-afe4-6d48cfa7ffd6/kube-auth-proxy/0.log"